Jump to content

Topic on Talk:Quality Assurance/Flow

Jeremyb (talkcontribs)

Where's a good place to list new tools that we might want to try out? (or for that matter inventory of everything already in use?)

e.g. reading mediazilla:57811 made me think of https://moztrap.readthedocs.org/

(but I guess that tool is not actually a good fit for the bug linked; bug is talking more about acceptance testing (is this a good UI?) rather than functional/regression testing)

Also, apparently the beta cluster is more used than people realized. (or so I heard someone say during a recent hiccup) If we have so many volunteers testing with it then maybe a tool like this would help organize their efforts. especially could be useful at strategic points in deploy train?

ping user:Greg (WMF). also, user:Qgil-WMF, why was that removed in special:diff/937805?

Cmcmahon(WMF) (talkcontribs)

Hi Jeremy,

You bring up several issues here I can address:

  • Functional testing by volunteers
  • Test case management for persisent, repeatable tests
  • Use of beta cluster by multiple teams for multiple purposes

We have tried several experiments enlisting volunteers to test features under development, and determined through experience that such projects cost more than they benefit. All of the software development projects done by WMF are iterative to some extent, meaning that we always have a backlog of features under development. Naive testers, that is, people not following closely the day-to-day activities of these projects, invariable report bugs in three categories and only three categories: WONTFIX; DUPLICATE; and ENHANCEMENT (with the ENHANCEMENT being known issues managed in the product backlog.)

Our needs for persistent test case management is mostly answered by our suites of automated browser tests. However, I do know that our Language team has a desire to manage a suite of test cases for manual testing, which is the sort of thing that Moztrap does. I've discussed this with them, and we intend to use Phabricator to do this. In the very near future, Phabricator will be the issue-tracking system for all of WMF, so it makes sense to use it for test case management also.

Finally, beta cluster has become an important part of our work across virtually every WMF project. The beta labs shared test environment today cannot really properly support all of the projects it is being used for. We've identified the need for at least one additional test environment similar to beta labs, and we are starting work to build such a test environment.

Reply to "new tools (moztrap?)"