Quality Assurance/How we prioritize
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
QA meeting Feb 26 -- how do we prioritize what we work on, re testing?
Attending:
- originally: QA was Chris:
- Michelle Grover for mobile
- Zeljko for test automation
- Quim for community events
Chris: overview of QA activities
Projects they've helped with: (13 entries)
- AFTv5 - E2
- Echo - E2
- Page Triage - E2
- VE
- Mobile
- Mobile
- Language
- Guided Tour - E3
- Account Creation - E3
- Math
- UploadWizard
- TMH
- Search
- PDBHandler - Community
- Environments: (11 entries)
Places:
- labs host (ee prototype)
- beta cluster (AFT, mobile)
- caching (Antoine says, "like 99% of our infrastructure is about caching so caching is the top root cause of issues :-]")
- search <--- Lucene could be setup in beta
- jobqueue
- thumbnails
- Lucene
- bits.wikimedia.org
- parsercache
- ResourceLoader
- swift
- mediawiki.org (echo)
- enwiki/prod (ve, e3)
- Test2wiki (upload wizard, math, etc.)
- future: vagrant
- Staff
- Pro volunteers
- Wiki volunteers
- backlog!
- needs subject matter experts (to write in plain english the testing needs)
QA Weekly goals:
- http://www.mediawiki.org/wiki/QA/Weekly_goals
- https://www.mediawiki.org/wiki/QA/Review_February_2013#Notes_from_QA_Quarterly_review_-_2013-02-08
- See "How do we decide what to prioritize? Stuff where it's useful to test before production." discussion
- also see "How do we decide what to focus exploratory testing energy on? How do we balance it on a per-area basis? How do we get those teams to accept the help?" section
- Outcome: things that would benefit from more software testing?
- needs to know what needs to be tested before projection?
E2 wants to be able to test stuff before it is merged -- beta cluster for non-master branches is being worked on.
matrix: Y - features, X- types of testing
X-axis - Types of testing:
- test environment creation
- Automatic
- unit testing (qunit & Jenkins) -- skip for today
- integration testing (Jenkins) -- skip for today
- automated browser testing (can stop because code is bad, or test environement broken, deployment broken). More for stable stuff?
- configuration testing (make sure it works on all browsers and operating systems)
- Exploratory testing
- developer/product manager exploratory testing - manual browser testing, acceptance testing
- volunteer exploratory testing
- QA staff exploratory testing (ex. AFT for Chris, Michelle for mobile)
Prioritization matrix: https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0Agte_lJNpi-OdGpJSHdMZV83UjdTdHc5UFVrdXBwaWc#gid=0 Want from features:
- 1 exploratory testing project/month
- 1 automated testing project/month
- synchronize focuses with community/communications activities (blog posts, community events, etc.)
Current idea of process: build out this spreadsheet, see that it has all our engineering projects & what kinds of testing they're amenable to, and regularly check in to update it with Howie/Chris/Erik.