Quality Assurance SIG/20180420

== 4 -20 -18 ==

Attendees: JR, Erik, Elena, Greg

Pre-meeting:

Conferences vs Unconferences

The format of having interactive sessions that you find at Unconfrerences seems more valuable than the lecture style that traditional conferences tend to have.

Topics:

Staging

• Use case 1: Developers have a place to show their work (to PMs, during showcases, etc)

• Use case 2: Developers have a place to ensure their change(s) don't blow up, isolated from other changes being tested

• Use case 3: Developers have a place to test if unanticipated changes to other parts of the system (extensions, service, etc) blow up when combined with their changes

• Use case 4: Deployers have a method of testing release stability in a production-like environment before deploying

• Use case 5: Deployers can reliably test the deployment procedure for complex and simple changes, including schema changes to the databases.

• Use case 6: Developers can test changes for performance regressions before they are released to production.

• Use case 7: Developers can test integration of their changes and the ones of others with data similar or equal to production (see ticket)

• Use case 8: Manual QA has an environment to test changes on, and work with developers to quickly iterate on fixes to bugs they find (timespan: days) [redundant? Overlap with UC 3?]

• Use case 9: Developers, PMs and testers can deploy and test a new MediaWiki extension or service before it’s ready and mature-enough for production (timespan: months)

Erik:

* very limited use right now, I check the emails when our smoke tests fail

** why? dunno

* Desire: a mirror of production traffic

* Greg: how about production data?

** Erik: maybe different languages

Elena

* what are people missing in Beta now?

** me: templates and maps, complex issue

** templates are essential for many types of tests (eg Content Translation)

** Be able to run non-gating tests. Just want to be able to run tests without impacting the build.

What would make it not usable to you?

* when it's down :)

* false positives

* if the code was updated less frequently (eg: daily)

* if I can't log into the nodes/machines and do debugging with the same level of permissions that I have in production

How should we collect use cases from broader population?

* Surveys, Bash

Priority of Staging/Beta Cluster vs other testing improvements

* Erik: not very high, better testing/more test cases are more pressing

**anomaly detection (of our log messages) being higher priority

* Elena: Being that it's been like this for a while, doesn't seem too pressing

Elena: Erik, do you run into any problems in production being that you don't really use Beta?  

*Erik: on occasion, but we feel confident in our test suites and the integration testing we do.  One example of something that we broke in production could have only been caught if we had more than one elasticsearch cluster, which is only available in production.  

Closing thought:  What does testability look like?  How do we help developers design/write more testable code. Will likely be a topic at the next QA Sig meeting.

* Erik: Talk to Guillaume, he's very interested in the topic.