Wikimedia Release Engineering Team/Wishlist

This is the wishlist for the Release Management and QA team.

Code Deploy Dashboard

Problem: We don't have an simple way of communicating what code is deployed to which wikis. Right now you need a combination of Gerrit and the MediaWiki Roadmap.

Stakeholders/use-cases

Product Managers: I want to see one page where I can tell if code my team has written is on all Wikipedias.
Developers: I want to know when code I have written is on the test wikis and when it'll be on English Wikipedia.
Technical Users/Readers: I want to see if there's an obvious excuse for a problem I am seeing on my home wiki due to a recent deployment.
Release Team: All of the above.

Solution ideas

Create a dashboard page that displays the currently deployed/active branches on the WMF Cluster (and, for bonus points, the Beta Cluster). The branches will link to a single page in Gerrit that lists all of the changes and their commit short message.

See this quick/"I'm not a designer" mockup from Greg.

Pieces needed

Timestamp of deploys (and which group of wikis was affected, eg phase0 or phase1) (from graphite/statsd if that ever works again bug 62667)
A way to associated a wiki group with the version that's live (see hashar's work on Special:Version parsing?)
Super basic but informative graphs (probably something pulled from http://gdash.wikimedia.org/dashboards/reqerror/ )
A place to put that all

Release Notes Sanity

Problem:

13:09 <     bd808> Gah! Release notes! F U release notes
13:10 <    greg-g> :)
13:10 <     ori-l> it's one of our "you forgot to say simon says" -1 tricks

Stakeholders:

All MediaWiki developers - When commiting a change to MW that needs a note in the RELEASE-NOTES-X.XX file, I shouldn't have to play games with gerrit and rebasing continuously.

Solution ideas:

rm -f RELEASE-NOTES-X.XX
Merge conflict resolution driver
....

True code pipeline

Right now code does not follow the same pipeline universally. For example:

Code merged to master on Friday will be on the Beta Cluster for one full week before going out on the testwikis
Code merged to master on Thursday early morning will be on the Beta Cluster for all of an hour (or a minute) before going to testwikis.

This is not ok. Code should propagate from master -> Beta Cluster -> test production -> other wikis in a consistent fashion.

Proposal

Nightly test tags - tracking: bug 65126

Every night at a specified time we tag master in core and all extensions in use on the cluster.
pre/post merge unit tests are run against the tag
the tag is deployed to the Beta Cluster using multiversion - bug 65127
browser tests are run against the beta cluster hitting the nightly tag version - bug 65128
in the morning we review the browser tests and unit test output, determine suitability for deploy
deploy the tag as the new wmfXX deployed version on Thursday morning
- If the latest tag is not suitable (due to failed browser tests etc) we use the last suitable nightly (eg Tues night's)

Auto-populate 'Important Changes' from RELEASE-NOTES-1.XX

See the 'important changes' section in most wmfXX release pages, eg this one.

It would be nice to auto-create that based on the RELEASE-NOTES file.

See the RFC: Requests_for_comment/Release_notes_automation

block commits on warnings... almost

Let's make higher quality code. One way is to have a voting test that fails on code compile/build warnings. That's a bit extreme.

An alternative is to compare the number of warnings (and TODOs/FIXMEs?) before and after the commit and only fail if that number increases.

Performance testing environment

See bug 65394