Release Management RFP/2014/Consortium

Hello.

We are an international consortium of core MediaWiki and third-party developers and system administrators for several large third-party projects including ShoutWiki and Uncyclomedia. The MediaWiki release process affects us all as volunteers, developers, designers and system administrators, and we see this proposal as an opportunity to take a larger role in ensuring a release process that works for us and the companies and communities we represent. Our skills and perspectives complement each other like a five-person rock band; any one of us could not do this alone, and together we can tap an extensive community of collaborators, coworkers, end-users and friends to ensure that MediaWiki releases are well-tested, well-documented and as bug-free as possible.

What users need

MediaWiki has millions of users, who tend not to be happy when something goes wrong with new releases – as it does all too often.

For system administrators

In order to serve their user communities, system administrators need:

Working, tested releases
- Releases should be tested across common configurations before release, including multiple versions of PHP, different types of databases, and other dependencies that MediaWiki supports.
- Upgrades should work regardless of how old the previous version is – if one wants to upgrade from 1.16 to 1.23, it should be possible to upgrade between these versions directly.
- Upgrades should not break major extensions: official Foundation extensions and popular non-WMF extensions, such as Semantic MediaWiki and SocialProfile, should be tested before release, and non-functioning extensions should be clearly marked and announced.
- Any issues that do come up with a release should be taken seriously and resolved quickly, as – despite everyone's best efforts – things do go wrong sometimes.
Working, tested releases on time
- Schedules should be consistent so that site upgrades can be planned accordingly.
Working, tested releases to be announced on time
- Announcements of both release candidates and new releases should go out on all relevant channels well in advance, not just to mailing lists such as wikitech-l, mediawiki-l, mediawiki-announce and mediawiki-enterprise, but also to the main page of MediaWiki.org.
Working, tested releases to be documented before release
- Updated release documentation should be easily found on MediaWiki.org, preferably linked on the main page. The original announcement should also include the necessary information and links, and should be written to be accessible to developers, system administrators and end-users.
In-place database updates to working, tested releases
- Upgrades should not take down the entire website for the duration of the upgrade: most won't interfere with active services, and not all of our users can afford a DBA to perform schema upgrades.
- Most updates can be done in-place. Those that cannot can often be done more efficiently, such as by initially adding with nulls, and populating them with proper values later using the job queue.
- Most schema updates can be made optional, at least for a while.
Well-managed lists of known regressions
- Humans are fallible, problems are unavoidable, and each one cannot be fixed before the official release. System administrators should have the right to know if updating will break something they depend on before it breaks.

For third-party developers

MediaWiki is highly scalable and modular by design. Many third-party projects, such as wikiHow, write their own extensions and skins to provide mission-critical functionality and essential features not present in core, such as complex data handling or user IP address lookup. When these extensions break, it's a nasty situation for companies, system administrators and developers alike.

Many large MediaWiki projects are running old versions right now, some using versions that are many years old and contain publicly-disclosed security vulnerabilities that have since been fixed. In many cases, the lack of upgrades is because of custom extensions and skins that no longer work with recent releases. While not following the MediaWiki coding conventions and standards on their end can be a contributing factor, some of the problems lie in the upstream codebase as well, due to hard-to-follow release notes, lack of consistent dependency testing, and poorly executed refactorings that lead to breaking changes.

To prevent situations where the site is displaying fatals all over the place, or worse, the site never gets upgraded at all, we want to make the process as smooth as possible and encourage collaboration and cooperation between core and third party developers. Developers on all sides should encounter a straight-forward and welcoming process to provide code, receive feedback, provide suggestions and help with translations.

To begin, third-party developers would need:

Reliable release dates
Reliable announcements of release candidates and releases
Notification of release blockers
Notification of breaking changes
- The sooner developers know what they need to fix in their extensions and skins, the better. Breaking changes in core may include:
  - New deprecations
  - Final removals of deprecated functions (yes, they may have been announced in the past, but that doesn't mean everyone remembers)
  - Random refactoring (there may have been a good reason for it, but this does often break things)
  - Unexpectedly changing dependencies
- All of these are supposed to be included in release notes in some fashion already, but MediaWiki developers don't always remember to add them. The release notes themselves are not always readable or discoverable. Input from active third-party developers would lead to better, more understandable release notes.
Better extension versioning
- Creating REL_* branches of extensions in batches is useful, but doesn't mean they necessarily work with that version. Release branches should be properly verified to work before being published.

What we propose to do

Budget required	$75,000
Time required	25 hours per week
Long-term impact	Things get better and sustainable testing and announcement methods are established

We will do the following:

Consistent scheduling, with long-range announcements and plans
- Punctual releases, including security releases being made available as soon as possible
- Complete announcements to all relevant mailing lists, including a breakdown of potentially breaking changes
Thorough testing before releases: multiple PHP versions, different operating systems and databases, and against common extensions
- Bug filing and fixes
- Communication with extension developers and maintainers regarding things that will need fixing, with suggestions where practical
Making the release tarballs, with major releases going out on a six-month cycle
Security and bugfix releases as they come up
Creating basic documentation on-wiki
- Release documentation
- Guides for sysadmins and developers for updates
- Guides for sysadmins for difficult but useful extensions (such as VisualEditor)
- Documentation of what we do as release managers
Development of automated testing infrastructure
Anti-spam improvements
- Bundling anti-spam extensions with the tarball, and providing configurations for them with sensible defaults
- AbuseFilter repository with performant and functional anti-spam filters

We also plan to work on the following, but as a lower priority:

Outreach to relevant parties
- Package maintainers (including Debian, Arch and others)
- Third-party system administrators, including enterprise, volunteer projects, folks using WMF branches, folks who need security, folks still on 1.16, folks with skins that won't update, folks with horrible spam problems ...
- Non-community developers
Relevant MediaWiki development
- Improved backend and skinning support
- Extensions
  - Maintenance for unsupported but widely-used extensions
  - Better packaging and documentation for third-party users (especially for important extensions such as VisualEditor and MobileFrontend)
Improved mechanisms for keeping skins and extensions up to date and functional along with the MediaWiki installation

Fallback, or what we would do should funding implode

Budget required	$0, or $250 (enough for 5 T-shirts that say "I'm a release manager for MediaWiki and all I got was this lousy T-Shirt")
Time required	A few hours a week
Long-term impact	Minimal; things would keep happening, and nothing would really change.

This is what could be done for free by volunteers. It is how MediaWiki's release process has been handled in the past, as well as how other projects such as the Signpost continue to work – reasonably consistently, but not anyone's priority.

The following are included:

Eventually making the release tarball and signing it, with releases going out on the current schedule
Making release candidates
Security and bugfix releases whenever anyone gets around to it
Sending out the announcements

Issues:

Testing is not guaranteed
Security releases may be slow
Announced dates may not be honoured due to volunteer priorities

Who we are

Kim Schoonover (Isarra)

Kim is a system administrator, software developer and UI designer for all Uncyclomedia websites and ShoutWiki. She has developed several complex skins – including ShoutWiki's Aurora skin, Greystuff, and Splash – and UI improvements for MediaWiki, and has contributed to MediaWikiAuth extension. She led the 2012 move of Uncyclopedia's large database to another server, coordinating volunteers and developing wiki grabber scripts to successfully complete this unprecedented move. She lives in the US.

Kim's experience with herding cats makes her ideal to manage each release, making sure that testing is complete, that the community is satisfied with the release candidate and that the appropriate mailing lists are notified and websites updated. She will also handle uploading the tarball and associated tasks.

Ryan Schmidt (Skizzerz)

Ryan is an experienced MediaWiki developer and system administrator. He wrote Extension:SimpleAntiSpam and improved permission handling via $wgRevokePermissions, both of which were eventually incorporated into core. He is experienced with developing for third-party wikis running unconventional configurations: he added MS-SQL support to core (again) and has worked on many other extensions. He was one of the original staff at ShoutWiki (where he helped write the CreateWiki utility), is a system administrator for Uncyclomedia websites, and runs thetestwiki.org. He provides paid consultancy in installing and configuring MediaWiki, integrating it with custom software, and writing custom MediaWiki extensions. He is well-known in both MediaWiki developer and system administrator community, especially from his time as an administrator and active user on the mwusers.org support forum. He lives in the US.

Ryan's extensive background in databases and system architecture will put him right at home working on testing frameworks to ensure that the tarball works correctly on a variety of platforms, and that upgrades from arbitrary MediaWiki versions work without issue.

Benjamin Lees (Emufarmers)

Benjamin is a long-time MediaWiki system administrator and developer with extensive experience in supporting users of MediaWiki. Benjamin has provided support to third-party users of MediaWiki for years on the mwusers.org forum, the mediawiki-l mailing list, the MediaWiki.org support desk, and the #mediawiki IRC channel. He is the author of Extension:QuestyCaptcha, a popular CAPTCHA plugin, and is based in the US.

Benjamin's established position in the community makes him an excellent point-man in gathering and managing feedback on releases and the release process.

Jack Phoenix

Jack Phoenix is an experienced developer of MediaWiki core, extensions, and skins. He has been developing MediaWiki since 2008, is a primary maintainer for ShoutWiki and its many associated extensions, and has developed more extensions and skins both as a ShoutWiki employee and as a volunteer than even he can keep track of anymore. He successfully advocated for and assisted in the open-sourcing of ArmchairGM's codebase, regular publishing of wikiHow's source code dumps, and publishing the entirety of Wikia's codebase under a free and open source license. He leads the customer support staff at ShoutWiki, where he works closely with users and MediaWiki community managers. He has extensive experience in working with translations and lives in Finland.

Jack will continue to liaison with third party users such as Wikia and wikiHow during the release, and will work on making MediaWiki releases easier for non-WMF users to use.

Bartosz Dziewoński (Matma Rex)

Bartosz is an experienced MediaWiki core developer with +2 rights on the Gerrit repository, several hundred merged commits and several hundred more reviewed and merged. He is knowledgeable on nearly every aspect of MediaWiki, from the parser through category collations to skin handling and frontend scripting. He has contributed to previous MediaWiki releases by writing and backporting patches and triaging bugs for supported release versions. He is based in Poland.

As an experienced developer, Bartosz will work on the multitude of Gerrit and Bugzilla activities that are associated with resolving bugs and writing and reviewing patches for them.

Endorsements

I've had the pleasure with working all of the members of The Consortium on various projects such as developing anti-spam extensions, the configuration database proposal, improving skinning support, and general MediaWiki code review. I'm convinced they have the necessary skill-set to improve the MediaWiki release process. They've all had a huge impact as volunteers, I can't wait to see what they can accomplish once people start paying them. :-) Legoktm (talk) 20:41, 17 June 2014 (UTC)[reply]
Legoktm stole the words out of my mouth for the most part. Many of these people have been reliable and helpful to me as a third party MediaWiki system administrator for Brickimedia, and judging by the amount of work they've put into volunteering for years, I think they'd be a great addition to MediaWiki's release process for all of us system administrators like them. --GeorgeBarnick (talk) 21:17, 17 June 2014 (UTC)[reply]
Per Legoktm and George. Kingcjc (talk) 21:26, 17 June 2014 (UTC)[reply]
This proposal seems more focused, which is what I think Release management needs. Given that the Marks did Release management last year, I'm supportive of giving the other group a chance to have a go at it. Bawolff (talk) 22:50, 18 June 2014 (UTC)[reply]
This group has a diverse skillset, and a constant presence in all areas of the wikiverse (from the mailing lists, IRC channels, bugzilla tickets, gerrit patches, to all their onwiki contributions - both Wikimedia and other). Per Legoktm, I think they'd be able to improve the MediaWiki release process. –Quiddity (talk) 17:04, 24 June 2014 (UTC)[reply]
Also per Legoktm and George. --Krenair ^{(talk • contribs)} 15:46, 25 June 2014 (UTC)[reply]
I've worked with these people on a variety of projects throughout the years, particularly Uncyclomedia. I can attest to their professionalism, responsiveness, and expertise. This is the A-Team we want. --Electrified mocha chinchilla (talk) 20:26, 25 June 2014 (UTC)[reply]
Having worked with three of the five while at ShoutWiki, and based on what I've heard about the other two members, I'm certain that the consortium are the perfect choice to ensure third-party MediaWiki users get the best experience with MediaWiki releases. Like many MediaWiki users, it's vital for us that new releases work right out the box with the big third-party extensions and are released on time so the upgrade process can be allocated into the system administrator's schedule. With a team with a wide range of skillsets and experience hosting and developing for MediaWiki sites, I don't see them failing on their promises. On behalf of ShoutWiki, --Cook879 (talk) 01:20, 26 June 2014 (UTC)[reply]
As one of the sysadmins for Uncyclomedia, I have worked with The Consortium for two years, and have found them eager, helpful and extremely knowledgeable on MediaWiki, MediaWiki extensions, RDBMSs, and PHP in general. I endorse them for this project wholeheartedly. ~BizzeeBeever (t) 16:11, 26 June 2014 (UTC)[reply]