Wikimedia Release Engineering Team/Process

Who we are

Mission

We get software from the developer to production as quickly, easily, and safely as possible.

We are big fans of the values of the Agile Manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Our values

Quick

Software needs to be in front of users for it to have value. The longer code spends in a development environment, a test environment, a deployment queue, the more it costs and the less value it has. Release Engineering is in the business of making that process as fast as it can reasonably be made.

Easy

People creating software for users should have all the tools necessary right to hand, without having to struggle with tasks and chores unrelated to development. From planning tools and shared development environments and code repositories to test environments to deployment processes, Release Engineering works to make the steps from idea to users as easy and as automatic as they can be made.

Safe

Speed and ease are important, but not at the cost of quality. Release Engineering includes testing processes of all kinds, from code linters to continuous integration to automated system tests to production scripting. Furthermore, we take Quality Assurance seriously, in the sense that QA has to do with improvements to methodology and process throughout the development cycle.

Development process

	Early in the dev process --->			---> After merge to master branch --->			---> Production
	Planning & Tracking	Shared Development Environment	Continuous Integration	Beta Cluster	Browser Test Automation	Browser Test Automation	Exploratory Testing	Exploratory Testing	Deploying to Production	Management
Q/E/S	Easily	Quickly/Easily	Safely	Safely	Safely	Safely	Safely/Easily	Safely/Easily	Easily	Quickly/Easily
Person	Mukunda Modell	Dan Duvall	Antoine	Antoine	Željko	Chris	Elena	Rummana	Chad	Greg
Tools and systems	Phabricator	Vagrant, Puppet	Jenkins, Zuul	Puppet, sysadmin	page-object, Jenkins job builder	page-object, Jenkins, Sauce Labs, beta cluster	beta cluster, test2wiki	Gerrit, Phabricator, beta cluster, test2wiki and enwiki	Gerrit, scap	SWAT, logs, schedules
Activities	Maintenance, new features in response to dev teams' requests.	Maintenance, improvements to code and infrastructure.	Maintenance, new features	Maintenance, updates to improve performance and emulation of prod systems	Jenkins infrastructure, shared code, community training	Failure analysis, test design, internal training, process	Exploratory testing, reporting issues, bug triage	Exploratory testing, reporting issues, bug triage,bug verification.	Weekly deployments, SWAT when necessary	Oversee production deployments, do schedules, planning
Collaborates with	Ops, ECT	Core, Features teams, browser test	Core, Ops, Labs, Devs, Volunteers	Ops, Labs	Language, Search, Wikidata, ECT	Mobile, VE, Flow, MMV, Labs, Team practices	Editing	Editing and Mobile Apps team	Core, Ops	Everyone
Working on	Expand Phab capabilities	Vagrant in CI, more shared code	Performance, browser tests voting, CI infra	Better prod emulation, staging	Browser tests voting	Refactoring; EAL Vagrant in Jenkins plan Process work (with TeamPractices)	ET for other WMF teams; help with automated tests	ET for other WMF teams,making browser tests complete and reliable for high-priority features in VE for Q3 ,helping with bug triage	More automation	More automation, more information

Tools and Frameworks

Release Engineering oversees many of the tools and frameworks that help in the delivery of software features, from development environments and test frameworks to deployment scripts. Creating and maintaining reliable automation is a constant concern for us on every front.

Here is a list of tools and frameworks that people in Release Engineering understand and maintain. We may not be the only ones who know about each item, but we are all knowledgeable to a certain extent about all of the systems listed here.

Initial	Tools and frameworks	People with answers
B	Browser tests	Chris, Željko, Rummana
BC	Beta cluster	Chad, Mukunda, Tyler, Chris, Antoine
ET	Exploratory Testing	Rummana, Elena (Željko, Chris also)
J	Jenkins/Zuul	Antoine, Željko, Tyler
P	Puppet	Dan, Mukunda
PH	Phabricator	Mukunda
SC	SCAP	Greg, Mukunda, Chad
SL	SauceLabs	Željko, Chris
SW	SWAT	Greg
V	Vagrant	Dan

Teams and Tools

Release Engineering has direct, reciprocal relationships with at least ten other teams within WMF. Here is a table that attempts to capture the nature of those relationships in terms of the tools and frameworks that we use in common.

Team	Tools
Editing	B BC ET J PH SC SL SW V
Mobile Web/Apps	B BC ET J PH SC SL SW V
Services/Parsoid	BC J PH SW
Collaboration	B BC J PH SC SL SW V
Language	B BC J PH SC SL V
Core	BC J P PH SC SW
Multimedia	B BC J PH SC SL SW V
Engineering Community Team	B(?) BC PH
Team Practices Group	PH
Ops	BC P SC
(Fundraising)
(Zero)
(Design)

How Release Engineering Works

Release Engineering creates, manages, and supports the tools and practices that allow developers to create software easily, manage software safely, and get software to users quickly. As such, we have several input modes for work that we do:

Anticipation

When a software developer says "I would like to be able to do X", we in Release Engineering strive to be able to say in reply "OK, we already know how to do X, let us show you".

For example, no one expected that a public, shared test environment would be particularly valuable until people who are now working in Release Engineering began to make the beta cluster useful. For another example, shared development environments using Vagrant were only an experiment until people who are now working in Release Engineering made them robust and useful.

Mandate

From time to time the Foundation mandates particular practices. Automated browser tests were a mandate in early 2012. A fast deploy cycle was a mandate in early 2013. Replacing Bugzilla with Phabricator was a mandate in 2014. People in Release Engineering were critical to the success of each of these projects.

Reaction

Many of the teams we work with request improvements to the tools and systems that Release Engineering work with. We work to accommodate these requests.

Setting priorities

Clearly, Release Engineering priorities can undergo large shifts in short amounts of time. In general, we prioritize reactions over mandates over anticipation. We serve our colleagues first.

But we must also always balance ease, safety, and speed. The easiest thing to do is not always safe or fast, and negotiating the work we do in the context of the values we share occurs daily, weekly, monthly, and quarterly amongst the members of the Release Engineering team, and with the teams we serve.

We have adopted a number of practices to serve this negotiation:

well attended weekly team meeting for all of Release Engineering
Roadmap and Deployments meeting for senior WMF staff
participating in Scrum of Scrums
institutional pair-programming sessions among RelEng team people
institutional pair-programming sessions for RelEng people and colleagues in other teams
- Željko Filipin and others
training sessions and public presentations
significant contributions to code repositories across WMF, from extensions to core to operations
significant interaction with a volunteer community who share our interests, including past projects with Google Summer of Code and Outreachy
close coordination with the Operations team, on whom we depend for production services
close coordination with Team Practices Group, with whom we share a common interest in cross-team process, practice, and methodology