MediaWiki Product Insights/Contributor retention and growth

How can we enable more teams to know MediaWiki and work effectively in MW core domains relevant to their work? How can we retain and grow the number of people willing and able to contribute to MediaWiki? What do we need to do to balance initiatives with our existing capacity to provide code review, consultancy and mentorship? Which areas and types of tasks should we focus on? How do we generate a lasting effect that scales for our needs?

These are some of the key questions behind the metric to increase by 20% the number of authors that have committed more than 5 patches across a specific set of MediaWiki repositories that are deployed to production. (WMF Annual Plan 2023/24, WE3.2).

The situation (a few frequently mentioned challenges)

The number of contributors to MediaWiki core and other MW repositories in WM production has been stagnating for years. This needs to change to be able to sustainably maintain and evolve the software.
MediaWiki is a large, complex and entangled code base. A change to core can have significant effects. This leads to a situation where teams and people may be hesitant to touch MediaWiki and need substantial empowerment, guidance and support.
We depend on a few individuals who have deep knowledge about MediaWiki. This creates bottlenecks when changes to MW core are needed, may block teams' work, and may lead to workarounds that add technical debt.
The number of people who know MediaWiki determines our capacity to provide guidance, mentor and onboard people into MediaWiki.
Volunteers and staff have called out the need for allocating time for code review. However, time for code review is not only a matter of allocating resources, but is likewise constrained by the number of people who feel confident reviewing and merging patches in a given area.

These are some of the reasons that led to the conclusion that a key focus of the work in FY 23/24 needs to be on enabling more people to know MediaWiki and contribute effectively. We're measuring this with one number as this presents a way to focus, intentionally design interventions, and track progress. However, we need a multi-faceted, intentional approach to create lasting impact.

Work in this field is also interconnected with questions of stewardship and the development of a product strategy for MediaWiki, which is likewise a focus in this fiscal year.

Approach and focus areas

At the high level, we're focusing on these three areas:

Design low-entry-level initiatives for contributions to MediaWiki core and a set of extensions, enriched by workshops, improved documentation, code review and support to enable more people to feel confident contributing to MediaWiki.
Provide consultancy for Wikimedia’s teams in projects that touch MediaWiki core to help sustain current contributors and to enable more people to contribute to MediaWiki effectively.
Invest in code improvements, logging, monitoring and test coverage in specific areas to make it easier to debug issues and ease onboarding in MediaWiki.

The first two areas are fields where we are able to directly see the effect in the contribution data and can think of ways to course-correct if we don't see the desired impact. The third area encompasses work where the impact will become more visible over time, may not directly translate in a measurable increase of contributors, but may result in better "anecdotal data". This is work that can be measured via qualitative feedback over time or via indicators such as test coverage.

In addition to our 20% growth metric, we are also monitoring changes to other developer productivity data such as time to first review, number of patches and total number of contributors. These data points help us evaluate whether the overall development process around MediaWiki is improving.

Baselines and progress

The baseline for the 20% increase is the number of authors who have submitted more than 5 changesets to the selected MediaWiki repository in FY 22/23 (July 2022-June 2023). This allows us to start with a relatively similar contributor base, directly monitor the impact that investments, initiatives and interventions may bring, and keep an eye on seasonal differences that may be unrelated to the efforts we're making (i.e. contribution data in December may be generally low/high).

Data source
Guidance on how to use these dashboards: Community_metrics.

MediaWiki core

MediaWiki core is the main code base we're focusing on over the course of this fiscal year (July - June).

Baseline: 70 authors
Goal: 84 authors (20% increase)
Status by the end of Q1 (July-September 2023): 37. This is about a 15% increase compared to July-September 2022.
Status by the end of Q2 (October-December 2023): 58. This is about a 20% increase compared to July-December 2022.
Status by the end of Q3 (January-March 2024): 71. This is a 14.5% increase compared to July 2022-March 2023.
Status by the end of May, 2024: 86. This is a 22.86% increase from the baseline! Done
Status by the end of Q4 (April-June 2024): 88. This is a 25.71% increase from the baseline.

Focus within MediaWiki core primarily lies on:

Areas where staff from the MediaWiki Engineering group onboard to as part of the teams' stewardship responsibility
Areas relevant for other teams as they are working on other annual plan priorities or essential work projects
Areas suitable for initiatives to onboard people to MW core and share knowledge on general/specific concepts (interest/need based)

Additional metrics

You can use this dashboard to explore data prior to FY 22/23.

Time to first review (days)

FY 22/23 (July 2022 - June 2023): median: 0.94, avg: 13.4
FY 23/24 (July 2023 - June 2024): median: 0.64, avg: 6.62

The median time to first review decreased by 31.91%, the average time by 50,60%.

Total number of authors

FY 22/23 (July 2022 - June 2023): 194
FY 23/24 (July 2023 - June 2024): 216

The total number of authors increased by 11.34%.

Number of changesets

FY 22/23 (July 2022 - June 2023): 4687
FY 23/24 (July 2023 - June 2024): 5494

The number of changesets increased by 17.22%.

Other repositories

CentralAuth

We've identified one main second repository to focus on: CentralAuth, which is a highly critical extension in Wikimedia's ecosystem. The goal here was to onboard at least 1 more staff member within MW Engineering to CentralAuth, as part of the MediaWiki Engineering's steward responsibility for this extension.

Baseline: 5 authors
Goal: 6 authors
Status by the end of Q1 (July-September 2023): 5
Status by the end of Q2 (October-December 2023): 6 Done
Status by the end of Q3 (January-March 2024): 10
Status by the end of Q4 (April-June 2024): 15 (200% increase from baseline)

Time to first review (days)

FY 22/23 (July 2022 - June 2023): median: 0.56, avg: 20.37
FY 23/24 (July 2023 - June 2024): median: 0.38, avg: 3.43

The median time to first review decreased by 32.14%, the average time by 83.16%.

Total number of authors:

FY 22/23 (July 2022 - June 2023): 32
FY 23/24 (July 2023 - June 2024): 38

The total number of authors increased by 18.75%.

Number of changesets:

FY 22/23 (July 2022 - June 2023): 121
FY 23/24 (July 2023 - June 2024): 326

The number of changesets increased by 169.42%.

Authentication-related extensions

The MediaWiki Engineering group took on stewardship for all authentication-related extensions (supported by the Security team), therefore we monitored changes also across CentralAuth, OATHAuth, OAuth and WebAuthn, measured accumulative:

Baseline: 10 authors
Goal: 12 authors
Status by the end of Q1 (July-September 2023): 6
Status by the end of Q2 (October-December 2023): 9
Status by the end of Q3 (January-March 2024): 14 Done
Status by the end of Q4 (April-June 2024): 20 (100% increase from baseline)

Time to first review (days)

FY 22/23 (July 2022 - June 2023): median: 0.69
FY 23/24 (July 2023 - June 2024): median: 0.41

The median time to first review decreased by 40.58% across these extensions.

Total number of authors:

FY 22/23 (July 2022 - June 2023): 46
FY 23/24 (July 2023 - June 2024): 46

The total number of authors remained the same.

Number of changesets:

FY 22/23 (July 2022 - June 2023): 282
FY 23/24 (July 2023 - June 2024): 590

The number of changesets across authentication-related extensions increased by 109,22%.

The "principle" of selecting a repository is to approach this intentionally, focused and need based, which also helps us to balance capacity to provide mentorship and support with responsibilities to maintain and evolve the software.

Q1 focus (July - September 2023)

Developing overall approach and focus area
Consultancy and support for the Observability, Trust & Safety Product, Design Systems, Growth and Web teams on projects that touch MW core domains
Code review for patches submitted by volunteers
Investments in technical documentation (mediawiki.org project pages update)
Code improvements and capabilities that ease onboarding in MediaWiki and make logging, monitoring, and debugging easier (Phab:T343098, Phab:T47514 and smaller improvements)

Q2 focus (October - December 2023)

MediaWiki Core concepts.
Part 1 of MediaWiki Introduction 2023.

Plan and conduct a first MediaWiki code jam in December 2023 (the idea is to test the approach at a smaller scale/Foundation internal and if it works well, run a second, tech community-wide edition later, if people are interested)
Consultancy, code review and support for a number of teams on projects that touch MW core domains
Code review for patches submitted by volunteers
Investments in technical documentation (deployment pipeline docs improvements; quick start workflow for contributors)
New onboarding videos for MediaWiki core and extensions
Code improvements and capabilities that ease onboarding in MediaWiki and make logging, monitoring, and debugging easier (Phab:T343098, Phab:T348206 and smaller improvements)

Q3 focus (January - March 2024)

Evaluate Code Jam and document the outcome
Write script to find new volunteer MediaWiki contributors who got their first patch merged in core or Wikimedia-deployed extensions and services, for inclusion in the monthly MediaWiki Insights emails
Consultancy, code review and support for teams on projects that touch MW core domains
Code review for patches submitted by volunteers
Code improvements that ease onboarding in MediaWiki

Q4 focus (April - June 2024)

Prepare and conduct a MediaWiki track for the Wikimedia Hackathon
Design new MediaWiki t-shirt as a way to recognise and thank contributors
Consultancy, code review and support for teams on projects that touch MW core domains
Code review for patches submitted by volunteers
Code improvements that ease onboarding in MediaWiki