MediaWiki Product Insights/Contributor retention and growth

Overview

Reports

Artifacts

Contributor Retention & Growth

How can we enable more teams to know MediaWiki and work effectively in MW core domains relevant to their work? How can we retain and grow the number of people willing and able to contribute to MediaWiki? What do we need to do to balance initiatives with our existing capacity to provide code review, consultancy and mentorship? Which areas and types of tasks should we focus on? How do we generate a lasting effect that scales for our needs?

These are some of the key questions behind the metric to increase by 20% the number of authors that have committed more than 5 patches across a specific set of MediaWiki repositories that are deployed to production. (WMF Annual Plan 2023/24, WE3.2).

The situation (a few frequently mentioned challenges)

• The number of contributors to MediaWiki core and other MW repositories in WM production has been stagnating for years. This needs to change to be able to sustainably maintain and evolve the software.
• MediaWiki is a large, complex and entangled code base. A change to core can have significant effects. This leads to a situation where teams and people may be hesitant to touch MediaWiki and need substantial empowerment, guidance and support.
• We depend on a few individuals who have deep knowledge about MediaWiki. This creates bottlenecks when changes to MW core are needed, may block teams' work, and may lead to workarounds that add technical debt.
• The number of people who know MediaWiki determines our capacity to provide guidance, mentor and onboard people into MediaWiki.
• Volunteers and staff have called out the need for allocating time for code review. However, time for code review is not only a matter of allocating resources, but is likewise constrained by the number of people who feel confident reviewing and merging patches in a given area.

These are some of the reasons that led to the conclusion that a key focus of the work in FY 23/24 needs to be on enabling more people to know MediaWiki and contribute effectively. We're measuring this with one number as this presents a way to focus, intentionally design interventions, and track progress. However, we need a multi-faceted, intentional approach to create lasting impact.

Work in this field is also interconnected with questions of stewardship and the development of a product strategy for MediaWiki, which is likewise a focus in this fiscal year.

Approach and focus areas

At the high level, we're focusing on these three areas:

1. Design low-entry-level initiatives for contributions to MediaWiki core and a set of extensions, enriched by workshops, improved documentation, code review and support to enable more people to feel confident contributing to MediaWiki.
2. Provide consultancy for Wikimedia’s teams in projects that touch MediaWiki core to help sustain current contributors and to enable more people to contribute to MediaWiki effectively.
3. Invest in code improvements, logging, monitoring and test coverage in specific areas to make it easier to debug issues and ease onboarding in MediaWiki.

The first two areas are fields where we are able to directly see the effect in the contribution data and can think of ways to course-correct if we don't see the desired impact. The third area encompasses work where the impact will become more visible over time, may not directly translate in a measurable increase of contributors, but may result in better "anecdotal data". This is work that can be measured via qualitative feedback over time or via indicators such as test coverage.

In addition to our 20% growth metric, we are also monitoring changes to other developer productivity data such as time to first review, number of patches and total number of contributors. These data points help us evaluate whether the overall development process around MediaWiki is improving.

Baselines and progress

The baseline for the 20% increase is the number of contributors who have submitted >5 patches in FY 22/23 (July 2022-June 2023) to the selected MW repository in Wikimedia production. This allows us to start with a relatively similar contributor base, directly monitor the impact that investments, initiatives and interventions may bring, and keep an eye on seasonal differences that may be unrelated to the efforts we're making (i.e. contribution data in December may be generally low/high).

• Data source
• Guidance on how to use these dashboards: Community_metrics.

MediaWiki core

MediaWiki core is the main code base we're focusing on over the course of this fiscal year (July - June).

• Baseline: 70 people have submitted >5 patches to MediaWiki core between July 2022 and June 2023.
• Goal: 84 people have submitted >5 patches to MediaWiki core between July 2023 and June 2024 (20% increase)
• Status by the end of Q1 (July-September 2023): 37. This is about a 15% increase compared to July-September 2022.
• Status by the end of Q2 (October-December 2023): 58. This is about a 20% increase compared to July-December 2022.
• Status by the end of Q3 (January-March 2024): 71. This is a 14.5% increase compared to July 2022-March 2023.
• Status by the end of May, 2024: 86. This is a 22.86% increase from the baseline!  Done

Focus within MediaWiki core primarily lies on

• areas where staff from the MediaWiki Engineering group onboard to as part of the teams' stewardship responsibility
• areas relevant for other teams as they are working on other annual plan priorities or essential work projects
• areas suitable for initiatives to onboard people to MW core and share knowledge on general/specific concepts (interest/need based)

Additional metrics

You can use this dashboard to explore data prior to FY 22/23. The results below show that the number of patches and authors at the end of May 2024 (11 months timeframe) is already higher than in FY 22/23 (12 months timeframe). We will update the count for FY 23/24 after June 30th, 2024.

Time to first review (days)

• FY 22/23 (July 2022 - May 2023): median: 0.94, avg: 13.6
• FY 23/24 (July 2023 - May 2024): median: 0.65, avg: 6.57

Total number of authors

• FY 22/23 (July 2022 - May 2023): 188 (194 at the end of June)
• FY 23/24 (July 2023 - May 2024): 212

Total number of patches

• FY 22/23 (July 2022 - May 2023): 4292 (4687 at the end of June)
• FY 23/24 (July 2023 - May 2024): 5108

Other repositories

CentralAuth

We've identified one main second repository to focus on: CentralAuth, which is a highly critical extension in Wikimedia's ecosystem. The goal here is to onboard at least 1 more staff member within MW Engineering to CentralAuth, as part of the MediaWiki Engineering's steward responsibility for this extension.

• Baseline: 5 people
• Goal: 6 people
• Status by the end of Q1 (July-September 2023): 5
• Status by the end of Q2 (October-December 2023): 6  Done
• Status by the end of Q3 (January-March 2024): 10

Time to first review (days)

• FY 22/23 (July 2022 - May 2023): median: 0.68, avg: 24.71
• FY 23/24 (July 2023 - May 2024): median: 0.38, avg: 3.23

Total number of authors:

• FY 22/23 (July 2022 - May 2023): 27 (32 at the end of June)
• FY 23/24 (July 2023 - May 2024): 34

Total number of patches:

• FY 22/23 (July 2022 - May 2023): 99 (121 at the end of June)
• FY 23/24 (July 2023 - May 2024): 287

Authentication-related extensions

The MediaWiki Engineering group took on stewardship for all authentication-related extensions (supported by the Security team), therefore we monitored changes also across CentralAuth, OATHAuth, OAuth and WebAuthn, measured accumulative:

• Baseline: 10 people
• Goal: 12 people
• Status by the end of Q1 (July-September 2023): 6
• Status by the end of Q2 (October-December 2023): 9
• Status by the end of Q3 (January-March 2024): 14  Done

Time to first review (days)

• FY 22/23 (July 2022 - May 2023): median: 0.74
• FY 23/24 (July 2023 - May 2024): median: 0.41

Total number of authors:

• FY 22/23 (July 2022 - May 2023): 41 (46 at the end of June)
• FY 23/24 (July 2023 - May 2024): 42

Total number of patches:

• FY 22/23 (July 2022 - May 2023): 257 (282 at the end of June)
• FY 23/24 (July 2023 - May 2024): 542

The "principle" of selecting a repository is to approach this intentionally, focused and need based, which also helps us to balance capacity to provide mentorship and support with responsibilities to maintain and evolve the software.

Q1 focus (July - September 2023)

• Developing overall approach and focus area
• Consultancy and support for the Observability, Trust & Safety Product, Design Systems, Growth and Web teams on projects that touch MW core domains
• Code review for patches submitted by volunteers
• Investments in technical documentation (mediawiki.org project pages update)
• Code improvements and capabilities that ease onboarding in MediaWiki and make logging, monitoring, and debugging easier (Phab:T343098, Phab:T47514 and smaller improvements)

Q2 focus (October - December 2023)

MediaWiki Core concepts.
Part 1 of MediaWiki Introduction 2023.

• Plan and conduct a first MediaWiki code jam in December 2023 (the idea is to test the approach at a smaller scale/Foundation internal and if it works well, run a second, tech community-wide edition later, if people are interested)
• Consultancy, code review and support for a number of teams on projects that touch MW core domains
• Code review for patches submitted by volunteers
• Investments in technical documentation (deployment pipeline docs improvements; quick start workflow for contributors)
• New onboarding videos for MediaWiki core and extensions
• Code improvements and capabilities that ease onboarding in MediaWiki and make logging, monitoring, and debugging easier (Phab:T343098, Phab:T348206 and smaller improvements)

Q3 focus (January - March 2024)

• Evaluate Code Jam and document the outcome
• Write script to find new volunteer MediaWiki contributors who got their first patch merged in core or Wikimedia-deployed extensions and services, for inclusion in the monthly MediaWiki Insights emails
• Consultancy, code review and support for teams on projects that touch MW core domains
• Code review for patches submitted by volunteers
• Code improvements that ease onboarding in MediaWiki

Q4 focus (April - June 2024)

Wikimedia Hackathon group photo

• Prepare and conduct a MediaWiki track for the Wikimedia Hackathon
• Design new MediaWiki t-shirt as a way to recognise and thank contributors
• Consultancy, code review and support for teams on projects that touch MW core domains
• Code review for patches submitted by volunteers
• Code improvements that ease onboarding in MediaWiki