Wikimedia Technical Documentation Team/Doc metrics

Start:	2024-07
Team members:	Technical Documentation Team
Lead:	Tricia Burmeister

Measuring the health of our technical documentation can help us assess the developer experience and the state of the Wikimedia technical ecosystem. This project will identify which metrics are relevant for MediaWiki documentation, how we can use those metrics to drive our technical documentation work, and how they can help us identify areas for improvement.

Project overview

As part of the 24/25 annual plan, the Technical Documentation team's goal is to identify and establish metrics that measure the health of Wikimedia technical documentation, using MediaWiki Core documentation as a test case.

We want to measure doc health because documentation is a core component of the developer experience. This project seeks to explore and define:

What questions in the field of developer/engineering workflows (see WE6) can documentation metrics help us answer?
What type of documentation data can help enable informed decision-making about how to improve developers' experience?

One expected outcome of this work is a clearer definition and shared understanding of what it means to drive tech docs work with data in the Wikimedia context, and how we can start to build a measurement approach that we can apply to docs in various Wikimedia technical domains (i.e. beyond MediaWiki core).

Goals

Define an initial set of documentation health metrics for MediaWiki core technical documentation.
Implement the metrics for a pilot subset of MediaWiki core docs.

Timeline overview

Timeframe	Phase	Description
July–September 2024	Research	Doc metrics and available data
August–September 2024	Design	Draft v0 metrics definitions
September–October 2024	Research/design	Doc collections and content scope
September–October 2024	Implement	Doc collections for test
November 2024–December 2024	Test	Generate test dataset to measure doc collections for data elements in v0 metrics
January 2025	Evaluate	Analyze test data and assess the test data elements as indicators for v0 metrics
January 2025	Decide	Propose a metrics implementation based on conclusions from analysis

Project milestones

The team plans to complete a research phase to help focus and scope our work, followed by a design and implementation phase where we generate and start using doc health metrics for a pilot collection of technical content.

Define a draft set of v0 metrics definitions for MediaWiki core technical documentation

Done phab:T372102.

Phase	Description	Timeframe
Research	Defined the dimensions of tech docs quality that are relevant for MediaWiki docs. Resolved RQ1.	By August 31
Research	Identified data signals that could help us measure those dimensions. Resolved RQ2.	By August 31
Research & Design	Mapped available data signals to quality dimensions. Identified signals that may require support from other teams, or extra work to implement.	By September 15
Design	Decided on and defined a set of v0 proposed metrics to explore with sample doc collections.	By September 15

Identify doc collections to measure

Done phab:T374722.

Phase	Description	Timeframe
Research	Define the scope of what to measure. Resolve RQ3.	By October 15
Design	Use either developer workflows, MediaWiki core code structure, or some other logical framework to identify several content collections to measure.	By October 30
Implement	Use PagePile or other mechanism to create collections.	By October 30

Test v0 metrics on doc collections

Done phab:T379431

Phase	Description	Timeframe
Implement/Test	Implement testing process and scoring rubric for test data to generate v0 metrics.	By Nov 15
Implement/Test	Generate test dataset using subset of data elements aligned with v0 metrics categories, for 5 sample doc collections (140 docs).	By Dec 30

Analyze test data and v0 metrics outputs

In progress phab:T382606

Phase	Description	Timeframe
Analyze	Use the scoring rubric to generate metrics scores for v0 metrics categories using test dataset.	By Jan 15
Evaluate	Publish a summary and evaluation of the testing process and its outcomes.	By Jan 30

Get feedback and community input on v0 metrics and doc collections

Phase	Description	Timeframe
Outreach	Present v0 metrics proposal for community feedback. Explain the concept of collections, why we're using it, and get feedback on the collection concepts for the metrics test.	February
Outreach	Meet with stakeholders	February
Design	Respond to comments and iterate on metrics definitions based on feedback.	February-March

Evaluate and iterate on metrics design

Phase	Description	Timeframe
Evaluate	Assess metrics and outcomes based on the questions defined in Project evaluation.	February-March 2025
Design	Define the v1 set of metrics to implement. Design the implementation of those metrics (which will be a more robust and scaleable implementation than used for the v0 testing).	February-March 2025

Implement v1 metrics for MediaWiki core tech docs

Phase	Description	Timeframe
Implement	Implement the metrics as defined in the final v1 proposal, and deploy them. May include: Experiment with implementing new data signals, like technical documentation readability scores, and/or updates to Tech Docs dashboard.	March 2025

Project phases

Research phase

Measuring the health of technical documentation is not a new problem, but measuring it in the Wikimedia context comes with some additional challenges and nuances. Wikimedia projects, the MediaWiki software, and our doc publishing platforms all have unique qualities that call into question the utility of standard tech docs or wiki assessment techniques, even if we focus just on MediaWiki core software documentation.

Research questions:

Design phase

Define criteria and requirements, analyzed data, and defined an initial set of metrics to test as part of our iterative design process:

Annual plan context

Quotation from the Foundation's 2024/2025 annual plan

Objective: Technical staff and volunteer developers have the tools they need to effectively support the Wikimedia projects. We will continue work started to improve (and scale) development, testing and deployment workflows in Wikimedia Production and expand the definition to include services for tool developers. We also aim to improve our ability to answer frequently asked questions in the field of developer/engineering workflows and audiences and make relevant data accessible to enable informed decision making. Part of this work is to look at practices (or lack of such) that currently present a challenge for our ecosystem.
WE 6.1 will "Resolve 5 questions to enable efficiency and informed decisions on developer and engineering workflows and services and make relevant data accessible by the end of Q4." The overarching goal of this work is to reduce the time, workarounds and effort it takes to gain insights in key aspects of the developer experience and enable us to make improvements to engineering and developer workflows and services.
—Wikimedia Foundation, Annual Plan/2024-2025. Wiki Experiences: Knowledge platform II (Developer Services). Objective 6.1

Project evaluation

We will measure the success of this project with qualitative and quantitative methods. In particular, we will be looking at:

Did we implement a tool and/or process via which data that was not previously used is now used to identify and prioritize documentation work?
Did we identify and/or provide data about MediaWiki Core's technical documentation quality that was not previously available?
Are MediaWiki stakeholders better able to assess the quality of a given component of the software thanks to data about tech doc health?
Are MediaWiki stakeholders better able to assess the developer experience for a given workflow thanks to data about tech doc health?