Wikimedia Technical Documentation Team/Doc metrics
![]() | Please do not mark this page for translation yet. It is still being drafted, or it contains incomplete translation markup which should be fixed before marking for translation. |
Documentation health metrics for MediaWiki
Documentation metrics to help measure key aspects of the developer experience.
|
Measuring the health of our technical documentation can help us assess the developer experience and the state of the Wikimedia technical ecosystem. This project will identify which metrics are relevant for MediaWiki Core documentation, how we can use those metrics to drive our technical documentation work, and how they can help us identify areas for improvement.
Project overview
[edit]As part of the 24/25 annual plan, the Technical Documentation team's goal is to identify and establish metrics that measure the health of Wikimedia technical documentation, using MediaWiki Core documentation as a test case.
We want to measure doc health because documentation is a core component of the developer experience. This project seeks to explore and define:
- What questions in the field of developer/engineering workflows (see WE6) can documentation metrics help us answer?
- What type of documentation data can help enable informed decision-making about how to improve developers' experience?
One expected outcome of this work is a clearer definition and shared understanding of what it means to drive tech docs work with data in the Wikimedia context, and how we can start to build a measurement approach that we can apply to docs in various Wikimedia technical domains (i.e. beyond MediaWiki core).
Goals
[edit]- Define an initial set of documentation health metrics for MediaWiki core technical documentation.
- Implement the metrics for a pilot subset of MediaWiki core docs.
Timeline overview
[edit]Timeframe | Phase | Description |
---|---|---|
July– | Research | Doc metrics and available data |
August– | Research | Doc collections and content scope |
September– | Design | Draft v1 metrics definitions |
September– | Design | Doc collections for pilot |
Implement | Doc collections for pilot | |
November– | Design & outreach | Community feedback on v1 metrics definitions and doc collections |
Implement | v1 metrics for pilot doc collections | |
January- | Evaluate | Assess the utility of the metrics for the collections we measured, and decide how to proceed or change our approach. |
Project milestones
[edit]The team plans to complete a research phase to help focus and scope our work, followed by a design and implementation phase where we generate and start using doc health metrics for a pilot collection of technical content.
Define a draft set of v1 metrics definitions for MediaWiki core technical documentation
[edit] In progress phab:T372102.
Phase | Description | Timeframe |
---|---|---|
Research | Define the dimensions of tech docs quality that are relevant for MediaWiki docs. Resolve RQ1. | By August 31 |
Research | Identify data signals that could help us measure those dimensions. Resolve RQ2. | By August 31 |
Research & Design | Map available data signals to quality dimensions. Identify signals that may require support from other teams, or extra work to implement. | By September 30 |
Design | Decide on and define a set of v1 metrics to propose for implementation. | By October 15 |
Identify doc collections to measure
[edit]Phase | Description | Timeframe |
---|---|---|
Research | Define the scope of what to measure. Resolve RQ3. | By September 30 |
Design | Use either developer workflows, MediaWiki core code structure, or some other logical framework to identify several content collections to measure. | By October 15 |
Implement | Use PagePile or other mechanism to create collections. | By October 30 |
Get feedback and community input on v1 metrics and doc collections
[edit]Phase | Description | Timeframe |
---|---|---|
Outreach | Present v1 metrics proposal for community feedback. Explain the concept of collections, why we're using it, and get feedback on the collection concepts for the metrics pilot. | November |
Outreach | Meet with stakeholders | November |
Design | Respond to comments and iterate on metrics definitions based on feedback. | November-December |
Implement v1 metrics for MediaWiki core tech docs
[edit]Phase | Description | Timeframe |
---|---|---|
Design | Finalize and publish the set of v1 metrics definitions, and the documentation collections we will use to test them. | By December 20 |
Implement | Implement the metrics as defined in the final v1 proposal, and deploy them for the pilot doc collections. May include: Experiment with implementing new data signals, like technical documentation readability scores, and/or updates to Tech Docs dashboard. | January 2025 |
Evaluate and plan next steps
[edit]Phase | Description | Timeframe |
---|---|---|
Evaluate | Assess project outcomes based on the questions defined in Project evaluation. | February 2025 |
Evaluate / Outreach | Get feedback from community and stakeholders about project outcomes and utility of the metrics. | February 2025 |
Plan | Determine how to iterate on and/or scale metrics based on what we learned. Update documentation for any processes that should now use new documentation metrics. | February-March 2025 |
Project phases
[edit]Research phase
[edit]Measuring the health of technical documentation is not a new problem, but measuring it in the Wikimedia context comes with some additional challenges and nuances. Wikimedia projects, the MediaWiki software, and our doc publishing platforms all have unique qualities that call into question the utility of standard tech docs or wiki assessment techniques, even if we focus just on MediaWiki core software documentation.
Research questions
[edit]How do we define tech doc health for MediaWiki core?
- Define what technical documentation health means in the Wikimedia context, for the MediaWiki core software.
- Identify where can we use industry-standard definitions of doc health, where can we use wiki-standard definitions of content quality, and where we we may need to diverge from both of those metrics frameworks.
What data do we have, and what do we want?
- Map existing data signals and tools to the quality dimensions we care about, and identify gaps.
- Clarify which Wikimedia analytics tools don't (yet) work, or aren't available, for our technical wikis.
What pieces of content are meaningful to measure?
- Identify the scope of "MediaWiki Core" documentation."
- Identify collections that can serve as pilot groupings of content for measurement.
- Define content, or characteristics of content, that we could measure but can't really impact...so we can save time and effort by realizing that is out of scope.
Annual plan context
[edit]Quotation from the Foundation's 2024/2025 annual plan
|
---|
|
Project evaluation
[edit]We will measure the success of this project with qualitative and quantitative methods. In particular, we will be looking at:
- Did we implement a tool and/or process via which data that was not previously used is now used to identify and prioritize documentation work?
- Did we identify and/or provide data about MediaWiki Core's technical documentation quality that was not previously available?
- Are MediaWiki stakeholders better able to assess the quality of a given component of the software thanks to data about tech doc health?
- Are MediaWiki stakeholders better able to assess the developer experience for a given workflow thanks to data about tech doc health?