Talk pages project/Usability/Analysis

This page outlines the Editing Team's proposal for how we – volunteers and staff – will assess the impact that changes to how desktop talk pages appear have on volunteers across experiences levels.

Developing a shared understanding for how we will evaluate this impact is crucial for deciding whether these changes are fit for being made available to everyone who uses article Talk: and User talk: pages.

Objectives

The changes to how talk pages appear have been designed to cause:

1. Junior Contributors to quickly recognize talk pages as places to communicate with other volunteers and locate the tools available to do so.
2. Senior Contributors to be able to quickly assess which conversations on a given talk page are worth focusing on.

Evaluation criteria

To evaluate the impact of the set of changes to how talk pages appear, and the information about the discussions contained within them, the Editing Team is interested in two categories of metrics:

1. Key performance indicators will be used to determine the success of the features and ultimately, decide whether we think the Usability Improvements, in their current state, warrant being made available by default.
2. Curiosities will be used to learn what – if any – other effects the Usability Improvements caused. This information will not be used to determine the success of the features. Rather, this information will be used to inform our broader understanding of these interventions and potential opportunities for further improvement in the future.

Metrics

Type	Hypothesis	ID	Metric
KPIs	Volunteers across experience levels and account states (logged in + out) will use talk pages in ways that align with their purpose (read: to collaborate with other editors).	1.	Proportion of published talk page edits, of all types, that are reverted within 48 hours
		2.	Proportion of talk page edits that are started and successfully published
		3.	Proportion of people who publish talk page edits and are subsequently blocked
Curiosities	Junior Contributors will intuitively understand talk pages as places where they can communicate with other volunteer.	4.	Average time duration between Junior Contributors visiting a talk page and engaging with the page in some way
		5.	Proportion of Junior Contributors who visit a talk page and engage with it in some way
	A greater percentage of Junior Contributors who visit talk pages will publish at least one non-reverted comment or new discussion.	6.	Proportion of Junior Contributors who visit a talk page and successfully publish at least one comment or new discussion (not reverted)
	Senior Contributors will be able to more quickly and easily decide where to focus their attention when arriving on a talk page.	7.	Average time between a Senior Contributor visiting a talk page and starting any new edit on a talk page

See T302358 for more information about the metrics above.

Experiment plan

To evaluate the collective impact of the set of changes that comprise the Talk pages project's Usability Improvements, the Editing Team is planning to do the following.

Phase	Objective	Description	Wikis included	Notes	Status
1.	Experienced volunteers are aligned on what information we will use to evaluate the impact of the suite of Usability Improvements and ultimately, to decide whether the changes, in their current state, are fit for wide-scale deployment.	Invite volunteers at wikis across the movement to review and share feedback about this proposal.	All wikis receiving Tech News	T332844	Done
2.	Decide what – if any – changes ought to be made to the set of Usability Improvements before running an A/B test in T302358 to evaluate their impact.	Analyze how/if the identified metrics were impacted after the set of Usability Improvements (desktop) being made available at ar.wiki, cs.wiki, and hu.wiki by default.	ar.wiki, cs.wiki, and hu.wiki	T302359	Done
3.	Decide whether the set of Usability Improvements (desktop) are fit to be made available to everyone, at all Wikimedia wikis, by default.	Run an A/B test at a yet-to-be-defined set of Wikipedias to learn what – if any – negative impact the set of Usability Improvements causes.	azwiki, bnwiki, dewiki, fawiki, hewiki, hiwiki, idwiki, kowiki, nlwiki, ptwiki, rowiki, thwiki, trwiki, ukwiki, viwiki	T302358	Done
4.	Ensure the set of Usability Improvements do not introduce regressions on talk pages at the largest, and most visible, Wikipedias.	Through a series of successive deployments, make each of the features that comprise the set of Usability Improvements available for everyone (by default). After each deployment, verify no significant regressions in key talk page editing metrics (same as those used in T302358)	Largest Wikipedias (e.g., English); exact list TBD	TICKET NEEDED

Feature overview

Usability Improvements is made up of four distinct changes to how talk pages appear.

1. Headings appear in a sans serif font and gain three pieces of metadata about the discussions contained within them:
   1. the date the last comment was published in the discussion,
   2. the number of comments within the discussion, and
   3. the total number of unique people who have published a comment in the discussion.
2. Table of contents includes information about the total number of comments within each discussion on the page. (This change only affects Vector 2022.)
3. Buttons for replying and starting a new topic appear in a bolder type face.
4. Breadcrumb is added to the top of talk pages that includes when the last comment was published on the page, the user name of the person who published that comment, and the discussion that comment was published within.

The four changes described above were designed together to have the impact described in the Objectives section above.

Motivation

For wikitext talk pages to be valuable, Junior Contributors need to intuitively recognize them as places to communicate with other volunteers, and Senior Contributors need to be able to spend minimal effort understanding the conversations happening on a given page.

The trouble is, volunteers across experience levels report that the current presentation of wikitext talk pages can make recognizing talk pages, understanding the conversations happening within them, and identifying what they need to click/tap to participate in these conversations unnecessarily difficult.

Specifically, user feedback, usability tests, the Talk Page Consultation (see: #Newcomers, #Confusion, and #Design), and academic research have highlighted the difficulties people have reading and using talk pages.

By making the components that comprise talk page conversations (e.g., conversation topics, conversation boundaries, comments, etc.) and the tools necessary for participating in them easier for people to parse, we are striving to make it easier for volunteers, across experience levels, to communicate on-wiki.

More information about the research that motivated this work can be found on the project page.