Over the first half of the 2024/2025 fiscal year, the web team plans on experimenting with a few feature ideas focused on making it easier for readers to discover information on the wikis, as part of our annual plan focus on content discovery for readers.

The team plans on running four experiments. This page will outline more details on each experiment, as well as our findings and timeline.

The results of the experiments will be used to identify which features we will build for the future, based on how useful they are proven to be in helping readers find useful and interesting information across the wikis.

Background

As new generations of readers come to our sites, we want to make it easy for them to learn. This includes making it easier to find the information they need across articles and pages as well as making it easier to discover new information based on their needs or interests. To do this, we hope to leverage advances in technology and build out new capabilities within our platform that will allow us to make discovery and browsing easier than it was in the past.This goal led the team to focus on more experimental work to begin with, as a way of being able to test out more features in a quick manner. We started with the following hypothesis:

Designing and qualitatively evaluating three proofs of concept focused on building curated, personalized, and community-driven browsing and learning experiences will allow us to estimate the potential for increased reader retention (experiment 1: providing recommended content in search and article contexts, experiment 2: summarizing and simplifying article content, experiment 3: making multitasking easier on wikis)

This hypothesis focuses on identifying ideas for features or projects that would make it easier to browse and learn across the wikis on the desktop and mobile websites. Work here will include identifying and building proofs of concepts for each idea and providing the results of initial tests on the idea.

The goal is to find ideas that we think would work well at a larger scale and commit to building them and providing them to the wikis.

It’s important to note that this work begins discovery into areas that we have not yet been working on in the past such as summarizing or remixing content. Working across communities to ensure proper editing and moderation workflows for these new content types will be crucial. We expect to collaborate heavily on this work with communities as we move forward through different stages of experimentation and building.

Experiment 1: Display article recommendations in more prominent locations, search

This experiment seeks to add suggestions to the empty state of our search bar, providing readers with recommendations on pages they could read next. We would like to experiment with whether readers click on recommendations in the empty state of the search bar to see if this type of feature is helpful to readers.

In order to perform this experiment, we will be building a browser extension which injects the proposed recommendations into the search bar. We will then be tracking clicks on the provided suggestions using our existing instrumentation. We may also use the extension itself to communicate to readers when the experiment is over.

Experimental setup

A quicksurvey will be displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to download a browser extension. Interested readers will be prompted to download a browser extension which adds the feature into the page.

Data will be collected on how often readers click on the suggestions generated by the extension. No user or session data will be collected.

Experiment hypothesis

If we display recommendations in search, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to existing recommendation entry points (Related pages)

Design

Phabricator tracking:

[EPIC] Test UI of recommendations in the empty state of search and compare to other locations

Experiment 2: Improve article recommendations APIs

We would like to run an experiment that compares various existing and proposed APIs recommending content independent of the user interface. This work will allows us to test different recommendation methods against one another to determine which ones are the most useful for readers.

Experimental setup

A quicksurvey will be displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to take a survey. The survey will present recommendations from three different APIs. Users will be asked to submit which recommendations are most useful and interesting to them.

Experiment hypothesis

If we compare readers preferences for suggested article APIs, we will learn which API readers find most useful

Phabricator tracking:

[EPIC] Run UI-less experiment comparing API recommendation quality

Experiment 3: Display article recommendations in more prominent locations, article pages

In addition to search, we would like to explore displaying recommendations in other parts of the page. This experiment will focus on displaying recommendations alongside article content on desktop.

Experimental setup

The experiment code will be loaded in the Wikipedia Recommendations browser extension. Users of the browser extension will see the recommendations immediately.

Experiment hypothesis

If we display recommendations alongside article pages, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to, or higher than, existing recommendation entry points (Related pages, Experiment 1)

Design

Phabricator tracking:

Build second recommendation experiment within browser extension

Experiment 4:

We would like to experiment with presenting ways for readers to access summarized or simplified versions of our content to improve the readability of our content.

The motivation for this work comes from research showing that much of Wikipedia's content is difficult to read, making it less accessible to a broader audience, including the majority of adults. We would like to experiment with creating a tool that can automatically generate simplified and summarized versions of articles using simpler language.

Experimental setup

TBD

Design

TBD

Phabricator tracking:

[EPIC] Test Summarization of Article Content

Timeline

July 2024:

Technical Research and Investigation
Design discussions and research

August 2024:

Building experiments

September 2024:

Running experiments 1-3
Building experiment 4

October 2024:

Running experiment 4