Reading/Web/Content Discovery Experiments
|
Content Discovery Experiments
|
Between July and December 2024, the Web team experiments with ideas focused on making it easier for readers to discover information on the wikis. This is part of our annual plan focused on content discovery for readers.
The team plans on running four experiments. This page will outline more details on each experiment, as well as our findings and timeline.
The results of the experiments will be used to identify which features we will build for the future, based on how useful they are proven to be in helping readers find useful and interesting information across the wikis.
Updates
Background
As new generations of readers come to our sites, we want to make it easy for them to learn. This includes making it easier to find the information they need across articles and pages as well as making it easier to discover new information based on their needs or interests. To do this, we hope to leverage advances in technology and build out new capabilities within our platform that will allow us to make discovery and browsing easier than it was in the past.This goal led the team to focus on more experimental work to begin with, as a way of being able to test out more features quickly. We started with the following hypothesis:
Designing and qualitatively evaluating three proofs of concept focused on building curated, personalized, and community-driven browsing and learning experiences will allow us to estimate the potential for increased reader retention (experiment 1: providing recommended content in search and article contexts, experiment 2: summarizing and simplifying article content, experiment 3: making multitasking easier on wikis)
This hypothesis focuses on identifying ideas for features or projects that would make it easier to browse and learn across the wikis on the desktop and mobile websites. Work here will include identifying and building proofs of concepts for each idea and providing the results of initial tests on the idea.
The goal is to find ideas that we think would work well at a larger scale and commit to building them and providing them to the wikis.
It’s important to note that this work begins discovery into areas that we have not yet been working on in the past such as summarizing or remixing content. Working across communities to ensure proper editing and moderation workflows for these new content types will be crucial. We expect to collaborate heavily on this work with communities as we move forward through different stages of experimentation and building.
Experiment 1: Display article recommendations in more prominent locations, search
This experiment seeks to add suggestions to the empty state of our search bar, providing readers with recommendations on pages they could read next. We would like to experiment with whether readers click on recommendations in the empty state of the search bar to see if this type of feature is helpful to readers.
In order to perform this experiment, we will be building a browser extension which injects the proposed recommendations into the search bar. We will then be tracking clicks on the provided suggestions using our existing instrumentation. We may also use the extension itself to communicate to readers when the experiment is over.
Experimental setup
A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to download a browser extension. Interested readers were be prompted to download a browser extension which added the feature into the page.
Data was collected on how often readers clicked on the suggestions generated by the extension. No user or session data was collected.
Experiment hypothesis
If we display recommendations in search, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to existing recommendation entry points (Related pages).
Results
We found that the readers interacted with the recommendations shown in the search bar at a rate higher than recommendations shown at the bottom of the page.
Report on the overall clickthrough rate (CTR) for search recommendations coming from the browser extension:
- On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
- On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
- The overall CTR across both wikis is 1.44%
Comparison to clickthrough rate for related pages on desktop, mobile:
We observed that the CTR for search recommendations is higher than that for related pages on desktop for both English Wikipedia and Spanish Wikipedia
- English Wikipedia (enwiki): 1.53% vs 0.41% (CTR of related pages on desktop, enwikivoyage), diff 1.12 pp
- Spanish Wikipedia (eswiki): 0.72% vs 0.13% (CTR of related pages on desktop, eswikivoyage), diff 0.59 pp
Detailed results are available on T374965.
Next steps
On mobile:
- Proceed with A/B test for the feature on one or more pilot wikis (pilot wiki list to be selected). Details on A/B test setup are available in phab:T378094
On desktop:
- Wait until the results of the article-level recommendations are ready and compare and proceed with the feature which shows better results.
Experiment 2: Improve article recommendations APIs
We want to compare various existing and proposed APIs recommending content independent of the user interface. This work allows us to test different recommendation methods against one another to determine which ones are the most useful for readers.
Experimental setup
A quicksurvey will be displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to take a survey. The survey will present recommendations from three different APIs. Users will be asked to submit which recommendations are most useful and interesting to them.
Experiment hypothesis
If we compare readers preferences for suggested article APIs, we will learn which API readers find most useful
Experiment 3: Display article recommendations in more prominent locations, article pages
We want to explore displaying recommendations in other parts of the page. This experiment focuses on displaying recommendations alongside article content on desktop.
Experimental setup
The experiment code will be loaded in the Wikipedia Recommendations browser extension. Users of the browser extension will see the recommendations immediately.
Experiment hypothesis
If we display recommendations alongside article pages, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to, or higher than, existing recommendation entry points (Related pages, Experiment 1).
Results
- On English wikipedia (enwiki) desktop, the overall CTR for article recommendations is 0.54%
- On Spanish wikipedia (eswiki) desktop, the overall CTR for article recommendations is 0.28%
We observed that the CTR for sidebar article recommendations is higher than that for related pages on desktop (listed at the bottom of articles) for both English Wikipedia and Spanish Wikipedia
- English Wikipedia (enwiki): 0.41% ->0.54% , increased 0.13 pp
- Spanish Wikipedia (eswiki): 0.13% -> 0.28%, increased 0.15 pp
Hypothesis was successful:
✅ Readers will want click on recommendations on the right of the article (measured by comparing the CTR of recommendations in search with other recommendation locations - specifically related pages on desktop)
Next steps
Sine the hypothesis was successful, we would recommend to proceed with an A/B test of the feature on the desktop website
Experiment 4: Article Summaries
This experiment will look at ways we can summarize and simplify the content of the article and present these summarizations/simplifications to readers. We believe that automatic summarization and simplification of articles will help move us towards the goal of making content easier to discover and learn from by solving existing issues with content readability and accessibility. This experiment will focus only on displaying the summaries. A future experiment will study ways of editing and adjusting this content.
In our previous research (Content Simplification) , we have identified two specific needs:
- The need for readers to quickly get an overview of a given article or page
- The need for this overview to be written in language the reader can understand
Currently, much of our quality content is long-form and thus difficult to parse in a short amount of time. In addition, it is written at a reading level much higher than that of the average adult, making it very difficult to read. Projects that simplify content, such as Simple English Wikipedia of Basque Txikipedia, explore this area already. They do this by having editors manually generate simpler versions of articles. However, these projects have so far had very limited success - they are only available in a few languages and have been difficult to scale. In addition, they ask editors to rewrite content that they have already written, which can be perceived as a very repetitive task.
We hope to use some of the explorations done by the research team (as part of hypothesis 3.1.3) in building a model which can provide these summaries to begin testing this concept with readers.
Experiment Hypothesis
By running a qualitative experiment focused on presenting article summaries to web readers, we will determine whether article summaries have the potential to increase reader retention, as proxied by clickthrough rate and usage patterns.
The motivation for this work comes from research showing that much of Wikipedia's content is difficult to read, making it less accessible to a broader audience, including the majority of adults. We would like to experiment with creating a tool that can automatically generate simplified and summarized versions of articles using simpler language.
Experimental setup
Two experiments will be run.
Userlytics experiment
The first will be a qualitative experiment run via the userlytics platform. This experiment’s focus will be on identifying overall concept matches and whether the proposed solution is clear to the user in terms of concept and usability.
Success criteria:
- Users report understanding the need for summaries
- Users report positive experiences with the user experience of the feature
Browser extension experiment
The second experiment will add the new feature into the pre-existing browser extension. This experiment will focus on gathering evidence for the following hypotheses:
Success criteria:
- Users are interested in the feature as presented (as measured through initial engagement with summary feature)
- Users report positive experiences with the user experience of the feature (as measured by the answers to the "was this useful?" question)
Note that due to limitations in our data we won’t have firm responses on either of the hypotheses. The goal of these experiments is to determine whether summaries are worth future investment for more costly experimentation.
Research
- Wikimania 2024, "Written by AI" How do editors and machines collaborate to create content. At Wikimania 2024, Wikimedians discussed ways that AI/machine-generated remixing of our existing content can be used to make Wikipedia more accessible and easier to learn from. The goal for the new features is to help a new generation of readers discover and learn from reliable, encyclopedic information on Wikipedia. The session focused on ways to give control and oversight to editors and communities over the structure and content of these new concept and feature ideas.
- Usability study on Simple Summaries. We ran an unmoderated Userlytics study on a prototype of our Simple Summary feature with 8 participants. The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions. Participants found simple summaries easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.
Timeline
| Estimated time | Planned activity |
|---|---|
| |
| Building experiments | |
| |
| Running experiment 4 |