Wikimedia Apps/Team/Android/Rabbit Holes

As part of the Android team’s 2024/2025 Annual Plan, the Android team will explore various ways to increase reader retention through browsing and learning experiences. This project page documents the Android team’s experimentation related to the Wikimedia Foundation 2024-2025 Annual Plan, specifically the Wiki Experiences 3.1 Key Result.

Current Status

• 2024-11: sharing hypothesis and designs
• Next: community consultation and in-app feedback to test wikis and regions

Summary

This project focuses on increasing the retention of a new generation of readers on our website, allowing a new generation to build a lasting connection with Wikipedia, by exploring opportunities for readers to more easily discover and learn from content they are interested in).

We are experimenting with a series of browsing experiences to determine which we would like to scale for production use, and on which platform (web, apps, or both). We will then focus on scaling these experiments and testing their efficacy in increasing retention in production environments. Our goal by the end of the year is to launch at least two experiences on representative wikis and to accurately measure a 5% increase in reader retention for readers engaged in these experiences.

To be optimally effective at achieving this KR, we will require the ability to A/B test with logged-out users, as well as instrumentation capable of measuring reader retention.

How does this work fit into the Wikimedia Foundation's Annual Plan?

Wiki Experiences 3: Consumer experience (Reading & Media)

Under the Wikimedia Foundation's Infrastructure Goal, and within the group of objectives focused on Wiki Experiences, is an objective related to improving the experience of consumers:

Wiki Experiences 3: Consumer experience Objective: A new generation of consumers arrives at Wikipedia to discover a preferred destination for discovering, engaging, and building a lasting connection with encyclopedic content.

Wiki Experiences 3.1 (WE3.1) Key Result: Release two curated, accessible, and community-driven browsing and learning experiences to representative wikis, with the goal of increasing the logged-out reader retention of experience users by 5%.

Several Wikimedia Foundation teams are committed to working on projects under the WE3.1 Key Result: Draft Hypotheses.

Android team hypothesis	Timeline	Phabricator epic
Wiki Experiences 3.1.6: If we introduce a personalized rabbit hole feature in the Android app and recommend articles based on the types of topics and sections a user is interested in, we will learn if the feature is sticky enough to result in multi-day usage by 10% of users exposed to the experiment over a 30-day period, and a higher pageview rate than users do not engage with the feature.	Oct 1 2024 - December 31, 2024	phab:T378612

Wikimedia Foundation teams are approaching annual planning more iteratively this year, so rather than committing to larger year-long projects, this hypothesis is fairly narrow in scope. This should allow us to learn from a brief experiment and deliver value in smaller increments throughout the year, while also ensuring we have the flexibility to pivot as we learn.

Community Discussion

We have discussed the broader concept behind this project with communities as part of the WMF annual plan Product & Technology OKRs discussion, but we will also initiate a more detailed community consultation with our test wikis and regions:

• Readers in South Asia and Sub Saharan Africa

Product Requirements

• Run as ABC test
  • A group is the control
  • B group sees a recommended search query from article view in the search
  • C group receives dialog encouraging them to see and save their recommended reading list
• Interface should be clear that recommendations are based on user interest
• Constrained to Sub Saharan Africa and South Asia
• User able to provide feedback about the quality of recommendations in-app
• Recommendations should pull from Categories, Topics or MoreLike. When doing instrumentation, there must be a way for us to know which selection a user made was from which API.
• After 20 days the experiment should be removed from the app

Measurement and Results

How will we know we were successful

The team is planning to run an experiment for 20 days to evaluate our hypothesis, along with running a survey to understand user satisfaction and feedback. We will validate our hypothesis for the experiment groups with the key indicators and guardrails below.

Validation

• 10% of unique users that engage with experiment do so more than once in a 30 day period

• 5% higher internal referral clicks of recommended articles from people that engage with the experiment compared to those that did not engage with feature

• 5% higher clicks on suggestion based on search queries vs. click throughs to articles from suggested reading lists

• 5% higher clicks to view suggested reading list vs users that hit enter on search queries

• 65% or higher of feedback scores are positive

Guardrails

• 60% of feedback from in-app users are negative

• 10 community members from target group express negative sentiments

Curiosities

• Do we see a difference in the retention rate for logged in users vs logged out users

• How does the metrics from Rabbit hole compare to Recommended Content

• Do we see higher pageviews for users that engage with the feature vs those that do not

After conducting data analysis we will share the results of the experiment on this page, and in partnership with the community determine if there should be further investment and rerelease of content suggestions or if we should scale another hypothesis instead.

Updates

Weekly updates can be found on our main updates page.