Wikipedia Subscription

Group:	Inuka team
Start:	September, 2023
Team members:	Runa Bhattacharjee; Stephane Bisson; Mary Munyoki; Eduardo Medina; Huei Tan; Sudhanshu Gautam; Uzoma Ozurumba; Eli Asikin-Garmager; Connie Chen;
Lead:	Purity Waigi (Product Manager)

Wikipedia Subscription is a test project to validate or invalidate a hypothesis that if we disseminated Wikipedia content through chat/messaging platforms, we would be able to directly reach and engage youth audiences with our content.

Background

One of the Wikimedia Foundation product and tech departments identified areas of work in 2023-24 is the Future Audiences objectives and key results. The Future audience bucket will explore ways for the movement to become the essential infrastructure of the ecosystem of free knowledge by making knowledge available to everyone wherever they are on the internet.

The above work aims to reach the global youth who consume information on other platforms with our content to increase their awareness and engagement with our projects (Future Audience KR 2.1).

The Wikipedia Subscription aligns with Future Audience KR 2.1 and will validate or invalidate one of the hypotheses in the Inuka team's 2023-24 annual plan.

“If we offered a wikipedia subscription service through a chat-based platforms (WhatsApp, Telegram etc), that sends daily reading recommendations in the form of articles/ summaries to younger readers, we would be able to test whether this type of experience with Wikipedia content would increase engagement. Reading recommendations would be sent directly to a group of readers to gauge their preferences by topic, platform, time of delivery, content formats etc. The impact of this test would be measured by how readers interact with the service from the number of opt-ins, recommendation preferences, CTR and subsequent page views.”

Rationale for Wikipedia Subscription

Readers are discovering content/knowledge “serendipitously” either through:

predictive & personalised recommendations running on socials & 3rd party platforms.
other reading patterns such as saved/ curated lists in the form of reading lists, playlists, highlights, guides etc.
creator/publisher-driven approaches that require subscribing to newsletters, blogs, pages or channels.

We also identified some recurring opportunities from previous research projects that indicate a need for personalization or predictive content by readers.

PROJECTS	THEMES
Journey transitions Research Project: Current experience of readers with Wikipedia is described as Transactional.	Reader Engagement.
Reading Wikipedia Research Project: The majority of the readers feel that a Feed which allows you to browse various content of interest and Recommended articles based on subject of interest as the most helpful additional features in finding information on Wikipedia. Reading motivation directly influences reading behaviour; readers have multiple / varied motivations and that emerge as different reading patterns/ behaviours.	Content Recommendations. Content Resonance. Content Discoverability.
Wikipedia Preview (WPP) Research Project: Most readers did not discover the WPP link and as a result did not interact with Wikipedia because they skimmed through the content, had limited interest in the content or focussed on getting an overall understanding about the information given in the article.	Content Discoverability.
Wikimedia Brand Tracking Wave 4 Respondents said they would like to have recommendations of wikipedia articles.	Content Recommendations.

Based on the opportunities identified in the various research above, the hypothesis we will be testing in this experiment is:

If we disseminated Wikipedia content through chat/messaging platforms, we would be able to directly reach and engage youth audiences with our content.

The goal

To test whether a daily reading recommendation service delivered through available chat/messaging platforms is a viable pathway for engaging global youth audiences.

We see opportunities where our content is surfaced to youth audiences where they are by:

Delivering relevant, interesting and useful encyclopedic content to their inbox/ mailbox at specific moments to read.
Presenting content in ways that match their existing patterns & experiences.

Experiment approach

Through an Exploratory and Evaluative study i.e. Dairy Studies + Survey, we wanted to answer the following questions:

How do readers perceive this type of content delivery?
- An evaluation of Moments of consumption can reveal if certain reading patterns will emerge from this type of content presentation.
How do readers engage with queryless discovery?
- Content Resonance will evaluate if our recommendation engine recommends articles that closely match their current interests.

To uncover opportunities around queryless discovery for younger audiences, the team will run 3 testing activities:

Product discovery activities that will be:
- unmoderated testing through Userlytics on general user interaction and perceptions on subscription-based content discovery
- Usability tests on prototype ahead of the Diary Studies
Launch a Qualitative survey in English and Spanish to:
- Uncover current reading experiences with subscriptions, the benefits users perceive, and the potential barriers they face.
Diary Studies with 3 participants per language (English & Spanish) across different age groups, exposed to a pilot test for one week where:
- The participants will subscribe and use the prototype daily on Telegram.
- The participants will be asked questions during and after testing the prototype.

The goal was to quickly collect feedback based on first usage and actionable learnings for future product iterations & considerations.

Experiment Results

Survey results (Wikipedia Subscription)

In collaboration with the Wikimedia Foundation Design Research team, the Inuka team conducted a Qualitative survey on user interactions and perceptions on subscription based content discovery and services. 227 people responded to the survey across the English and Spanish readers. The age distribution of participants can be seen in the table below.

Target participants	Number Respondents : 227	Age distribution
Target participants	Number Respondents : 227	18-25	26-35	36-45	46-55
English Wikipedia	103 English respondents	7	34	38	24
Spanish Wikipedia	124 Spanish respondents	15	19	30	60

Through the survey we collected information to understand what is relevant to users in different aspects, and tabulated our findings; more details in the full report.

Themes		Findings
Information Rituals		Information rituals for users depends on the triggers and type of response/interaction required and the tool available as a medium
Information Expectations: Users need the right triggers to engage.Triggers are classified under 2 types.	Automatized triggers, like ads or reminders. They are concrete calls to action, with no expected response from the user.	Most of the time, these kinds of messages are left behind pretty quickly. Users can just read and share the information and people visit the site. Important note: Encyclopedic knowledge is part of a personal journey. People want to showcase what they know and are interested in.
	Human triggers, such as a message on a board or group. They have different iterations which can be in search engines and others.	Two use cases were identified with the human triggers, they are: Quick & trustworthy messages or responses, and are mainly used to make a point. Rabbithole exploration, that is meant to result in a continuous back and forth between the conversation and the source. Important notes: The Identified barriers in encyclopedic knowledge are still the lack of “ready to be found & shared” features that other types of content such as status, photos or videos have. The above can be in the form of sharing interesting facts or quotes directly from an article or finding something impromptu when you need it. Encyclopedic knowledge here becomes part of the interaction, helping the user build arguments and answers to other conversations.
Information Journey		Understanding that triggers work differently may allow us to define the type of encyclopedic content we want to share, but also, the types of users we might find. Subscription services are not like search engines; the distance between them is still wide. There is a clear growth in the use of Social networks and other platforms for information searching purposes. Still, the English respondents were more cautious about social networks and like their information to be in one trustable place. Important note: Still, there’s a great opportunity for growth if both Social networks and Channels are paired with a trustable sources, such as Wikipedia.
Reading rituals		Engaging by rituals, instead of specific times, could be more effective for encyclopedic knowledge. For some, the morning is the best time to read subscribed content as they are fresh… For others, being able to use it as entertainment while they are waiting or commuting is relevant. A last group thinks closing the day with them is relevant. Important notes: The ability to decide how and in which way subscribers receive the information is important, still, a second chance comes when saving for later is a possibility. Another important feature for the near future is to resume where you left your last read, that could build a good sense of control in them, but also keep them engaged.
Subscription services: 67 Spanish respondents out of 124 are used to subscription services. 56 English respondents out 103 are used to subscription services.		Subscription services are beneficial, if they have an edge because it’s a commitment. For people to use it regularly, they must understand the differential or direct benefit of the subscription and it must meet their specific expectation of: Accessible content, fit for the user's needs, which are: customization to their everyday rituals. alignment to their interests (they expect services to learn their interests fast). Ability to share what they have learnt from their perspective. Important notes: Subscribers want quicker and accurate content recommendations, but it has to match their own reading pace. There is an opportunity to learn everything we need at the beginning of the subscription process, so we get a head start on what could be relevant for them.

Diary studies result (Wikipedia Subscription)

In collaboration with the Foundation research team, using a Subscription Bot developed in Telegram, we tested and analysed how users perceive, adapt, and integrate the Wikipedia subscription into their everyday reading habits for seven days through questionnaires and daily tasks. The table below shows the age distribution of participants.

Target participants	Number of participants: 26	Age distribution			Gender
		18-25	26-35	36+	Male	Female
English Wikipedia	50% with subscription service	50%	25%	25%	37.50%	62.50%
	50% with no subscription service
Spanish Wikipedia	50% with subscription service	50%	25%	25%
	50% with no subscription service

Through this tests we learnt the following; more details in the full report.

Platform selection:

The Selection of a good and accessible platform not only based in programing, but also cultural background and regional usage is fundamental for the success of any launch.

In our case, the usability of the bot was perceived as good, the Telegram environment made the learning and adaptation process efficient, even for those who didn’t use the platform before; the above might not be the case for all the platforms.

The users' journey:

Subscribers went through the following stages in using the bot:

1. The onboarding stage

The onboarding stage was the most complex to analyze and the one that might need more iteration. Users also felt more invested in this step. Finding the bot in Telegram was easy for the testers. However, some of them struggled to understand why they should subscribe to the service. It starts becoming clearer to the testers what to do next at the point they see "Read Wikipedia". The media below illustrates the onboarding steps to subscribe and start getting recommendations.

The testers doubted the service's discoverability by others not involved in the test as sending notifications for people to subscribe would most of the time encounter the barrier of request being lost in people's inboxes filled with other requests.

Important learnings:

It is useful to explain what the Subscription is about, why people should join and the nature of service instead of letting subscribers figure it out by usage. This kind of service should have a unique logo to differentiate it as a service on its own.
Instructions are important to guide users through their first days, like hacks on how to make better searches, reminders on how the bot works and even sharing some automated answers could be a great way of dealing with context.
Users are already used to how big engines work, so they are perceptive and analytical when things don’t go as they expected. Making sure the search bar is the best on platforms is helpful.
In providing subscription services, it has to be verified by the source, that could make it easy for trust.

2. The recommendation stage

Users' first impression of the articles displayed in the recommendation stage was curiosity and confidence to click, read and move back and forth to see to what extent the bot worked. Notifications of articles to read for the day were well received; some found it helpful to be reminded just once, and by the same hour they subscribed. Still, reading happened later. After seeing the suggestions, some opened immediately and decided when to read (most of the time, it was in decompression spaces such as at night or rest times).

The articles' presentation was also well received. Including a photo and a short description made it easy to decide if they were connected to their past searches. Still, some mentioned they wish the article could open on the same platform, while others thought the description could be more specific.

The rating system was helpful, but some feedback was given about the ability to give a more specific retro on the articles. The users also felt strongly about being able to suggest specific subtopics instead of the whole genre or confirm that the recommendations are appropriate for their needs through 1 - 10 ratings or 5 stars.

Important learnings:

The opportunities for improvement in the future seen in this recommendation stage are:

Including a more nuanced rating system with the ability to rate recommended articles in a non-binary system can help with the accuracy of recommendations.
Providing the bot with useful information through messages is something that users with experience in this type of machine expect, and emphasis on the benefit.
Gamification or streak reference could make users more involved towards teaching the bot and receiving the appropriate daily article recommendation.
Users want the recommendation to have good photos, a clear description, a clear call to action, a personalized message, etc. Although it is hard to include everything users evaluate as necessary, it is essential to include them in the service through prioritization.

3. Integration

After using the bot for several days, the users wanted the following capabilities integrated into the bot:

The ability to make different searches in the bot that result in different recommendations that won't interfere with the other one from a previous search.
Making accommodations for the desire to start fresh and reprogram the subscription bot.
They can bookmark (maybe through emojis) the recommendations that are piling up due to time constraints, and they would want to read them later.
The ability to change the recommendation schedule (the timing for recommendations).

Potential services or tools

As participants tried to figure out the use of the subscription bot service, the following products were mentioned:

A subscription service for current events. Regardless of your interests, if something happens, you receive a notification. This could be together with a subscription or by itself. It could also apply for relevant days according to their region.
A subscription service by topics, not by search. Offers a list of much general topics such as biographies, science, history, math, etc., and letting the user be surprised by the aleatory selection.
Limited time recommendations. If someone is visiting a place, recommend relevant articles related to its history, geography, weather, etc.

Final note:

How we allow subscribers to turn this and other initiatives into a tool to satiate their curiosity and maintain the element of surprise may be a challenge in the long run.