Reading/Web/Projects/Related pages

Read more is a feature that is currently stable on almost all Wikipedia mobile websites. It aims to drive page views by engaging users by directing them to related content. The rationale is, if readers are offered suggestions that are similar to the topic they are reading about, this will further engage their reading session time, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics. The concept already exists on apps, where you can check the performance report.

The Problem

If a reader has reached to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.

The How

Using the Extension:RelatedArticles extension will show suggested articles at the bottom of pages encouraging the user to read another page.

A user who gets to the bottom of an article on either mobile or desktop web is shown a list of other articles that are related to the one they just finished. The notion is that if the reader has read to the end of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.

The Effect

This has been released on apps and resulted in a 16% click-through rate (For users who saw it). Additionally, 25% of the users who read more results, clicked through at least 1x in a 1-day period (data).

Moving to stable plan and staging

After 4 months of running in beta on both mobile and desktop, a proposal was moved to stable on mobile.

FAQ

Where do article suggestions come from?

A more like query which will programmatically choose related pages. Here is an overview of the algorithm the function is based on: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html.

The more-like query service ranks articles in order to show the top 3 and uses a score that computes the similarity between documents, this can be fine-tuned as documented on the wiki page for it: Help:CirrusSearch#Morelike. The defaults in code can be seen on github here.

Given the concerns expressed by community members here and elsewhere about popular results making little room for relevance or serendipity, changes were made to the algorithm in August after testing and communication. As of August 30th, 2016, boostlinks (a measure of importance) and boosttemplates (usually a measure of quality) were removed from the scoring, as they caused popular, less relevant pages to appear with high frequency. The ticket for this work is captured here.

Results can be overridden by editors using the {{#relatedarticle:}} magic word.

Do editors have any control or are they given any preview of the suggested articles?

Though this feature is not considered part of the article, editors can change the suggested articles given by adding up to 3 manually curated examples to this part of the page navigation.

For example:

On Korur language the related pages have been over-ridden to:

User:Jkatz (WMF) thinks that making them editable is a problem because:

the results are not automatically updated
it means that improvements we make to the algorithm/selection will be lost to pages where an editor has overridden the automated selection.
right now the manual selection option only applies to the web (not the apps), which is misleading
some editors have requested that we do automated refreshes every time you load the page--this would not be possible if the above keywords are used.

How is related articles different from the See also section, navigation boxes and the category system?

They have pretty pictures, are limited to 3, for greater simplicity (unlike see also, are only intended for users who have gotten to the bottom of the article without finding another link more appealing to visit). It is thought that the images and simplicity increase the click-through rate. Which is the metric of success for this feature.

Initial Community Feedback

The following is an attempt to summarize the feedback collected on the talk page for this feature (as of Feb 25, 2016), along with responses to each one.

Disambiguation pages are pretty bad. See the holocaust disambiguation page as an example: w:Holocaust (disambiguation)

Response:

Luckily, the feature has no real utility on those pages. There is now a ticket to remove this: T127068 prior to any further roll-out.

FIXED ~~On English Wikipedia, the use of non-free (fair use) article images for navigation goes against En Wikipedia policy.~~

~~Response:~~

~~This is being remedied by this task: T124225 and is considered a blocker~~

The click-through-rate metric is not air-tight and reader value is not certain.

Response:

It is true that click-through rates can be gamed. I (JKatz) submit that sustained click-through rates are indicative that readers are finding value, and placement at the bottom of the page, as well as some analysis of raw pagelogs above, suggest that cannibalization of blue links is not occurring. However, enough others disagree that some additional work is necessary, at least prior desktop roll-out. Here are some ideas:
Roll-out on some small % of anon-traffic. It is not entirely fair to test only with logged-in users a feature that is intended primarily for users who are not logged in.
Then, a/b test and look at overall pageviews (for those of you who find session depth to be a bad metric: please suggest a better one)
The, use the quick-survey feature to ask users who click on 'related pages' if they are satisfied with the results or to get feedback

This is the same as see-also and does not add additional value

Response:

The sustained high click through rate on mobile suggests that users are finding it valuable enough to warrant promotion. On desktop this is not yet the case--though the click-rate still exceeds any given link. (see above about obtaining better measures of reader value).
The best thing to do would be to combine with see also, but this is thorny and a lot of technical work. I think we should be assessing whether this is a net benefit to Wikipedia or net negative, not whether or not is a perfect solution.

Sometimes pages return anywhere from suboptimal to damaging results

Response:

for 'damaging' results, which are rare, editors can over-ride results. I was against mixing algorithm and editing, but the value here is now clear.
for 'suboptimal results', for now I think we should live with it because the readers seem to be deriving value (see below on underlined portion)

Algorithms are non-NPOV and go against our ethos OR algorithms should be adapted to community norms

Response:

In the FAQ, we have pasted put a link to the code somewhere and describe the variables/logic? Ultimately, a solution might be to allow each community to tweak the weighting to their needs, but I think that is not warranted by the value of this feature at this point. How do you feel about transparency with regard to the algorithm logic as a solution?
we might want to denote that they are machine-driven somewhere to clarify. Is this a blocker?

In German and maybe Russian, the tool may violate a principle against 'Themenring' policy

Response:

I see no reason why we should push on these wikis if they don't want it, but have asked community members to poll their wikis and find out if the concerns are representative

The community was not consulted first and we would have helped you avoid major pitfalls.

Response:

Unfortunately, yes. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature and others, we hope that you can help us move forward together.

Success criteria

%CTR (click through rate, clicks/views) is higher than 10%.
Ideally, we will be able to ensure that these clicks are not-cannibalistic (increasing overall clicks as opposed to taking clicks from blue links)

Roadmap

Prototyping

This was tested in beta first.

MVP

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished.

We were able to measure the engagement clicks/impressions with this feature.

User Stories

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished so that they can continue reading about that topic. Someone editing a Wikipedia article can manually change the article suggestions using wikitext so that they can correct any erroneous or sub-optimal suggestions. A project stakeholder (such as PM or data analysis) is able to measure the engagement clicks/impressions with this feature so they can determine if it is adding user value.

Metrics Implementation

We wanted to track:

Impressions of read more suggestions (the lump--not each item suggested)
Clicks to read a suggested article
The position of article (1,2,3). Ideally: whether or not article was edited manually
Ideally: a)overall referrals from a page with read more, b)overall referrals from same page without read more

Timeline Estimate

(this has been updated on 2/24/16 to reflect the actual order to-date)

build read more according to mobile web specs Done
launch on mobile web beta Done
launch on desktop web beta Done
measure impact using event logs CTR (referral data won't be helpful at these numbers) Done
discuss with community
launch on mobile
launch on desktop (open discussion about whether we launch in beta first and/or do progressive rollout)
Delivery Estimate: end of Q3 on mobile across wikis...desktop is too much of a question mark as of February 24, 2016.

Metrics analysis

Behavior in Stable

The related pages feature was deployed to 90% of all users on the mobile version of English Wikipedia on March 30, 2017. We collected data in the timespan between March 30 and May 15, 2017. The purpose of this test was to determine the impact of the feature over time. The main metrics we focused on throughout this analysis were clickthrough rate and variations in total events recorded between the test group and the control group.

Summary

On enwiki, clickthrough rates for related pages links were especially high (6%) when users see the related pages section at the bottom of the article. The overall clickthrough rate on related pages links for mobile was 1.9%. No significant changes in these rates were observed over time, indicating that users will continue using related pages with a similar frequency. When compared to the control group, no significant impact on pageviews was observed hinting at the conclusion that while the feature's content curation is preferred among users, it does not significantly affect user behavior - if a user was interested in continuing to read, she will select an article suggested by related pages with a higher probability than another article on a page. However, the feature has little to no effect on users who were not interested in accessing more articles.

Details

Clickthrough Rates

On mobile enwiki, clickthrough rates (ratio of pages clicked/pages seen) for the feature average d 1.91% (ranging between 1.85% and 1.97% with 95% convidence) over the period 03/30/2017 - 05/15/2017. When users view the related pages (by scrolling to the bottom of the article), the clickthrough rate is 6.0% (range 5.89% - 6.14% with 95% confidence). No significant changes in this trend were observed over time.

Clickthrough rate comparison for related pages feature over time

Total Events Comparison

On mobile enwiki, we observed negligible differences between the expected pageviews for the test (related pages on) and the control (related pages off), with the ratio of expected events for the test group to expected events for the control group being 1.003 (range 0.998 to 1.010 with 95% confidence), suggesting that the change has little effect on total pageviews

When comparing the ratio of unique events per total events per session for the test and control groups, similar results were observed, with the control group having 0.9986 unique events per session, and the control group having 0.9994 unique events per session. No significant changes to these trends were observed over time.

Queries

SELECT count(*) AS total_Events, event_eventName, left(TIMESTAMP, 8) AS mmddyyyy 
FROM `RelatedArticles_16352530`
WHERE wiki = 'enwiki' and event_skin = 'minerva-stable'
GROUP BY mmddyyyy, event_eventName;

SELECT count (DISTINCT event_userSessionToken) AS total_Events, event_eventName, LEFT(TIMESTAMP, 8) AS ddmmyyyy
FROM RelatedArticles_16352530
WHERE wiki = 'enwiki' and event_skin = 'minerva-stable' 
GROUP BY ddmmyyyy, event_eventName;

Event logging

As measured for the first two weeks of 2016 on a sampled basis, related articles items were frequently tapped on mobile web beta Wikipedias (the skin "minerva-beta" in the table below) - 19.7% of the time when seen. The relatively high tap rate on the mobile web beta is believed to be attributable to the interesting content, as well as in part automatically collapsed sections in its phone form factor layout yielding a higher impact visually to the related articles at the end of the article.

Interestingly, while users on the desktop web Wikipedias (see "vector" in data below) were likely to see the related articles panel (35% ready-to-seen event ratio versus mobile web's 27.3% ratio), desktop users evidently were far less likely to click on a related article - they apparently clicked only 3.4% of the time when seeing the related articles panel.

select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where timestamp > '2016' and timestamp < '20160115' group by event_skin, event_eventName;

event_skin	event_eventName	count(*)
cologneblue	ready	3
minerva-beta	clicked	217
minerva-beta	ready	4025
minerva-beta	seen	1099
modern	ready	114
modern	seen	62
monobook	clicked	1
monobook	ready	270
monobook	seen	82
vector	clicked	48
vector	ready	4053
vector	seen	1429

The relatively lower click rate on the desktop web is believed to be attributable to the richly laid out expanded sections yielding a relatively lower visual impact for the related articles. It may also be related to the mechanics of beta opt-in: on the mobile web beta opt-in enables all beta features, whereas on the desktop web users are able to opt-in to specific beta features. It should be noted that some users on desktop web also enable the feature to auto-enroll on all beta features.

Tablet devices by default are presented the mobile website, which commonly expands sections, but still has a relatively lighter weight layout. But users have the ability to change to desktop mode, where again sections are expanded but there's a relatively heavier weight layout.

Note that for a confined set of tablet devices, although it's an imperfect analysis and there are a variety of potentially impacting factors that could muddle the analysis, the data suggest that we can at least say with some level of confidence that

mobile web beta treatment dramatically outperformed the desktop treatment
although users with a confined set of tablets on mobile web beta had not insignificant engagement with related articles, their rate of engagement was somewhat lower than the fuller mobile web beta population (confined tablet population's engagement at 16.7% versus full mobile web beta population's engagement of 19.7% for a two week period and 13.2% versus 18.4% for the full period since it was introduced on mobile web beta) - but this was still quite promising on mobile web beta

select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where timestamp > '2016' and timestamp < '20160115'
and (userAgent like '%Nexus 7%' or userAgent like '%Nexus 9%' or userAgent like '%iPad%')
group by event_skin, event_eventName;

minerva-beta	clicked	11
minerva-beta	ready	262
minerva-beta	seen	66
monobook	ready	1
monobook	seen	1
vector	ready	37
vector	seen	9

All events and skins since related articles available in beta for confined set of tablets...

select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where userAgent like '%Nexus 7%' or userAgent like '%Nexus 9%' or userAgent like '%iPad%'
group by event_skin, event_eventName;

minerva-beta	clicked	39
minerva-beta	ready	1485
minerva-beta	seen	295
monobook	ready	2
monobook	seen	1
vector	ready	70
vector	seen	21

All events and skins since related articles available in beta for all UAs...

select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
group by event_skin, event_eventName;

cologneblue	ready	9
cologneblue	seen	2
minerva-beta	clicked	768
minerva-beta	ready	19009
minerva-beta	seen	4170
modern	ready	185
modern	seen	105
monobook	clicked	4
monobook	ready	535
monobook	seen	171
vector	clicked	110
vector	ready	8005
vector	seen	2848

Do note that a CTA experiment to heighten the general enrollment in mobile web beta was running in the latter part of 2015 but was disabled, hence relatively lower event counts for the two week period observed at the start of January 2016. Engagement with the related articles was relatively high with both the CTA in force and has continued to be relatively high (albeit slightly lower) since the CTA was removed. See the data above. Note also that a small experiment with collapsed sections has been running, but should have a negligible impact on the analysis in this page.

HTTP Referer header analysis

One additional signal in the environment suggesting the efficacy of the mobile web beta related articles feature is the share of internally referred pageviews as a percentage of total pagevews. Again, caveats could apply, but it appears that prior to the related articles feature, mobile web beta internally referred pageviews were in the low 50s percentagewise, whereas with the feature they are now in the high 60s percentagewise (the stable channel where the feature was not implemented stayed relatively consistent around 25%). However, this is only a signal; partial rollout in the stable channel of the mobile web would be more telling.

What follows are relative internally referred pageviews rates on the mobile web for a set of Thursdays - two Thursdays in the first two weeks of January, and two Thursdays prior to the related articles feature being released in beta on the mobile web and desktop web. Dates coinciding with holidays or known high rate fundraising campaign banners are not included, as they would complicate the analysis.

Thursday, January 14, 2016

Note on this query: feature running in mobile web beta Wikipedias.

select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2016 and month = 1 and day = 14
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];

beta: 48450/(48450+14958+8465) = 67.4%
stable: 58400028/(58400028+56362895+113239899) = 25.6%

external	NULL	113239899
external	b	8465
unknown	NULL	56362895
unknown	b	14958
internal	NULL	58400028
internal	b	48450

Thursday, January 7, 2016

Note on this query: feature running in mobile web beta Wikipedias.

select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2016 and month = 1 and day = 7
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];

beta: 51732/(51732+14540+8137) = 69.5%
stable: 59881357/(59881357+55286651+109964317) = 26.6%

external	NULL	109964317
external	b	8137
unknown	NULL	55286651
unknown	b	14540
internal	NULL	59881357
internal	b	51732

Thursday, December 3, 2015

Note on this query: feature was not yet running in mobile web beta Wikipedias.

select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2015 and month = 12 and day = 3
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];

beta: 184067/(184067+55814+119778) = 51.1%
stable: 50273872/(50273872+51372508+102363512) = 24.6%

external	NULL	102363512
external	b	119778
unknown	NULL	51372508
unknown	b	55814
internal	NULL	50273872
internal	b	184067

Thursday, November 19, 2015

Note on this query: feature was not yet running in mobile web beta Wikipedias.

select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2015 and month = 11 and day = 19
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];

beta = 189359/(189359+58549+121871) = 51.2%
stable = 51119529/(51119529+54209700+105205731) = 24.3%

external	NULL	105205731
external	b	121871
unknown	NULL	54209700
unknown	b	58549
internal	NULL	51119529
internal	b	189359

Proposal for moving forward

In light of the feedback noted above, I would like to propose the following next steps for this feature:

Publish variables and code that impact article selection (see FAQ) Done
Fix non-free image issue via: T124225 Done
Work on T127068 to remove from disambiguation pages
Rollout on mobile web on select wikis (it seems like stakeholders are less concerned with the experience here, and the 'performance' is much higher)
Rollout small traffic a/b test on desktop on a few wikis and either measure session depth change or use quick survey (the latter is more difficult)
Use results of 3 to inform possible desktop rollout (Hear back from De and Ru Wikipedias as to whether or not a rollout is desired, given 'Themenring' policy)

Any thoughts, suggestions? I will post on the talk page as well here: Topic:Sz30fah2s3enrql4

On June 15th 2016 21:00 local time the German speaking Wikipedia will most likely reject this feature with a large mayority. Sorry for all the money spend. --Eingangskontrolle (talk) 16:12, 4 June 2016 (UTC)[reply]

Improvements/Feature requests

enable close button on individual items to alert editors (or tweak algorithm) when a particularly recommendation is not helpful
merge with see-also
Recast as a "Find Similar Articles" button with a long list of search-style results