User:Melamrawy (WMF)/Related pages RFC

Update: Conclusion from different Wikis

Overall summary:

- Related pages is an initial logic and interesting idea, however the current beta implementation seems to have room for improvements, on both design and behavior aspects. In addition, the clear overlap with the existing See Also section on wikis, proposes the question, of whether the feature needs to behave differently on each language project based on the value it could bring to the project (it could promote orphan articles, it can replace See Also, for projects that don't have high percentage of articles with See Also section). Given the previous, the team is interested in moving forward with gradual roll-out on mobile only. After a time of stable mobile deployments, we can revisit what should change if we want to move forward on desktop.

From eswiki: (one contributor)
- There is room for suggestion improvements
- UX could be better on desktop
- Works well as vertical layout on mobile

From enwiki:
- Different concerns on irrelevant suggestions
- A question: Is this better/worse/could be integrated into See Also
- Design comment on too much blank space in the card on desktop

From dewiki:
- A pre-discussion gave green light to starting an actual vote process. Actual concerns were translated here. Thanks to Gestumblindi for his translations help.

From meta main RFC:
- Algorithm concerns
- Overlap with See Also

Overall summary:
- Related pages is an initial logic and interesting idea, however the current beta implementation seems to have room for improvements, on both design and behavior aspects. In addition, the clear overlap with the existing See Also section on wikis, proposes the question, of whether the feature needs to behave differently on each language project based on the value it could bring to the project (it could promote orphan articles, it can replace See Also, for projects that don't have high percentage of articles with See Also section). Given the previous, the team is interested in moving forward with gradual roll-out on mobile only. After a time of stable mobile deployments, we can revisit what should change if we want to move forward on desktop.

About

The reading team is sharing this RFC to help the determine how to move forward with Related pages feature. We are hoping to decide on two things: What are the modifications needed in the feature, how to design our testing plans. We could only translate a brief of the RFC, if you would like to fully translate it that will be great, or else please feel free to comment in your language. We need to incorporate as much voices as possible, for this cross wiki feature.

El equipo de Lectura de la Fundación Wikimedia ha iniciado una Petición de Comentarios (RFC) para determinar los siguientes pasos de la funcionalidad de Páginas Relacionadas. Esperamos decidir en dos puntos: qué modificaciones son necesarias en esta funcionalidad y cómo diseñar nuestros planes de testeo. Sólo hemos podido traducir un resumen de la ¨¨¨RFC¨¨¨, y vuestra ayuda en completar la traducción es bienvenida. En cualquier caso, podéis añadir vuestros comentarios en español.

Das Reading-Team der Wikimedia Foundation teilt diesen RFC (Bitte um Kommentare), um zu entscheiden wie es mit der Funktion "Verwandte Seiten" (Related Pages) weitergehen soll. Wir hoffen zwei Dinge entscheiden zu können: Welche Änderungen in der Funktion nötig sind, und wie das Testen geplant werden soll. Kommentare in deutscher Sprache sind willkommen. Wir konnten leider nur einen kleinen Teil des RFC übersetzen, daher ist auch Hilfe bei der Übersetzung willkommen.

SVP, aidez nous à décider comment on va évoluer la fonctionnalité "Pages connexes", en beta depuis 2 mois. Par cette RFC, nous souhaitons décider deux choses : quelles sont les modifications que l'on doit effectuer sur cette fonctionnalité, et comment organiser les prochaines étapes. Seule la présentation pourra être traduite (pas les réponses), mais n'hésitez pas à la regarder et à faire des commentaires en français si vous le souhaitez

مرحبا بالجميع، بعد تطوير خاصية المقالات المرتبطة يود فريق :القرآء بمؤسسة ويكيميديا أن يستطلع رأي المستخدمين في المشروعات المختلفة للمساعدة في تحديد، كيفية المضي قدما في تطوير الخاصية سواء للمحمول أو للوب، ما التفاصيل الواجب تعديلها وكذلك كيفية التخطيط لقياس أدء المقالات المرتيطة كخاصية جديدة.

RFC: Related Pages extension

{{rfc}} This is an RFC relating to the Related Pages extension. It is currently available in beta. You may test it by enabling Related pages under Beta features in Special:Preferences. The WMF is not looking to deploy it at this time, and does not want to push it on Wikis that do not want it. We are willing to make reasonable changes if they will make the feature desirable or more valuable. This is an experimental use of the RFC process and has no outcome other than to generate feedback for the WMF. The Wikimedia Foundation Reading team wants to know:

What needs to improve or change in the feature?
Should any changes be made to our plans for testing the feature and measuring its performance?

Collaboratively drafted by Alsee, Melamrawy (WMF), and Jkatz (WMF). ~~~~

Background

The Related Pages extension (also known as the Read more feature) is an experiment carried out by the Reading team in order to further engage readers and help them learn more. It is primarily geared towards users who are interested in learning, but do not have a specific goal in mind. A user reaching the bottom of an article is shown the title and lead image for three articles that relate to the article they just finished. The related articles are selected by software based on textual similarity to the current article, with a preference for articles with many inbound links, and a preference for articles carrying the templates for Featured article, Featured picture, Featured sound, Featured list, or Good article. (More details at Related_pages#FAQ.) Editors may override these software selections by adding up to three {{#related: page title}} to the page.

The premise of Related Pages is that users who get to the bottom of an article have passed over blue links and "See also", have finished the article, and are still interested in reading more. Surfacing articles that are similar might be exactly what they are looking for.

While Related pages is primarily intended to increase learning by offering suggestions to the reader, learning is a very hard value to measure. As a result, we have been looking at engagement with the feature. This has been released on apps and saw a 16% click-through (for users who saw it). On mobile web, there is a 19% click-through (for users who saw it) The desktop numbers are much lower at 3.4%. While this rate is far higher than the rate of any 3 links on the page (see Research:Improving link coverage) and certainly higher than those towards the bottom of the page (see graph) there is some concern that this is simply because there is an image. To counter that, there is data that suggests the new clicks are additive--meaning they lengthening reading sessions and are not taking clicks away from other links, but these results are not definitive.

Rationale: If readers are offered suggestions that are similar to the topic they are reading about, this will further engage them, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics.

We are using the RFC process as an experiment to see if can be a useful tool for determining product direction.

A brief summary of initial feedback at the project talk page

The community was not consulted first and we would have helped you avoid major pitfalls.

Reading Team Response: There was no structured consultation in ideation phase, because we don't yet have clear mechanisms for doing so, however, there was an update around the team's thinking to start working on a "Read More" feature, similar to the one on the web, this was published in Reading team updates, published in early November 2015. It was announced on Wiktech-l. In addition, there was a post made on the English Wikipedia Village pump in mid November 2015, in order to update the community about the work happening on the extension. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature and others, we hope that you can help us move forward together.

Some language Wikipedias, notably German and Russian, have policies prohibiting collections of related links unless it is an objective and complete set.

The Reading Team does not want to push the feature on wikis that don't want it. The reading team is not looking forward to break any community policies, however, it is essential to understand how exactly does Related Pages conflict with policy on German Wikipedia. It also is important to understand the different angles of the discussion on Russian Wikipedia, if there was a discussion.

English Wikipedia policy on non-free images does not allow their reuse for decorative or navigational purposes.

Non-free images will be excluded.[1]

This is the same as see-also and does not add additional value.

See also sections don't exist on all articles. The section has less presence on smaller Wikipedias. At this is a point of the discussion, and one aspects which this RFC can hep define, is how a feature that helps readers further learn about a specific topic by offering more articles related to the topic that they are reading about, can also help the editing community better shape our content in articles. The Reading Team believes the sustained high click through rate on mobile suggests that users are finding it valuable enough to justify the feature. For some articles software might provide objectively better at links, especially as the software improves.

Wikipedia, unlike many commercial websites, is not about generating as many clicks as possible.
- This is not about generating clicks. The click itself isn't of value. This is about availing more knowledge by offering more articles related to the topic that readers are learning about, and helping them learn more efficiently. A good website is composed of interlinked pages, that are easy to discover and find, which is one reason why Wikipedia identifies an orphan page as a problem that needs action.
Many Read more suggestions are redundant to links in the article, and other suggestions can range from suboptimal to bizarre or even damaging. Generic products tend to be given links to arbitrary brands, for example Read more at the Hard disk drive article suggests Seagate Technology and at Car it suggests Mercedes-Benz. Potentially offensive articles and images can show up unexpectedly, such as images of penises and maggot infested wounds. At Korur language it suggests Anus, Anal and Poop. At List of serial killers before 1900 it suggests Peru national football team. At Murder of Kelly Anne Bates it suggests Batman.

Reading Team Response: For 'damaging' results, which are rare, editors can over-ride results. For 'suboptimal results', for now I think we should live with it because the readers seem to be deriving value.

The software returns particularly poor results for Disambiguation pages, for example at Holocaust (disambiguation) it suggests Batman.

The Reading Team sees little utility for the feature on Disambiguation pages. The feature is being disabled on those pages.[2]

What needs to improve or change in the feature

If the feature is deployed here or on other Wikis in the future, how can we improve or change the feature? More specifically, is there anything that would need to change before you would want it on your wiki? ~~~~

Should any changes be made to our plans for testing

Our first step in testing new products is to make them available under Beta features in Special:Preferences. Logged-in volunteers can try experimental new features. This is where we are now.

Next steps:

If Beta goes well then our next step in testing is normally to see how it performs for a limited number of randomly selected logged-out users (A/B testing). Due to privacy reasons, the best metric we have is going to be the % of people who see related pages who click on them--we are aware that this is not an ideal metric, but it is directionally helpful. For logged-in users, currently on mobile we are seeing a 19% Click-rate and a 4% click rate on desktop. Another alternative would be to run a survey to see if users who click on a related page entry have any difference in satisfaction over other users, but self-reported satisfaction results rarely change.
Or we can partially roll it out on several wikis at a time and see. The argument for the latter is that this has been on the apps for almost a year and seems to be doing well. ~~~~