Topic on Talk:New Editor Experiences/Flow

Finding what to do as a new user is one of the aspects that intersect with different issues surfaced by this research. Several ideas have emerged around the area of task recommendation/suggestions. So I thought it would be relevant to share our experience providing suggestions of articles to translate in Content Translation.

We wanted to suggest articles for editors to translate, and we wanted those suggestions to be relevant. The Language team worked with the Research team to integrate their recommendation system, but before the integration was ready the Language team used a very simplistic approach as a placeholder (showing featured articles that were missing in your language).

Graph showing the initial increase since suggestions were introduced to Content Translation, and the spike resulting from improving the algorithm.

At some point we changed the basic suggestions algorithm to the more advanced one from Research. Content Translation was sending the recent translations from the user and getting relevant suggestions related to those. During user research around that time it was noticeable that the more advanced recommendations were working much better according to the users perception. But it also resulted in a big spike in the number of suggestions that users picked to create an article, going from under 50 per week to more than one thousand (the graph of that initial growth is shown at the side).

Since then, the number of suggestions selected by users to start a new translation has grow to more than ten thousands per week. We still don't know which percentage of those end up becoming successfully published articles, but our current data indicates that we at least are able to provide suggestions that are relevant enough for users to be interested in contributing on topics they may not contribute by themselves otherwise.

I just wanted to share the story to highlight that we have a powerful recommendation system that has proven useful in a specific context such as translation. Knowing that we are able to provide good recommendations may help to reduce the uncertainty when evaluating the risks of projects in this area.

Thanks Pau. I have always felt that we don't do enough to enable users to pursue their natural subject interests. Clearly, this recommendation system works because it does that, suggesting tasks that are aligned with areas where users have demonstrated interest.

There was some work (five or more years ago?) on suggesting "easy" editing tasks. One outcome was that new editors would much rather edit the page that they're reading right now, than to go edit a different, easier page. We got more editors, but since they were likely to be reading popular and/or controversial topics, then they got more conflict.

I think the work @Whatamidoing (WMF) refers to is the Onboarding work the Growth team did. In particular, the last experiment, OB6, where users were given the choice to either improve the current page or get random easy tasks. On the metrics meeting of November 2013, there was a short presentation about this (jump to minute 21 of the video) with more details.

My interpretation of the results is that many users create an account having a specific editing task already in mind. They may want to fix a typo on the current page or create a new article about their favorite music band. In any case, proposing them to do something else is not effective at that point.

In addition to quality, the way and the moment in which suggestions are presented have an impact. Compared to the approach to suggestions used in Content Translations, the onboarding experiment was not providing a persistent point for users to return to and get suggestions later when wondering what to do (since the focus was on the first-time experience) or connecting those suggestions with the topics of interest for the user in any way. Both of those aspects were considered in the case of Content Translation.

An interesting update regarding the impact of suggestions in Content Translation. During the last quarter (known as Q2 or just the Oct-December period) we saw that about 20% of new translations were started from the suggestions offered. Among other things, it would indicate that suggestions exposed a significant amount of interesting topics for users to translate they may not have thought of otherwise.

Another way to put it: providing personalised suggestions resulted in a 25% increase over the translations that users proposed by themselves, a significant number of translations that may not have been started otherwise.