Task recommendations for Wikipedia Editors/Usability testing

Wikimania survey

During Wikimania 2014 in London, the UX research team conducted a variety of usability tests for new products. This included testing for task recommendations on the beta cluster. Unfortunately video results of all but one of these tests were corrupted by the recording application, but survey results were available.

Respondents

There were 10 survey respondents.
Four male, five female, and one who declined to self-identify.
Respondents reside in Argentina, Canada, India, Israel, Russia, the U.S., and the U.K.
Respondents edited in English, Hebrew, Catalan, and Russian
Two of the nine respondents edited only infrequently. All other respondents were regular editors of Wikimedia projects.

Key questions and responses

If you are an editor, would you please walk us through / your normal process for deciding what to edit?

“I usually have a list of articles I always plan to edit, but every once in a while, I notice any article and want to change it, so I choose that for editing”

“I usually look something up and find it needs improvement, or I check my watchlist and see things that need work. I sometimes look at pages like FAs in other languages. When I happen to spot something that needs improving when I visit as a reader, I do not generally log in.”

“Review my watchlist and also have a personal list of articles I want to edit. Sometimes look for articles to copy edit but it takes me a while to find the right lists!”

“Make list from internet news feed; paste link to user page; edit using link adding reference, links, mass upload etc.”

“[I visit] Special:Random and fix what I see.”

What do you think about recommendations?

“I had to ask a helper [to find the flyout], then click the word "recommendations". There were three recommended pages, given twice.”

“Highlight any known issues with the current article, verifying links”

“The suggestion takes you to page, but does not help make an edit, or suggest action on the page, saving requires captcha, requires scroll down to save, work flow, buttons not intuitive”

“I think the pop-up looks like the sort of pop-up I click "closed" on without seeing it.”

What needs improvement with task suggestions?

“Recommend some low-level tasks -- and some high-level tasks i.e not only edits but also document creation”

“More suggestions for one edit”

“These recommendations were not easy to locate until the link was pointed out to me. The suggested articles were already linked in the article, so I was likely to have ended up on them anyway.”

“[Give me a] task suggestion at article I am currently on, like a popup citation template to add reference”

“Placement of suggestion link, and [lack of a] counter [of the number of recommendations]”

What did you enjoy about task suggestions?

“[The] pop up feedback, how about more like some wikilove or thanks notification”

Did the task suggestions make sense?

“Needs a larger variety, should be connected to users worklog, suggestions not needed for users with a large workstack”

Takeaways

Several users did not easily find the recommendations link to open the flyout, even if they were otherwise familiar with the personal toolbar. Solutions may include using a one-time guider to point out the link, or using an indicator of new recommendations. Users requested a counter similar to the notifications counter, though this may not be appropriate since currently the number of new recommendations is always the same, and would thus be more binary in nature.
Several users requested having current issues with a page highlighted as tasks, and to point out the issues that need resolving with recommended pages.
Users requested the ability to save or discard individual recommendations. The flyout and post-edit may be a good tool to prototype this functionality, though neither are yet designed accommodate a large list of saved recommendations while presenting new content as well.
The recommendations content is probably not yet useful to more experienced editors. The simplicity of the recommendation logic (only the last edit) is not sufficient for a more sophisticated user with more editing history. Combing through the user's entire contribution history, ala SuggestBot, is an obvious solution for future consideration.

Remote user testing

Test scenario and questions

In this scenario, we asked users to imagine that they were interested in editing Wikipedia for the first time.

Tasks

Visit the test wiki (en.wikipedia.beta.wmflabs.org)
Log in (with provided credentials)
Choose one of the following pages and edit it: Cat, Cheese, Breakfast, or Europe
After editing, look for something next to do. How would you find another relevant page to edit?

Questions

Have you ever edited Wikipedia before this test, even once?
Did you find any recommendations of things to edit? If you did, were these interesting and useful? Why or why not? How would you improve these recommendations?
Would you be willing to publicly share your test with other Wikipedia users? This will help us show how to improve the site for everyone.

Results

Participants

We tested with five users. (Three women, two men.)
Two of the five users had some experience with editing in past. None were regular editors.

Task completion and assessment

All users successfully completed at least one edit. Some completed further edits despite not being instructed to.
4/5 of the users were successfully given and noticed the post-edit recommendation. Users who did not describe or choose to accept the post-edit notification still were able to dismiss it quite effortlessly. Users clearly noticed the post-edit recommendation's confirmation message right away, and its prominence appeared to be beneficial overall. However, since users only completed one edit, this test is not an accurate measure of the experience of seeing the post-edit recommendation repeatedly.
None of the users noticed or used the Recommendations flyout without being directed to it. Once users were directed to it, however, they clearly understood its purpose and were able to paginate through the recommendations easily. This may support lightweight use of a guided tour to point out recommendations.
Performance is a major issue with testing on the beta cluster. Even short delays (1-2 seconds) can cause users to ignore or misinterpret what's going on. Our current strategy for producing the post-edit recommendation can produce a FOUC on slower connections. Long delays of initially loading the flyout caused a few users to assume it was broken. (This will be partially ameliorated in production by smarter loading strategies already in place.)

Video snippets
A user describing the Recommendations flyout
A user clicks through using the post-edit recommendation