Onboarding new Wikipedians/user testing

We've conducted four remote usability tests.

Testing scenario one

We asked users to complete the following tasks.

Create a new account on Wikipedia. (Use any name and password you prefer. Email is optional.)
Take a moment to scan this page [Ed. Special:GettingStarted]. What do you think this page is asking you to do? What do you expect to happen next if you click on any of the links?
Follow the instructions provided to get started with editing. Take no more than 10 minutes. If you get stuck, don't worry. We're looking to learn how to make this experience easy for people, so detailed feedback really helps, even if you aren't sure what to do.
If you're finished, close your browser tab or window. Imagine you might be meeting some friends for dinner, and want to end your Wikipedia session.
Now let's say you're back from dinner and want to go back to editing Wikipedia. Open a new browser tab or window, and return to Wikipedia. Please talk through the process. If you're not logged in, please log in again.
If you completed an edit in step #4, where would you expect to find this edit?

Test A

Test A

Did you edit Wikipedia before this test, even once?: No, I have never edited on Wikipedia.

What frustrated you the most? What improvements would have made the process easier?: The editor was very stratightforward, however it took me a few moments to see the 'edit' tab, top right. If it were easier to see I would have more quickly understood how to navigate between regular layout and edit.

What did you like about the process, if anything?: I liked the editor. It could have been much more imposing and intimidating, but the layout was friendly and easy to understand.

Test B

Test B

Did you edit Wikipedia before this test, even once?: No, never.

What frustrated you the most? What improvements would have made the process easier?: I didn't see anything that I would really consider instructions. There were notes but nothing to walk me through or really explain anything about the process. I would want more step by step 'how to' instructions.

What did you like about the process, if anything?: I like that you can edit these pages, but other than that I didn't like much because everything seemed so difficult and too big project.

Test C

Test C

Did you edit Wikipedia before this test, even once?: No, I did not edit Wikipedia before this test, not once.

What frustrated you the most? What improvements would have made the process easier?: What frustrated me the most was not being able to go back and locate my edit. I don't feel the editing process was that hard, just finding what I edited was difficult. I would improve this by adding a button to see what you've edited in the past, that would make it easier.

What did you like about the process, if anything?: What I like about the process was the simplicity of it. It seemed pretty easy to do the actual editing.

Conclusions

We specifically asked the testers what they thought Special:GettingStarted was asking them to do, and what would happen if they clicked on any of the links. All of the users understood the landing page correctly, and the request to choose a page and contribute to it.
While users understood the GettingStarted interface was asking them to edit, they assumed that they would have to have some prior knowledge/expertise in the subject of the articles in the list.
There was clearly not enough instruction about the kind of task requested and how to complete it when users were on an article they chose from the list.
Users had less difficulty with wikitext than one might expect, almost certainly due to the nature of the task we were asking them to consider. They generally ignored markup that was irrelevant to their work, and some even identified sections within articles. The smaller maximum length of onboarding articles, 10k bytes, may be a factor.
None of the users completed an edit during the test, though 2/3 of them made it to the edit screen, then described precisely what they would do. (The one user who attempted to actually save an edit ran in to the CAPTCHA delivered to users when they save an edit with an external link addition.)
Users all navigated to Wikipedia through the main wikipedia.org portal, using either the link to English Wikipedia or the search function to find "their version".
There are a number of other insights in to how users learn to navigate Wikipedia, including key functions such as search, history pages, watchlists, and more. Those processes are currently outside the scope of this project, so we won't elaborate on them yet.

Test scenario two

This was a test of the new Special:GettingStarted landing page, sans guided tours, launched on Thursday, March 7th. We asked users to complete the following tasks:

"This is a test of the signup process for Wikipedia and a new interface for getting started as an editor of the site. In this scenario, imagine you've decided to join Wikipedia to update and improve its information. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible.

Create a new account on Wikipedia. (Use any name and password you prefer.)
Take a moment to scan this page. What do you think this page is asking you to do? If you want to become a Wikipedia editor, what would you do next?
Once you've scanned the page, go ahead and try to make your first contribution as a Wikipedia editor. If you get stuck, don't worry! Just explain what you would do next and why. It's okay to say you feel like you would give up at any point."

Test A

Test A

Did you edit Wikipedia before this test, even once?: I've done it before, but never consciously being aware that I was being a copyeditor perse, more just clicking on edit on the top when I've noticed something needs to be hyperlinked or spell-corrected. Probably done it over a hundred times by now.

What frustrated you the most? What improvements would have made the process easier?: You might explain the process with some kind of information or something on that 2nd page that one comes to after signing up to sord've let them know what the next step is, that page kind've leaves it up to someone to guess what to do next.

What did you like about the process, if anything?: I liked editing the errors using the edit feature, works really well. The signup process was easy too.

Test B

Test B

Did you edit Wikipedia before this test, even once?: I have never done it before today.

What frustrated you the most? What improvements would have made the process easier?: Reading the strange, foreign format confused me the most. That would take some getting used to. I would suggest the editor read the article first and then go to edit. I would also show some examples of what is importatnto be edited- that would help.

What did you like about the process, if anything?: It is fun to improve things- I liked reading and revising.

Conclusions

Issues identified from the tests:

In the first two tests, we unintentionally primed users to choose the "Copyedit" task by using the terminology "Wikipedia editor". This was corrected in a third user test script.
There was confusion about what kind of decision the user needed to make on the page. The fact that they needed to choose both a task type and article associated with that task type was not immediately obvious.
Descriptions of the task types were not clear to users.
The description of the landing page did not make it clear what the process was (i.e. choose a task type, choose an article, edit it).
Users definitely read from left to right, implying the randomization of order was a correct choice if want even distribution between selection of different task types.
Users found and used the question mark flyouts to help understand the task
Some users attempted to click the task icons to select that option.

User experience enhancements:

Users obviously expected something like guided tours, which implies it would be valuable for us to build tours for each task flow.
The copy in the new Getting Started landing page needs work. The descriptive paragraph should probably be expanded, and the description of each task type should more literal (e.g. "Fix grammar and spelling" instead of "Copyedit").
E3 needs to test a version of the landing page that simplifies the choice needing to be made (i.e. not task and article, perhaps not even from multiple options). This should be done either with progressive display or removing the options to choose altogether.

Testing scenario three

We tested a prototype (i.e. a clickable mockup) of two redesigns of the toolbar given to users on GettingStarted task articles. We most wanted to explore whether the visual design fit with user expectations about a progress bar and other elements we added. We additionally wanted to get feedback about whether the request to complete five edits fit the mental model users had about the GettingStarted process.

Our test script was as follows:

For this test, you'll be looking at a prototype (i.e. a preliminary version) of a new interface for Wikipedia. The purpose of this interface is to suggest ways to improve the content. Imagine you just created your account, and you are interested in learning how to contribute to Wikipedia. For the duration of the test, please move your UserTesting.com panel to the bottom left of the screen to make sure it's not obstructing any content. Since this is a prototype, not all links or other functionality will work. If something seems broken, don't worry, just tell us what you expect it should do.
Imagine you are new to editing Wikipedia and want to help with simple typo fixes. This page is what you would see immediately after creating your account. What do you think this page is asking you to do?

Go ahead and choose the option to "Fix Spelling & Grammar", if you haven't already.

Can you identify and fix a typo in the article? For the purpose of the test, you don't need to actually type in a change, just go through the steps you think would be necessary to complete one.

If you've completed the steps to fix a typo in the article, is this what you expected to see after? What do you think you're being asked to do next? Please describe what you see in detail after saving, and what you think would happen if you click on any of the links at the top bar.

Did you noticed the progress bar at the top of the article page? Please describe what you think you need to do in order to complete the progress bar.

Test group A

With this testing group, we had them try out one of the two mockups we've made for updating the GettingStarted toolbar on articles.

Before you make your first edit
After the first edit is completed

The first four tests above did not include highlighting of the "Try another article" step as specified in the mockup, so we ran two additional tests after adding that.

Test group B

With this testing group, we had them try out one of the two mockups we've made for updating the GettingStarted toolbar on articles.

Mockup similar to what we tested in this version

Conclusions

The following are notes about the UX questions we were attempting address in these tests, including solutions.

Landing page

Getting Started initial options are clear to users and what they find after selecting a task meets their expectations.

Entry point for editing

The edit tab is hard to discover. For many users it gets lost into the chrome.
The edit link from the template is frequently used as the main entry point for editing. This might partially be due to the fact that the template appearance differs from production (it lacks orange borders and other styles that might distract from the links).

Improvements

Show the help guider initially by default will help to locate the way to edit.
- The 2 extra steps are probably understandable from the user perspective in a “getting started” process.
- There is probably no need for a persistent help button. The problem seems to be more discoverability than learnability. Once the user discovers where the entry point for edit is, has no problems later.

Discoverability of the toolbar

Users focus on the content area during and after editing. This makes the top bar to go unnoticed.
The current implementation seems to work better than the prototype since the bar appears some seconds later and makes the page to move, making it more noticeable.
The blue bar version seems slightly more discoverable than the green one, but still users don’t look for it until they are asked about the next steps.

Ideas for improving

Appropriate animations, including loading of the toolbar, animation of the progress indicator after load, or highlighting of the Try Another button.
When the initial modal dialog is shown, we have an opportunity to make the user more aware of the toolbar.
For the green version, the current step indicator can turn green progressively or use a transition to call user attention.
Improvements described in next section will also help for general discoverability.

Understanding the process

Users that do not discover the top bar don’t realise they are in a 5-edit process.
When users identify the top bar, they understand they are involved in a process but it is not clear what is the process about.
Providing a “Make four more edits to complete the editor training” helped in the “blue bar” version to inform users were in “tutorial process for editing”.
Users tend to associate work units with different articles.
The users that went through the 2nd edit are able to keep going forward.

Ideas for improving

Provide context on the first steps. Add “Make four more edits to complete the editor training” to the green bar version.
Reinforce the “next article” mental model by highlighting this option after first edit. This also helps with the discovery of the toolbar.
Additional edits on the same article can also show progress, but we will be encouraging the behaviour that seems more natural to users.

Understanding what happens after completing the process

Users don’t had a clear idea of what to expect after the 5-edit process, but there is a notion of “going to a next level”.
Leading them to a different and more advanced task seems to fit in their expectations.
Becoming a more advanced type of user may be also expected.

Ideas for improving

The initial getting started page can reflect the tasks the user completed, and add some more types of tasks as other tasks are completed.
One of the more advanced tasks can be “dealing with templates”, where the user is asked to improve an article and remove the template when done.

Testing scenario four

In this test, we asked users to try the new version of the onboarding process where they are first redirected back to where they were prior to signup, instead of going through Special:GettingStarted. In order to shorten the process, we assigned users login credentials and a test link (either to an editable article or to the Main Page). We then asked users their impressions of the calls-to-action and the help given to them. We tested with five users, with three given the editable article link and the other two given the link to the Main Page.

Understanding the calls-to-action

It was clear that most users noticed and understood the two calls to action possible. Particularly helpful was the first invitation to edit the example article, with the sub-heading of "We'll show you how." All users had expectations about the next step that would appear after clicking on either call to action.

The secondary call to action is not always understood in contrast to the first one. Although the user guesses are aligned with the action, we could tweak the wording a bit to present the secondary action as a more clear alternative to the primary one. For example, "Find more pages that need easy fixes".

Clip A
Clip B
Clip C

Discoverability and distractions

On the example article given (which was quite short) users had no trouble noticing the calls to action in the modal window. It was clearly the most important thing presented to them. However, on the Main Page, users had much more trouble understanding that the modal with CTAs was the number one thing for them to address before moving on.

Potential solutions for this include the following design changes we plan to implement: dimming the background page, similar to a center-aligned guided tour modal, and making sure the vertical offset of the modal is not too far down the page.

One major disadvantage of using the modal is that if the user closes it for any reason, they have lost the ability to return to the instructions given, unless they happen to refresh the page or move forward then back. One idea for solving this is after a user dismisses the CTA, use a guider to point to a persistent navigation element where the "getting started" guided can be called up, such as on user Contributions.

In a similar way, when the process is completed (the user makes the first edit), users assume the process is over, providing no way to chose a different path (e.g., find more articles). We can congratulate the user when the process is complete and invite to keep improving the article or look for more (and introducing the place where to do so, if that place exists).

Other elements that had similar on-boarding goals such as "Sandbox" or tutorial links, can distract the user. We may consider different approaches (a) disable them during the process (and optionally introducing them when the process is over), (b) integrate them in a more consistent way (e.g., inside the "contributions" view if that is defined as the entry point for contributing).

Clip D: this user closed the CTA after reading it, leading him to get lost later
Clip E: this user closed the CTA without even really reading it

The editing process, including guided tours

The challenges faced by users in editing wikitext are nothing new. The guided tour provided to users who chose the primary "edit this page" CTA was clearly advantageous, teaching users about the difference between read, edit, preview, and save modes.

Where our process of instruction fell apart was in two key areas. First off, in the case where the user accepted the secondary call to action (same as the previous and current GettingStarted copyediting guide) we do not yet give the full tour by default, but wait for the user to take the tour. This is a mistake. Second, after a user clicked Edit, the next immediate step shown was of the Preview button. Some users expected a step introducing the edit area, toolbar, and wikitext in general. This is not a bad idea, including potentially showing users the Help dropdown menu with markup examples. Even without this step, we should perhaps wait until the user types in the edit window to point to Preview.

Clip F: "I'm not really sure... all this business." [Talking about wikitext]