VisualEditor/Design/User testing

We test VisualEditor with users who are unfamiliar with the software and often who are unfamiliar with editing so that we can see how intuitive the interface is.

Process

Use a test version of a wikipedia article
Determine a set of tasks that we want a user to complete for each test
Find random people to view the test article and complete the tasks
Observe those people, see if they can complete the tasks, where there are problems
Look for other problems not related to the tasks at hand
Recommend changes to address these problems

2013–14 Q4 testing

Test 1 - May 20, 2014

screenshot of toolbar as existed in June 2014

We surveyed 5 people with usertesting.com. Each participant was introduced into an example article in VisualEditor mode and asked to complete a few tasks, typical to basic editing:

Enter text
Add a special character to text using toolbar
Create a link
Make text italic
Enter title, make it a subheading
Enter text, make it a paragraph under the subheading
Add a picture
Save their edits

In general, users were able to quickly find what they needed on the toolbar and complete the tasks. But often they were not sure if they had completed the task successfully.

Finding things

Users generally found things either right away or by looking through toolbar menus from left to right.

Sometimes when users couldn't find something looked to the left sidebar under tools. clip

Recommendations:

Less clutter:
- Done in 1.24wmf7 Move bullet lists into their own menu
- Consider moving bullet items elsewhere – possibly the text formatting menu.
- Try moving the section menu into the text formatting menu.
- Run a card sorting test to see if users expect different ways of organizing the toolbar.
- Adopt a principle of having the most commonly used tools to the left of the toolbar.
- Re-prioritize some of the items in the menus.
- Break up into groups.
Special Character: widen the dialog to make it easier to scan through, and break characters up into smaller groups

Link

Most users were not sure they successfully added a link.

Recommendations:

Add a check button or something to show a user confirmation.
Add a way for user to go open the link in another tab.

Picture

Some users struggled to add a picture. 2 users tried to go to insert > gallery, and got really lost. Once a user had added a picture, they weren't sure that they had successfully added it to the correct place.

Recommendations:

Done in 1.24wmf7 Remove "gallery" from the insert menu because it's confusing for most users and very rarely something they want.
Move media icon to top-level toolbar.
Change dialogue title to "Add images or video" – field in phab:T175960
Be able to add gallery from this dialogue?
Improve drag and drop, and indicate somehow that images fall where they can fit on right rail

Save

Most users saved easily and felt they had done it right. One user wanted to fill out a detailed edit summary and struggled to remember what she had changed.

Recommendations:

Move save dialogue to be anchored to the top right, closer to save button, so that user can more easily see their changes.
Have summary filled automatically

Edit source vs Edit beta

One user upon entering test immediately clicked edit source and never found VisualEditor.

Recommendations:

One tab, with ability to switch in the toolbar.
Tooltips – Give more information in link and reference tooltips so a user can quickly read / decide if they want to edit them.

Next test ideas

Watch more experienced editors use VE.
Do a comparison with 1st group - Wikitext users, 2nd group - VE users

Q1 2014–2015 testing

Test 2 - July 31, 2014

July31 toolbar

We again surveyed 5 people with usertesting.com. The tasks:

Enter text
Add a special character using toolbar
Create a link
Add a citation
Edit an existing citation
Add an image
Save their changes

Overall

Testers completed most tasks easily. 2 users were confused while creating a link – they weren't sure that they were successful. Almost all users struggled to find an existing citation to edit it.

Creating a link

"I was expecting something like a page. I guess this is already a link? But I'll just click Done"
"I don't know whether this is functional link because it doesn't have a page connect, it needs to be connected to some other page, but it seems for the sake of the test to have worked, it has this line on hover..."[video clip]

Recommendations:

Give users more information about the page they will be linking to.

Include the image thumbnail
Say "page exists" with a check or "page does not exist"
Include first sentence of article
Allow for hovercards, which shows both image and text, but harder to discover
Easier way to visit the page

Finding an existing citation to edit

The test question read: "At the end of the first paragraph, you find a citation that contains a typo. Edit the citation to fix the misspelling of "British"." Which is probably not the best way to test this, a more realistic workflow is that a user would notice a typo or whatever in a citation while in read mode, then enter edit mode, so they would know where it is located. Even so, this test shows that its difficult to find citations in while in edit mode.

Users scrolled back and forth from references at bottom to citations in article, tried to do Find on this page, and were confused by citation tooltips. (Maybe because test didn't ask them to find a website citation.)

Recommendations:

Include the entire reference text in the tooltip.

Test 3: Links - September 15, 2014

Based on findings from the last test, we wanted to test the link inspector further.

Tasks:

Highlight a word and make it a link (to an article different from the link title)
Use the open button to view an article
Use the remove button
Add an external link

Make a Link

The first task was to highlight Impressionist, then make it link to Impressionism. Most users struggled with this task, only 2 out of 5 were able to complete it. One of those users were not sure that they completed it successfully. "It has an underline, so I would assume that will link correctly to the article." A few users had trouble with changing the text to "Impressionism" and looked for other places in the article that had the word instead of typing it out themselves.

View video clip

Recommendations

Make search results recommendations shown above.
Include a button that says specifically "change" or "edit".
Include search icon.

Open

Most users were able to use the open article feature quickly.

Remove

Most users were able to use the remove link feature quickly.

External Link

No users were able to complete this task. Users were hesitant to click after the last link in the bullet list and hit enter to start a new bullet. (This could be because the cursor goes between the text and the external link icon. "I don't want to delete this link here". View video clip

A few users were able to enter a new bullet, but struggled with pasting a URL into the link search field. Many looked all over the toolbar, and even the left sidebar and top navigation for an easy way to add an external link. Several users tried filling the cite web form for external link. View video clip

One user searched google for "how to add an external link to wikipedia", which took her to a help page for external links in wikitext. So she added [http://giverny.org/monet/welcome.htm] directly into the text.

Some users correctly pasted a URL into the search field, but did not include the http://, and because of a bug, VE did not recognize the link.

All users tested expressed frustration while trying to add an external link.

Recommendations

Show placeholder text in search field "Search articles or paste URL"
Show a guider explaining that you can search or paste URLs
Fix bug allowing software to recognize URLs without http://
Add a new item to toolbar "External Link"
Add a new section or toggle to link inspector for external links

Q2 2014–2015 testing

Coming

External links

Why You Only Need to Test with 5 Users – The best results come from doing multiple tests with three to five users each, rather than one big test with more people