Wikimedia Research/UserTesting.com, Basic editing tasks: brand new editors

Basic Editing Tasks in Visual Editor; New Editors

Goals of Research and Results

Goals

How successful are brand new editors at a variety of basic editing tasks in Visual Editor, including:
1. Knowing how to begin editing
2. Adding text
3. Making existing text bold
4. Making existing text italicized
5. Creating a new subsection, with correct subheading level and section content
6. Adding a citation
7. Editing a citation
8. Adding an internal wikilink
9. Removing an internal wikilink
10. Adding an external link to the External Links section
11. Finalizing/saving edits
What challenges and confusion do users face when attempting to complete these tasks?

Findings

High level findings:

Here is a deck that we built to describe the main issues found. This deck is the synthesis of all the details below.

Detailed findings:

Task Success
- Knowing how to begin editing: 5 of 6 users actively identify the edit button on top right. 1 user skips this first task.
- Adding text: 6 of 6 users complete this task.
- Making existing text bold: 6 of 6 users complete this task. 2 of 6 users use ctrl+b or cmd+b instead of formatting button in toolbar.
- Making existing text italicized: 6 of 6 users complete this task. 2 of 6 users use ctrl+i or cmd+i instead of formatting button in toolbar.
- Creating a new subsection, with correct subheading level and section content: 2 of 6 users complete this task. 3 of 6 users are unsuccessful. 1 user creates a section and demonstrates awareness of subheading levels and where to find it in the toolbar, but does not make a subsection specifically as asked (could be due to a misunderstanding in task phrasing).
- Adding a citation: 2 of 6 users find and use cite web in the citation dropdown.. 1 user does not complete the task initially; in the process attempting the follow-up citation task about adding a title, accidentally finds auto-cite. 1 user also accidentally finds auto-cite, after trying insert dropdown, cite dropdown, formatting dropdown, omega icon, etc. 2 of 6 users are entirely unsuccessful at this task.
- Editing a citation: 4 of 6 users complete this task.
- Adding an internal wikilink: 3 of 6 users complete this task, 1 of these after much trial and error. 3 of 6 users do not complete this task; 2 of these add external links instead.
- Removing an internal wikilink: 5 of 6 users complete this task (1 removes the external link he created). 1 user deletes the text of the incorrect link he made during the add citation task.
- Adding an external link to the External Links section: 5 of 6 users complete this task. 1 user copy-pastes the link in (with the quotations included, as from the task prompt) and saves without making the url an actual external link.
- Finalizing/saving edits: 5 of 6 users complete this task. 1 user does not actually save edits at the end of the testing session.
Citation-related
- Only 2 of 6 users find the auto-cite button in the toolbar, and both of them found it by accident.
- 2 of 6 users specifically point out that the cite icon looks like a bookmark. It does not seem readily apparent that it correlates to 'citation' concept/functionality.
- 4 of 6 users head to the insert dropdown first when attempting to add a citation.
  - Users experience contextual confusion about citation / reference / (internal and external) link and sometimes footnote. This affects both users' understanding of what to add to begin with and secondarily where users think to look for the functionality to add these elements.
  - There is a lot of trial and error before being able to successfully add a citation.
Other findings
- 4 of 6 users make a subsection heading bold first, and only 1 of these ends up realizing that making a subheading involves more than making the text bold.
- 2 of 6 users end up adding external links when prompted to add internal wikilinks. The link button workflow does not adequately explain/depict the internal page 'search' functionality.
  - Internal page 'results' only show up in the dropdown when 1) the user clicks within the text entry field (but most don't do so because they have already highlighted text and it is pre-populated upon clicking the link button) or 2) the user 'opens' the link (which uses the pre-populated text to pull up a page, if it exists or not).

Bugs/Recommendations

Bugs
- 2 of 6 users experienced multiple instances of a page glitch that jumped the user to a different part of the page when the user attempts to click on a specific part of the page to complete a task.
  - Recommendation: Review video to determine cause of glitch and hopefully find a fix.
Other Issues
- Insert dropdown is vague, yet also gives off a connotation of all-inclusiveness.
  - Recommendation: We should either rename/specify it or potentially put some functionalities (like adding various types of citations) within its dropdown.
- Cite icon is construed as a bookmark, and its function not readily apparent.
  - Recommendation: Reinstate Cite text button instead of bookmark citation icon, or brainstorm a better icon representative of 'cite/citation'.
- There is inconsistency around the current cite icon.
  - First, it is spatially separated from its corresponding dropdown arrow. All other icons/buttons with dropdown arrows are close together and clearly correlated.
    - Recommendation: Move them closer to each other.
  - Second, the citation/bookmark icon activates the auto-cite functionality. All other icons/buttons with dropdown arrows do not activate an altogether separate functionality; when clicking those icons/buttons, they simply open up the dropdown menu.
    - Recommendation: Clicking the cite icon opens up the citation dropdown menu. Auto-cite can simply be an additional 'quick' option in addition to the other citation options currently in the dropdown.
- Internal wikilink 'search results' within link icon workflow should be clearer to user.
  - Recommendation: If user highlights text or has cursor over a word, clicking the link button should not just pre-populate that selected text into the field, but also immediately reveal internal wiki page 'search results'.
  - Recommendation: Consider merge between the insert link window as it exists in Wikitext editor and the Visual Editor link icon workflow, as the former clearly distinguishes and communicates to the user whether an internal Wikipedia article link is being added or an external link.
- Help within VE currently exists as a pop-out new tab to the user guide page, or a hover menu showing keyboard shortcuts. However, users can potentially click the help link on the left panel, which would prompt them to leave the editing environment.
  - Recommendation: This is a larger issue than just within VE, but brainstorm a way to have more in-context but non-obtrusive help within VE, and streamline where help can be found throughout the site.
  - Recommendation: A main goal of in-context VE help should be to clarify the conceptual confusion users have about citation / reference / (internal and external) link and sometimes footnote.

Testing Methodology / Raw Notes

Set-up

User will log in with test account to a 'Dog' article on wmflabs beta.
The beta page reflects the most up-to-date iteration of Visual Editor.
Users are directed to use Firefox or Chrome for the test.

Tasks

Introduction

Please sign into this page using the following credentials [redacted]. Now, please take a minute to familiarize yourself with this Wikipedia article about dogs. Let's say you know a great deal about dogs, and would like to make some changes to the content on this page. If you can't find something, or find something difficult don't worry. Just let us know what is happening by thinking out loud. If you are at a point of giving up, just move on to the next question. (We're testing the prototype, not you!) On tasks with longer descriptions, remember that you can click on the pin icon to keep the window open to view the entire task description.

Tasks

Let's get started. Where would you think to go to start editing this page? Please talk us through what you would do to begin editing.
If you are not there already, please press the "Edit" tab on the top right of the page. Be sure to click on the "Edit" tab, and not the "Edit Source" tab. Now, let's do some editing!
Let's say you'd like to add a fact about noses on bloodhounds to the "Smell" section of the article. Please find the "Smell" section and add the text "Bloodhounds have nearly 300 million receptors." at the end of the first paragraph in the section.
Typically, the subject of the page in the first sentence on an article is in bold. However, the term 'domestic dog' in the first sentence on this article is not in bold. Please make it bold now.
Additionally, the latin term in parentheses after "dog" on the page should be in italics. Please make the term 'canis lupus familiaris' italic.
Imagine you have some information about the tails of dogs, and want to add it to the article.

Please create a sub-section entitled "Tail", under the section title called 'Physical Characteristics'.

Here is some content for you to use in your new section:

There are many different shapes for dog tails: straight, straight up, sickle, curled, or cork-screw. As with many canids, one of the primary functions of a dog's tail is to communicate their emotional state, which can be important in getting along with others. In some breeds, such as the Braque du Bourbonnais, puppies can be born with a short tail or no tail at all.

As you may know, in Wikipedia, facts require citations. Please add a citation to the last sentence in the section entitled 'Hearing'. Here is the source website for you to use: http://www.k9puppydogs.com/html/dog_sense_of_hearing.htm
Now that you have added the citation, you would like to add the title of the website to the citation. How would you do that? Here is the title of the website: K9 Puppy Dogs
Internal wikilinks are used to reference other Wikipedia pages within an article. In the section titled 'Taxonomy', please add an internal Wikipedia link from the "Carl Linnaeus" article to the name 'Carl Linnaeus' in the first sentence.
Now, let's say you changed your mind and would like to remove this internal wikilink. How would you remove the link to the Carl Linnaeus page you just created?
Also, external links are used as references, and you can add a description for the reference. Let's add one. Please add the website "http://www.ancient.eu.com/article/184/" to the 'External Links' section, and add a quick description for the site as well.
Please finalize your edit.

Post-task Questionnaire

Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?

Test Videos / Questionnaire Responses / Raw Notes

Six participants (3 male, 3 female) completed the editing tasks through UserTesting.com, and their answers to the post-task questionnaire.

Raw notes on the six sessions can be found here.

jamariii
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. the question PopUp its blocking all my other webpage , so i cant fill it
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. no ive never edit or contribute a Wikipedia , its my first time
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. emm , i think that everything was in this experience from the Text to the links , so i guess i wont add something else
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. yeah i have no problem with it , you can share it
geert.je
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. Yes I will (is this a question?)
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. One time. long ago (5 years). As far as I remembered it was OK. Not difficult. I think I had to use more things that looked like (html)code but I'm not sure anymore.
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. I know you can edit all pages, but it is still a treshold to do it. Don't know really why... Maybe because all things I know I can contribute are from my professional organisation, and it feels a bit like making a page about your own when I should edit these pages. (wasn't there a rule that you may not make a page about yourself?)
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. Yes
RandyJackson
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. ok
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. no
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. Directions were not very clear, but all in all I understood how to make the edits.
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. Yes
amyp
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. I would be happy to give feedback, however it won't allow me to cut and paste the link
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. No
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. It's actually relatively easy to figure out how to use. I little more info with respect to the various icons found in the reference tools up top would be helpful
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. Absolutely
AllisonmayerWSA
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. Yes
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. No
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. There were some glitches as you saw. Popups that display when you hover a section while editing might be helpful to direct you to the right way of doing things.
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. Yes
Elena Tentis (UserTesting.com unintentionally deleted test video)
1. Would you please fill out this (very short) survey for us? It will give us some feedback on how you felt about the experience you just had? https://wikimedia.qualtrics.com/jfe/preview/SV_2tRziZhqpgWoiEZ
  1. Almost all the tasks I was given were handy, clear and in my oppinion easy to be done with not much experience in computers.
2. Have you ever edited or contributed to Wikipedia in any way (besides participating in this exercise)? If so, please tell us about your previous editing experience here.
  1. I never did, so this is why I din not expect to manage to finish almost all the tasks. I do though have used Word so because it includes editing like Office programs and email compose scrtions do - I called it handy.
3. Is there anything else you would like us to know about your experience with Wikipedia? or is something you wish you could do in Wikipedia but currently can not do? Please let us know here.
  1. From my experience, Wikipedia isone of the most reliable source of info I aways return to. Especially for all the controversed themes. I like tha fact that the information is always objective and I think the content should always stay like that no matter what articles we upload / write.
4. Do we have your permission to share this video with our volunteer community who also build and maintain Wikipedia?
  1. You most certainly can.