Guerrilla Testing, March 24, 2015: Description editing

Guerrilla Testing, March 24, 2015: Description editing

When/Where

Tuesday, March 24, 2015: Jared and Daisy went to Yerba Buena. Kaity and Vibha also helped with testing.

Goal

The goal of this research was to observe people interact with the CTA ("Tap to add a description!") line in the header under article titles in Wikipedia Alpha. After tapping the CTA, users experience 3 editable scenarios: 1. an meaningful description suggestion, 2. a blank form field, and 3. a random/irrelevant description suggestion.

1. Do users notice the CTA prompt? How effective are they in triggering action?
2. How do users feel about the CTA?
3. How effective are descriptions that are auto-generated and meaningful? Do they assist users with finalizing the description or confuse users as to why they are prompted to edit a description that is already automated and correct?
4. How effective is not giving a user a pre-populated description field?
5. How effective are descriptions that are auto-generated and random?
6. How do users feel about the process of editing description lines overall?

Questions

To get started, we asked people a few basic questions first:

1. What kind of phone do you use/what OS?
2. Do you use a tablet, and if so, what model/OS?
3. Do you use a computer, and if so, what model/OS?
4. Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not.
   1. If they answer, "Yes, I use Wikipedia", ask "how do you use it?”
   2. If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?”
      1. Ask if they feel there are any specific blockers to editing.

Protocol Tasks

We presented users with the Wikipedia Alpha prototype on an Android phone. They were then prompted to navigate to pages with the 3 different description results (we randomized presentation of the three scenarios during the test). At the page, they were first asked to describe the page to establish the discoverability of the CTA element, then to interact with it.

• Questions for pages with meaningful description suggestions: Sudan Golden Sparrow, Pablo Picasso, Eiffel Tower

1. Please navigate to the Sudan Golden Sparrow / Pablo Picasso / Eiffel Tower page.
2. (no need to ask again if already asked) Please describe what you are seeing on this page.
   1. (if they do not point out the cta) Scroll back up to the top of the page. Do you notice anything here asking you to take a particular action?
      1. (if still not) Do you notice anything in the header portion of this page that is asking you to take a particular action?
   2. How would you react to it if you saw it just browsing the app on your own?
3. (if they do not tap cta, direct them to do so) How do you feel when you see this pop-up?
4. How do you feel about this description?
5. What would you do at this point? (idea is to edit or finalize/exit)
6. How do you feel about that process?

• Questions for pages with a blank form field: San Francisco

1. Please navigate to the San Francisco page.
2. (no need to ask again if already asked) Please describe what you are seeing on this page.
   1. (if they do not point out the cta) Scroll back up to the top of the page. Do you notice anything here asking you to take a particular action?
      1. (if still not) Do you notice anything in the header portion of this page that is asking you to take a particular action?
   2. How would you react to it if you saw it just browsing the app on your own?
3. (if they do not tap cta, direct them to do so) How do you feel when you see this pop-up?
4. How would you describe San Francisco, in a short phrase?
5. How would you add that phrase to this page?
6. Can you show me?
   1. (if needed) Please finalize your entry
7. How do you feel about that process?

• Questions for pages with random/irrelevant description suggestions: any other page in Wikipedia Alpha (the description was set to display as "Mexican actor and voice actor")

1. Please navigate to any page about a topic you're familiar with.
2. (no need to ask again if already asked) Please describe what you are seeing on this page.
   1. (if they do not point out the cta) Scroll back up to the top of the page. Do you notice anything here asking you to take a particular action?
      1. (if still not) Do you notice anything in the header portion of this page that is asking you to take a particular action?
   2. How would you react to it if you saw it just browsing the app on your own?
3. (if they do not tap cta, direct them to do so) How do you feel when you see this pop-up?
4. Let’s say you know for a fact that [topic] isn’t about a Mexican actor. How would you edit the description?
5. Can you show me?
   1. (if needed) Please finalize your entry
6. How do you feel about that process?

• Final Follow-up Questions

1. What is your impression of this description area on Wikipedia pages?
2. Generally on apps, would you say you notice blank fields with italicized prompts?
3. If you were browsing Wikipedia on your own, how would you say you generally respond to seeing blank fields with italicized prompts?
4. Which one of the three description scenarios you experienced was most helpful?
5. Do you have any other feedback?

Findings: Patterns Observed

1. 3 of 5 users required some level of facilitator prompting to notice the CTA.
2. Interactivity breakdown:
   1. 2 users would most likely overlook this field, 2 users might notice/interact depending on the situation, and 1 user was not sure.
3. All users either specifically indicated field interaction was easy and intuitive or had no specific complaint or struggle that was observed. Only 1 user was briefly confused about the blank SF description field, thinking he couldn't type because he didn't see a blinking cursor.
4. 3 scenarios feedback breakdown
   1. Meaningful description suggestion
      1. most helpful: 2 users
      2. most helpful, but pointless because I can't see it on page: 1 user
      3. confusing: 1 user
   2. Blank
      1. fine if you know about topic: 1 user
      2. fine and having the CTA here made most sense: 1 user
      3. most engaging: 1 user
      4. easiest: 1 user
   3. Random/irrelevant description suggestion
      1. if visible, could prompt action: 2 users
5. 2 of 5 users expressed some level of confusion around why the CTA hides the description. One user was confused about why he was prompted to action when the description was correct on Picasso. Another user was confused about the same thing, and also mentioned that he would be much more likely to take action on the random article if he could see that the description was incorrect. The latter also mentioned that the CTA really only makes sense on the SF blank description page.
6. 1 user was confused about whether these descriptions were for himself or for all of Wikipedia.
7. No users indicated confusion about the CTA pop-up language.

Design Research Recommendations

1. Further iterations on this concept are recommended to 1) make it more readily apparent and/or inviting and 2) to logically streamline the display and workflow to clarify the purpose of the concept.
   1. Though users did not experience much difficulty once they entered the feature, most did not notice the CTA and afterward, most expressed ambivalence about whether they would act upon seeing it in practice.
      1. In the app, a description (or lack thereof/CTA line) can be displayed with an edit icon on the right. A few users pointed out the pencil edit button in this test, indicating a potentially higher awareness/recognizability factor with an icon instead of a 'tap to edit' text CTA.
   2. A user expressed confusion about whether these descriptions were displayed only for his view of Wikipedia or if they were meant to be displayed on all of Wikipedia.
      1. Language on the 'thank you for improving Wikipedia" can include a brief line indicating that the newly added/edited description is now live on the app.
   3. Some users did not understand the purpose of the auto-generated text appearing after activating the CTA; if a (good) description already exists, why am I prompted to add/edit it?
      1. The first window that explains auto-generated helpful text (I am assuming random/unhelpful auto-generated descriptions will be nixed at this point) can be either made more brief or more descriptive. Ideas on iterating the language can be provided upon further action on this feature. Given the generally positive reaction to a blank field (with no auto-generated text), that is also a viable option for pages that need descriptions.

Raw Notes

Test A

• Male (15-25)
• Android phone, no tablet, HP laptop
• Reads Wikipedia, used it mainly for research and general inquiry. Knew anyone could edit, but doesn't himself - doesn't feel it is his place to edit, doesn't want to touch other people's things

Task:

1. Pablo Picasso
   1. Describes the picture, various information in the introduction, pronunciation. Did not mention the CTA.
   2. After being prompted to review the CTA, he calls it 'interesting', but doesn't know how to feel about it because he is the one trying to look up information instead of input.
   3. CTA pop-up about auto-generated description makes sense
   4. Description is a lot of words that describe Picasso. Says he likely wouldn't edit it.
   5. Says it is interesting again, says that it is only a limited number of words, and not a full story. Then finds the full-page edit button.
2. SF
   1. Taps CTA, adds a short phrase about the city and taps done automatically.
   2. Says it was easy to edit the description of something he already knows.
3. How I Met Your Mother
   1. Confused about random description, says did someone edit this before me?
   2. Says that if he saw something wrong he'd want to fix it, but overall indifferent feeling about this sort of thing

Overall

1. Says he would most likely overlook an element like this
2. Would notice this only if he were in a particular mood, had the time and didn't have anything to do specifically
3. Says he thinks the blank is fine if he knew about the topic, but if not the auto-generated accurate description is best.
4. Says it is totally redundant if the lead paragraph is there, or you already know what you are looking for.

Test B

• Female (15-25)
• Android phone, no tablet, Dell computer
• Reads Wikipedia mostly for research. Knows about editing but doesn't, doesn't feel she knows enough.

Task:

1. SF
   1. Does not see the CTA. Second prompt, doesn't see it, but mentions the pencil edit icon. Final prompt, she mentions it.
   2. Thinks she is supposed to add a summary there.
   3. Types in a description, and taps done. Thinks it will change on the actual wiki page.
   4. Thought the process was easy.
2. Jeddah
   1. Taps in CTA, but then wants to scroll around the page. Confused about how to get out of keyboard/editing mode. Taps everywhere, and finally jumps out.
   2. Thinks maybe someone else put the random description there.
   3. Even though it's incorrect, says she usually doesn't edit so she probably wouldn't, but only slightly more likely just because it's incorrect.
   4. Doesn't feel any particular way about the fact that it was an incorrect description, vs the previous blank field for SF page.
3. Pablo Picasso
   1. Says it has a correct description already, and would just leave it be.

Overall

1. Feels the CTA area is easy to navigate and understand.
2. Whether she would notice it would depend on her level of interest.
3. Didn't have any particular opinion on which of the 3 scenarios was best.
4. Wondered if anyone could edit this field? 'Wouldn't the description for SF change every day?'

Test C

• Male (26-35)
• iPhone, unspecified tablet, HP laptop
• Reads Wikipedia, used it a lot more when back in college. Accesses mostly through search in Google, has used mobile web. Knows about editing, but is mostly interested in getting information

Task:

1. Roman Empire
   1. Describes background photo, title, and the opportunity to add description. Probably wouldn't think much of it.
   2. Understands the pop-up, would be for adding a description, a summary of some kind
   3. Thinks 'someone messed up' or that the random description is a joke.
   4. Edits the description, and hits done. Thinks it is very simple.
2. SF
   1. Hits CTA, but user doesn't see a blinking cursor in the form field and is confused. Does mention later that he noticed there was no longer placeholder text.
   2. Feels it is very simple after realizing that he could still type in the field. However, mentions that the last word in his description is not shown.
3. Pablo Picasso
   1. Says the correct description is already there. 'I'd leave it the way it is'.

Overall

1. Wonders, 'is [the description] just for me or for everyone?' Who edits this and monitors edits?
2. Don't recall this type of prompting on other apps and sites, so can't say how much he'd notice something like this.
3. Felt correct auto-generated was most helpful, blank was most engaging. Incorrect descriptions would probably make people want to fix it.

Test D

• Male (15-25)
• feature phone, no tablet, Lenovo computer
• Reads Wikipedia, tried editing when he was younger, but couldn't do it. Haven't tried it again yet.

Task:

1. Die Antwoord
   1. Tapped on the CTA right away, doesn't remember why exactly but remembered it asking for action. Clicks through very quickly, remembers seeing an incorrect description and thought it was a joke.
   2. When asked to provide a description instead of the incorrect text, he instead types in a description of what he's seeing and would like to see: 'a picture of the artist, summary...' Taps done without prompting when finished.
   3. Says the process 'seems fine'.
2. SF
   1. Hits CTA again quickly. Again, types in a description of the photo he's seeing and more of what he'd like to see: 'a bridge and a quick summary'.
   2. Notices no pre-filled text. Says it is nice not to have to delete anything but that it's 'still silly'.
3. Pablo Picasso
   1. Says the description is actually telling you about the person.
   2. Wouldn't change an existing description unless it was wrong or if I had to add one.

Overall

1. Felt that it was easy to interact with.
2. Whether he'd notice the prompt depends on if he's in a hurry. If not, then maybe.
3. Thought random was funny, blank was easiest, good description confusing (why do he need to tap to add/edit?)
4. Probably wouldn't be using this app, just generally.

Test E

• Male (15-25)
• Android, no tablet, Mac laptop
• Reads Wikipedia and uses it for looking up random things; knows about editing, but hasn't ever felt like it.

Task:

1. Pablo Picasso
   1. Scrolls through the page, mentions the photo and intro paragraph, section headings.
   2. Doesn't notice CTA until prompted to scroll back up to the top. Also mentions edit button.
   3. Understands the pop-up language, and feels the description is fine. Confused about 'am I supposed to change it? If it's correct, why can't I see it on the page instead of the CTA?'
   4. Doesn't edit, clicks done.
   5. Feels that that was 'unnecessary'.
2. Mac Dre
   1. Tapped on the CTA automatically and clicks through the pop-up.
   2. Surprised at the random description: 'he is definitely not that', and starts changing it. Taps done without prompting.
   3. Feels like that was better. Mentions again that he would've probably actually been more motivated to edit if he had seen the incorrect description on the page.
3. SF
   1. Hits CTA again. Seems a bit confused about not seeing pre-filled text, and says so.
   2. Is prompted to add a short sentence this time (instead of starting himself), and hits done.
   3. This (CTA) makes more sense though, he says, because there actually wasn't anything in the description this time.

Overall

1. Felt that it was easy to use.
2. He probably wouldn't care to interact with it and probably wouldn't notice it. However, he would edit a description if it was incorrect, and he had time and he cared about the topic.
3. The Picasso description was the most helpful but it was pointless because he couldn't see it on the page. SF was fine, and made most sense. If he could see the incorrect description on the Mac Dre page, he would be most motivated to edit.