Flow/Moderated Testing, November, 2014: talk pages and Flow

Goals for Moderated Sessions

To better understand from new/casual contributors and readers how they experience existing talk pages and Flow discussion pages, we are spending an hour with 5 participants of each demographic over a few weeks. Here are some of the things we are looking to learn:

How do new/casual contributors and readers experience existing talk pages and Flow discussion pages?
What is easy for this two groups to do in existing talk pages and Flow discussion pages?
What could be made easier for these two groups in existing talk pages and Flow discussion pages?

Discovery Process / Testing Methodology

Define the users to invite to participate:
1. "New/casual contributors" = less than 100 edits total
2. "Readers" = never edited
Design research protocol and structured data collection for the research sessions
Find the right participants who are willing to volunteer an hour to collaborate with us
Implement 10 hour long research sessions and collect data; usability testing and discovery of people’s experience
Analysis of data: review collected data and look for patterns, insights, and findings
Synthesis: what does the data mean for the design of existing talk pages and Flow discussion pages?
Report findings: results can be found on this page
1. Recordings of the research sessions with participants who opted to share with everyone are posted below

Participants and Links to Video

We talked with 10 people, a few of whom were willing to share their research sessions recorded via Google Hangouts on Air:

Pilot: Rachel DiCerbo (pilot - video not retained)
New/casual contributors:
1. Jan
2. Dan (requested restricted release)
3. Eric (requested restricted release)
4. Elias
5. Rich (requested restricted release)
Readers
1. Jerry (requested restricted release)
2. Meg (video not recorded)
3. Tony (requested restricted release)
4. Rosemary
5. Kat (requested restricted release)

High Level Findings

Generally, participants were more able to successfully navigate and accomplish tasks asked of them in Flow vs. Talk pages. Because wikitext has a learning curve, and some participants don’t even think to use it, they are less likely to follow best practices for participating in discussions on talk pages.

Participants who haven't edited before don't know about or understand the need for indenting and signing within wikitext editor unless they see the call to action. The outcome of this is that those unfamiliar with editing will just add a comment without indenting or signing, potentially irritating others. Flow allows those who don’t know about indenting and signing to do so anyway, because in Flow these actions are completed automatically. (This is true for 9/10 participants - only one new/casual contributor knew to both indent and sign and how to do so in wikitext.)

Notifications is a complex functionality, and though we attempted to test their awareness of how to be notified, where the notifications could be found, where to set preferences for what type of notifications to receive and through what means, etc., we didn’t have time to do that for everyone. As a result, we only know that it is complicated - there are many ways and locations on the site to set notification settings and receive notifications - and we need to do more research to be able to provide clear recommendations and findings in this area.

Talk Pages vs. Flow

Those who have never edited often do not have any idea they can even have a discussion with the other contributors regarding an article. They have no mental models to work with about editing or having a discussion in context about an article. It is all brand new to them, and they are in discovery mode.

When they are looking around for how to have a conversation with the other contributors to an article, they get confused and distracted by all the calls to action and lack of information hierarchy of a discussion/talk page, and of Wikipedia in general. So, many times, readers with no talk page experience get lost, and don’t understand how to accomplish the task until they have been patient enough to do some trial and error and discover what to do. As this is research, and we are asking them to “start a new topic” on the discussion page, they are patient and try to figure it out. In their normal use of Wikipedia, it is possible that readers may just give up as a result of the amount of work it will take to learn, and as a result, decide not to participate in the conversation (or never find out there is a way to engage others in this way to begin with).

It would be helpful to new users to alert them that they can discuss the contents of the article with other people interested in and contributing to an article. Also, there need to be clear calls to action so people know where to go to accomplish this. In this case, the design should get out of their way and provide a path to contribute. Currently, discussion/talk pages put all kinds of bumps in the road (distractions, wikitext to learn, confusing contexts) on the way to a conversation with editors. It would be helpful to remove those bumps in the road and make it easier to contribute. Flow mostly accomplishes this in general and for the specific tasks we tested in sessions.

Color Codes for Findings

Red = Show-stopper; users cannot accomplish the task. Issues prevent completion or cause users to give up or abandon task.

Yellow = Users can accomplish the task, but it is harder than it needs to be. There are usability issues.

Green = Users can easily accomplish task, minor/no issues.

Findings: Tasks on Talk Pages

Red (Show-stoppers; users cannot accomplish task and/or give up)

For Readers: Readers didn't know where to go to start the conversation on a talk page. They hadn't thought of the concept of discussing things with authors, and can’t figure out how/where to start the conversation. (4 of 5 readers)
- Recommendation: Provide information to people about what they can do on an article page (edit / talk with editors, etc)

For Readers on both talk and Flow pages: People get confused about whether to "talk" with an author on their user "talk" page or the article's "talk" / "discussion" page. This can confuse them enough to give up.
- Recommendation: Provide in-context (or otherwise easy-to-find) help about best practices for discussion pages.

For Readers: The big yellow infoboxes at the top of talk pages distract and confuse people. They see wikiprojects and go to the discussion pages of those, getting completely derailed. They are large and brightly colored, and sometimes people don't go beyond them. (Note that there were no yellow boxes on the test pages for the new/casual contributors, so we didn’t have any data about this for that set of participants.)
- Recommendation: Instead of yellow infoboxes being so prominent, there could be guidance or a prominent link to a best practices or help page. The information currently in the yellow boxes could remain nearby but be made less visually prominent - perhaps collapsed for more experienced editors or similar. They currently are so prominent they distract people from most important actions on a talk page (start a new topic, reply, edit, etc).

Yellow (Users can accomplish task, but there are usability issues)

For New/casual Contributors on both talk and Flow pages: People get confused about whether to "talk" with an author on their user "talk" page or the article's "talk" / "discussion" page. They were more likely to keep moving until they figured out what to do and had more experience with the context, rather than possibly get lost in a trail of links and give up like Readers. However, this confusion generally didn’t stop this group from accomplishing the task.
- Recommendation: Provide in-context (or otherwise easy-to-find) help about best practices for discussion pages.

For New/casual Contributors and one Reader: "Username (talk)" within a topic is confusing, because it also says "talk" like the "talk tab". People sometimes get distracted by this and go to the user page thinking that is where they need to post.
- Recommendation: Take some time to define clear naming conventions for user talk pages / article talk (and Discussion) pages. This should be consistent from wiki to wiki; this can be explored further during the UI consistency project.

For Readers and one New/casual Contributor: Didn't know any wikitext mark-up or even to look for it.
- Recommendation: Provide clearer in-context guidance for wikitext mark-up or enable Flow for those who are new to contributing to wikis.

For New/casual Contributors and one Reader: Knew about markup generally but were not very proficient in it (they needed to look up the syntax to indent or sign).
- Recommendation: Provide clearer in-context guidance for wikitext mark-up or enable Flow for those who are new to contributing to wikis.

For both New/casual Contributors and Readers: People are not aware of customary ways to post on discussion pages; they are not sure of culture and best practices (e.g. contact user on his/her talk page vs. adding content directly to article topic talk page, that one should sign posts and indent replies, etc.). This could cause irritation to experienced editors.
- Recommendation: Provide in-context (or otherwise easy-to-find) help about best practices for discussion pages.
- Recommendation: Instead of yellow infoboxes being so prominent, there could be guidance or a prominent link to a best practices or help page. The information currently in the yellow boxes could remain nearby but be made less visually prominent - perhaps collapsed for more experienced editors or similar. They currently are so prominent they distract people from most important actions on a talk page (start a new topic, reply, edit, etc).

For 1 New/casual Contributors and 3 Readers: "Add new topic" tab is a good call to action, but not everyone finds it because it is removed from where the topics are.
- Recommendation: Consider whether this is the right place for this call to action. It would be easier to see as a more in-context (of the talk page) call to action.

Green (Users able to complete task, minor/no issues)

Every participant was able to add a source (by replying to an existing talk page topic with moderator-provided source content) on the talk page. People were more familiar with the context at this point, so knew more about what to do versus with the first task of adding a new topic for the first time.

Every participant was able to edit his/her own contribution to a topic on a talk page (changing Sam to Simone).

Every participant was able to change the title of an existing topic on a talk page.

Findings: Tasks on Flow

Red (Show-stoppers; users cannot accomplish task and/or give up)

For Readers on both talk and Flow pages: People get confused about whether to "talk" with an author on their user "talk" page or the article's "talk" / "discussion" page. This can confuse them enough to give up.
- Recommendation: Provide in-context (or otherwise easy-to-find) help about best practices for discussion pages.

Yellow (Users can accomplish task, but there are usability issues)

For New/casual Contributors on both talk and Flow pages: People get confused about whether to "talk" with an author on their user "talk" page or the article's "talk" / "discussion" page. They were more likely to keep moving until they figured out what to do and had more experience with the context, rather than possibly get lost in a trail of links and give up like Readers. However, this confusion generally didn’t stop this group from accomplishing the task.
- Recommendation: Provide in-context (or otherwise easy-to-find) help about best practices for discussion pages.

For both New/casual Contributors and Readers: There was some confusion about the reply link vs reply box. The difference between the reply link (which nests and indents the reply text entered) and the reply box (which generally replies to the talk topic itself and does not indent the reply text entered past the original topic poster's original content/message) is not necessarily clear to users. People sometimes look more carefully and use the reply link (as was appropriate for the task for entering the moderator-provided source information), but most do not note a specific difference. Thus, there could possibly be inconsistency in people indenting or not.
- Note: 4 of the New/casual Contributors used the reply box. 1 New/casual Contributors used the reply link under the topic title, which takes user to the reply box (this participant was also confused by the text prompt "submit the form to finish posting this" after selecting "Preview", but the finalizing button reads "Reply", not "Submit").
  
  3 of the Readers used the reply link, 1 Reader used the reply box only, 1 Reader used both the link and the box and discovered the difference in functionality.

Green (Users able to complete task, minor/no issues)

Every participant was able to start a new topic successfully. 5/10 had no problem and the other 5 had some difficulty, but were successful pretty quickly.

4 New/casual Contributors and 3 Readers preferred the visual design and layout of Flow to talk pages.

3 New/casual Contributors and 3 Readers were able to edit their contributions with no issue.
- Note: 1 New/casual Contributor was not directed to perform this task (moderator error). 1 New/casual Contributor was not able to complete the task due to time constraints. 1 Reader was not able to complete the task due to time constraints.

3 New/casual Contributors and 5 Readers were able to edit the title of a topic with no issue.
- Note: 1 New/casual Contributor was able to figure out this task after a little difficulty/trial and error). 1 New/casual Contributor was not able to complete the task due to time constraints.