Talk pages consultation 2019/New user tests

The talk pages consultation set out to gather perspectives on communication from as many parts of the Wikimedia community as possible. For that reason, a lot of outreach was done to specific wikis and in many different languages, which gathered perspectives from many experienced Wikimedians. But it is also important to incorporate the perspectives of newcomers -- in other words, the future Wikimedians who aren't yet part of our communities, but that we hope will join us one day. Those Wikimedians may come from different cultures and have different expectations of technology than existing Wikimedians, and we don't want the current state of talk pages to keep them away.

In order to try to understand how new users feel about the current state of communication on wiki, we used UserTesting.com. UserTesting.com manages panels of people who receive compensation for recording themselves while testing software and talking aloud about their thoughts. For these tests, we recruited ten people who have never participated in wiki discussions to record themselves trying out real talk pages for the first time.

Goals

1. Test the discoverability of talk pages: how hard is it to even find the place to have a discussion?
2. Test the usability of talk pages: how hard is it to reply in conversations using the correct formatting, and to start new conversations?
3. Test the expectations for after they use a talk page: when will they get a response, who will respond, and how will they find out?
4. Ask users how communication could be improved: what would they change about the experience they had?

Choosing the testers

We wanted our testers to reflect the sort of people who would be likely to encounter talk pages. That would mean a certain amount of technical literacy, familiarity with Wikipedia, and to be someone who might want to edit. To narrow to those people, we asked a series of screening questions, such as "How often do you look something up on Wikipedia?", "Have you ever engaged in a discussion with other users on Wikipedia?" and "If you have not edited Wikipedia in the past, what would you say is the main reason why you have not edited?" Only if users gave all the answers we were looking for were they allowed to take the test.

All ten of the users in these tests used desktop Wikipedia in English. It is possible to run tests on mobile, and in other languages in the future.

Test procedure

The first test of five users was centered around a copy of the Mountaineering article from English Wikipedia. Users were prompted to go through multiple steps and talk aloud about the experience. A summary is below.

1. Make a small edit to the copied article itself, to make sure the user understands how editing works in general.
2. Talk about how they would find the discussion area.
3. Go over to the talk page and explain what they are seeing, including WikiProject templates at the top.
4. Add a comment to an existing conversation, with prompts to figure out how to indent and sign their comments.
5. Add a new topic to the talk page with a signature.
6. Talk about their expectations for when they will get a response and how they will find out about their response.

Results and quotations

Below are aggregated notes from the five interviews, along with some select quotations in italics directly from the user tests.

• Overall
  • None of the five testers expressed surprise or indignation at the way on-wiki communication works. That said, they were all confused at times, and all suggested a general modernization. "It would be nice if it was more like an actual message board.  A bit more simple."
  • All the users generally expressed that on-wiki communication felt "old". "It seems more old school and html-like. It's fine but it's definitely old fashioned."
  • There are many unintuitive elements of talk pages that users can figure out after some experimentation. An important question is whether those initial moments of confusion drive potentially good contributors away. "It will bring me to a web-based forum full of threads.  But this is not at all what I thought it was going to be.  That's kind of weird, isn't it?"
• Discoverability
  • All users struggled to find the talk page. Most thought that clicking "edit source" in an article section would lead them to a discussion forum about that section. None of them found the "Discussion" tab at the top of the page. This may show an expectation that users have about being able to discuss specific sections of an article, instead of the article as a whole. "Yeah, I guess I would click 'edit source' again, because I don't really see any other place I could go."
  • Similarly to the point above, once on the talk page, many users thought the headers of the discussions corresponded to the sections of the article. In other words, they thought that discussions about articles were organized around article sections.
  • After looking at the WikiProject boxes at the top of the talk page, no users correctly intuited what a “WikiProject” is, and none of them gave a good explanation for why it would be on the talk page. "The tan boxes are telling me that the information from this page can also be used within those other two pages."
  • To describe talk pages, they all used phrases like "message board" or "discussion forum", and assumed that there would be threaded discussions like on other websites they are used to.
• Usability
  • Some users thought that the correct way to join a discussion would be by clicking the "talk" link in a user's signature. They thought that was like a "reply" link. "I think I would click on talk, this button right here."
  • All users perceived that they would join the discussion by typing their comment at the bottom of the section.
  • All users realized that using colons would indent their reply. They realized this because the test had them reply to a conversation that already had several indentations. It seemed unlikely that they would figure out how to indent without having an example to follow.
  • Two of five users were able to figure out on their own how to sign their comment. One did this by clicking the signature button in the editing toolbar, and the other did this by noticing the "use four tildes" prompt that is present on English Wikipedia talk pages. The other users intended to sign by just typing their user names. "Okay, so I didn't even know you could do that."
  • All users commented that it would be better if signatures and indentation were done automatically. "Be a bit more up-to-date with easy one-click access to reply and add signatures. Use verbiage like 'click to discuss' instead of 'edit source'".
• Expectations for after posting
  • All users perceived that their comments were in public and would be seen by anyone who visited the page.
  • Users understood that the time that it would take to receive a response varies by how busy the page is.
  • Many users expected to find out about replies either through their on-wiki notifications or through email.
• Suggestions for improvement
  • All users commented that it would be better if signatures and indentation were done automatically. "I feel like it should automatically indent them.  I don't feel like I should have to do the colons in order to do that.  And I feel like there should be an easier way to add your signature at the end, because I don't feel I would have known how to do that without doing this user test.  Otherwise I think it was fairly easy to reply."
  • Users commented about how it should be easier to find the talk page.
  • Several users said that this format felt "old".
• Other interesting moments from the tests
  • One user wasn't sure whether "edit source" on the talk page would open the editor for the talk page or for the article. "I don't know if that actually will edit the source inside the [section on the page], or you can actually talk inside of here."
  • One user expected that clicking on someone's username would show what other discussions the user is involved with. "Maybe we could click on them and see what other conversations they're involved with."
  • One user wasn't sure whether newer comments were at the top or the bottom of a thread.
  • One user wasn't sure whether they would be alerted if their comment was answered. It shows that there is potentially some work to be done to orient users when they are first engaging in discussions so they know what to expect.

Next steps

Having completed these tests, there are a few other elements we want to learn more about in some future tests. The tests done so far use minimal WikiProject templates at the top of the talk page that are only a couple lines. We want to see how much more confusing it is when a talk page has much more non-talk content on it, like extensive WikiProject templates, warning templates, instruction templates, and archives.

May 2019 supplemental tests on templates

After analyzing the first round of tests above, we decided that we want to learn more about the effect of article metadata (e.g. WikiProject templates) at the top of talk pages. Article metadata is frequently not directly related to discussion, and so we wondered whether it confuses how newcomers understand and interact with talk pages.

To test this, we did five more interviews with almost the same protocol as the first set, except the "Mountaineering" talk page included in the test now had full WikiProject templates, along with many of the other templates that one might find on a highly-trafficked article talk page. The talk page for Earth is an example of a page that includes a lot of templates above the discussion.

The results makes it seem like article metadata can be confusing and distracting for newcomers, because they click on something that says "Talk" or "Discussion", but they end up looking at content that is not a discussion. It's possible that some newcomers then navigate away from the page without seeing the discussion at all.

Results and quotations regarding templates

• For most users, templates at the top of the talk page seemed out of place. “I honestly didn’t expect [all this information] to be here."
• Several users became focused enough on the templates that they did not scroll down to the discussion without being prompted. In other words, they seemed to believe that the templates themselves were the full extent of the talk page. “At first I thought it was a discussion, but in reading it it really doesn’t look like a discussion to me.  I don’t know what this is."
• It seemed that the users who were most confused perceived the colorful templates to be the important content on the page, and the black-and-white discussions below to be footnotes or less important content. For talk pages with a long table of contents, there is a good deal of whitespace under the templates and above the discussion, which may contribute to this effect.
• In reading the templates, users generally understood that they conveyed metadata about the articles. “It's a breakdown of the article as a whole.  Facts about the article.”

Other results and quotations

Since these supplemental tests followed the same protocol as the original tests, we also gathered more observations in general.

• All users in both tests except one struggled to find the Discussion tab at the top left of the page (on English, a left-to-right wiki). They generally searched in the upper right of the page, thinking that the "Talk" link (for their own user talk pages) was the right place to ask a question.
• When asked to sign their post, one user out of five noticed the text at the top of the edit window instructing them to use four tildes; the other four didn't.
• Like the users in the previous tests, these users frequently assumed there would be a general correspondence between headers in the article and headers on the talk page.
• Like the users in the previous tests, several of these users thought they should click to edit the article in order to find the discussion area. Some also suggested visiting "View history" to engage in discussion.
• The two points above potentially reflect an important association between editing and discussion that the current interface does not promote. In other words, it is generally healthy for discussion to accompany editing, so that people can collaborate and make sure they are coming to the best result. But these two activities are found in different places -- a user can't discuss while they edit, and the pathway to discussion is no clearer from the editing interface than from the reading interface. For instance, when prompted to start a discussion, one user clicked to edit and then clicked the "?" icon in the editing toolbar.