Wikimedia Research/Design Research/Reading Team UX Research/Top Chrome Usability

Study authors: Sherah Smith & Nirzar Pangarkar

Stakeholders: Mobile Web

Phabricator Ticket: https://phabricator.wikimedia.org/T147641

Study Plan

Background

In ongoing efforts to reshape the mobile web browsing experience, some branding changes have been taking place. A series of tests are going to measure the usability of said changes.

Goals

Gather usability feedback: We want to make sure that the Menu and Search buttons on the left side of the top bar on mobile web are easily switched between, and that the user’s goals for each action are readily and easily achieved on the first try. This will inform how we iterate the layout of the top bar on mobile web.
Gather branding feedback: Do users recognize the brand? Do they recognize they are inside Wikipedia?
Another side-goal is to test out the execution of smaller usability tests, in order to streamline the process and make it easier for teams to rely on simple, quick feedback from the UserZoom platform.

Research Questions

Do users recognize that they are inside Wikipedia? (Assess brand awareness level)
Can a user navigate to the side menu on first attempt?
Can a user navigate to a search on first attempt?
Which is more common: users try to tap the magnifying glass to initiate a search, or users tap inside the search box to initiate a search?

Methodology

UserZoom remote unmoderated usability test. 8 participants.

Participants

Readers, half men, half women. Minimum 6 participants.

Schedule

Gather info/assets: 10/17-10/19
Build test inside UZ & get designer signoff: 10/20
Recruiting: 10/20-10/21
Test results come in/QA’d: 10/24-10/26
Analysis/Synthesis: 10/27-10/28
Delivery: 10/31 or first available date w/stakeholders

Protocol

(Starts with this page: https://trending.wmflabs.org/en.wikipedia/Halloween) Tell me, out loud, a little about what you’re seeing here on this page. Take a minute to poke around a little.
Thanks for your descriptions. Next, please search for the word “Sloth”. Say aloud what you are doing, and describe what happens when you have performed the search for “Sloth.”
Now search for “Octopus.” Say aloud what is happening, and describe what you see after you have performed the search.
Now go to the main menu. What are the options listed in the main menu, and (without selecting the options) what do you think they would lead you to?
Now close the menu. Click on any interesting link in the page you see when you close the menu. Describe where you are on the internet, and what the pages you’ve been viewing are for.
Open the main menu once again, and then close it. What do you think about the menu?

Follow up Questions

Did you think it was easy to search for something?
What did you think you were testing today?
Do you have any feedback about what you saw today?

Study Findings

Answers to Research Questions

Do users recognize that they are inside Wikipedia? (Assess brand awareness level)

Based on the feedback from the video recordings as well as survey responses, we feel confident that participants do understand that they are using Wikipedia, rather than browsing "the Internet" or "a search engine."

Can a user navigate to the side menu on first attempt?

Yes. There were no issues locating the menu. One out of 16 people had issues understanding how to close the menu, so we are rating its usability as an A+.

Can a user navigate to a search on first attempt?

Yes. There were zero issues locating and using the search bar throughout the tests.

Which is more common: users try to tap the magnifying glass to initiate a search, or users tap inside the search box to initiate a search?

Only one person tapped the magnifying glass inside the search bar on Prototype A. There were no usability issues with tapping into and using the search bar throughout the tests.

Final Questionnaire Answers

There are two batches of answers. The first tables come from Prototype version A tests, the second from Prototype version B.

1. Did you think it was easy to search for something?

Answer
Yes
Yes. The search was at the top of every wiki page, so it’s easy to search for what your looking for, because you never have to exit the page you’re on. Also, whole you’re typing, wiki looks through its database to find what you’re possibly searching for. This is especially good if you’re not sure how to spell something, but you have the 1st few letters.
Yes it is as easy as any other search on the internet.
Yes
Yes, and I loved the suggestive searching.
Yes
Yes
It was easy to search but it took a long time to load.

Answer
Yes
It’s super easy to search for things. No hassle whatsoever
Yes it was easy to search
Absolutely
Yes
Very easy
Yes
Yes it was very easy to search for articles on topics I was interested in.
Yes

2. What did you think you were testing today?

Answer
The look and accessibility of mobile Wikipedia.
The Wiki mobile website, and the ease of navigating and searching using it through Wikipedia and its database.
I think I was testing the ease of a Wikipedia search and the main menu.
User experience of the web app
Wikipedias new memu.
Website usability
Wikipedia’s search engine
The Wikipedia site

Answer
The Wikipedia mobile page
I’m not sure. Just a new Wikipedia home page
The search engine for Wikipedia
The mobile page for Wikipedia. I believe I was testing usability, functionality, layout, navigation
Usability
Wikipedia search function
Functionality of the Wikipedia site
The main search and navigation settings within the mobile site for Wikipedia.
New menu and features such as random and article of the day

3. Do you have any feedback about what you saw today?

Answer
No.
Nope. I think the mobile page is great!
I was confused a little bit on question 5 and 6 of the test referring to main menus, links and where I was on the Internet but maybe I was doing right after all. I was not %100 sure though.
Wikipedia web app is mobile friendly, it could be better in terms of responsiveness. Loading pages happened slowly. There was a small bug when I typed in a query and it didn’t change the UI.
I loved the new look Wikipedia. Clean design, and I love how they are trying to give Wiki more of a reddit feel.
Make directions more specific
It was a bit slow
The main menu was vague.

Answer
I didn’t clearly see an option to complete the last task of going to the homepage and closing it. I was a bit confused.
Nothing really
No feedback
I really enjoyed today’s study,
No
I loved the options that pop up as you’re searching
Fairly easy to navigate and find articles on this website.
Everything worked very easy, and the navigation was simple, clean, and effective.
Looks good