Reading/Web/Desktop Improvements/Repository/Sentiment Survey

Throughout the development process of Vector 2022 we have collected various forms of feedback and data, mainly regarding usability. We recently collected a different kind of data via a public survey — sentiment data, or first-impressions data — which allows us to understand first-impressions the general public might have upon launch. It is important to remember that this isn’t usability data, it is rather a measure of people’s feelings towards change.

Based on sentiment after large design changes on other websites, as well as various articles about resistance to change, we expect there to be a significant amount (but not a majority) of negative initial responses to the skin. Our results are in line with this expectation. Actions we can take in this regard are:

maintaining realistic expectations as a community of the potential of a large number of negative public reactions
developing in-product onboarding, and other reference materials, to help the general public get used to the new experience, and understand the reasons for the change

Introduction

A survey was run, focused on logged-out users, with the goal of identifying initial sentiment towards the new experience upon deployment. The initial goal of the survey was to provide insight into overall sentiment and usability. However, due to the experimental structure, only responses after seeing the skin once were collected.

The survey showed that the majority of logged-out users received the change with either positive or neutral sentiment.

However, a large portion of users reacted negatively to the proposed changes, with these negative reactions being generally related to resistance to a change in habits. This is in line with similar design changes across other websites. In order to help users with the transition, the team will focus on onboarding and better-preparing logged-out users for the upcoming changes.

Experimental Setup

The new Vector 2022 skin was displayed on a set of 10 previously selected pages on English Wikipedia. On these pages, a Central Notice banner was displayed informing users that they are seeing the new skin and linking to a survey asking them about their thoughts.

The survey asked questions about overall experience, usability, and welcomeness of the new skin, compared to the old skin. Responses were collected reflecting the thoughts of each survey respondent.

Issues with current experimental setup

We believe that the results of our survey might have been affected by a number of factors:

Insufficient usage: respondents taking the survey were only allowed a single pageview and a static screenshot of the new experience, which is likely insufficient to be able to answer questions around usability
Vandalism: many responses included information not relevant to the questions asked, foul language, and other instances of vandalism.
Survey length: we saw many instances of incomplete survey results. This could be due to the length of the survey
Lack of clarity in questions: we saw many instances of incomplete survey results. This could be due to lack of clarity or lack of following best practices for positive sentiment analysis.
Page selection: because we were unable to randomly select the pages which displayed the new skin, it’s possible that bias was introduced by the selection of pages. (For example, the most popular page within the set was the Wikipedia page on Wikipedia)
Uneven sampling of target audiences: the majority of our respondents reported daily usage of Wikipedia. We were hoping to receive more responses from users that use Wikipedia less frequently to counteract potential bias based on familiarity, and to ensure the skin is received positively with the main target audience, which is new readers and editors

We will be reviewing these questions with our research team for extra validation. If we see significant concerns for any of the above biases, we will correct in the follow-up survey.

Results

Out of the 550 responses received we removed the following:

Vandalism: instances of survey responses that contained unrelated content or foul language
Unfinished responses: responses which did not answer all of the questions within the survey

This gave us a set of 152 valid responses. See section above for more information on what we would like to investigate in future surveys in order to ensure more of our data is usable and accurate.

Usability of the new skin

The majority of respondents reported that the new experience is easier to use or that the new and old experience are equally easy to use.

86 responses reported the new experience as easier to user or the new and the old experiences as equally easy to use. Of these, 49 respondents reported that they find both skins equally easy to use and 37 respondents reported that they find the new skin easier to use

Frequently reported factors for making the new skin easier to use:

Improved organization and page hierarchy
Content is easier to read
Less clutter on the page
Easier access to information
Cleaner design
The new table of contents, and less time spent in scrolling

60 responses reported the old experience as easier to use

Frequently reported factors for making the old skin easier to use

Familiarity with the old design:
One-click access to certain tools
Higher content density

Overall, users’ feelings towards the perceived usability of the new skin were positive ( with an average of 6.4/10).

Welcomeness of the new skin

The majority of respondents reported that either the new experience is more welcoming or that the new and old experience are equally welcoming

81 responses reported that the new experience is more welcoming or that both experiences are equally welcoming. Of these, 37 respondents reported that they find both skins equally welcoming and 44 respondents reported that they find the new skin more welcoming

Frequently reported factors for making the new skin more welcoming:

Less clutter and visual noise on the page making the page format easier to understand
Cleaner look and feel, lack of content and tools that are not important
Improved style
Better understanding of the context of an article
The limited width leading to better readability
Values function over form

65 responses reported the old experience as more welcoming

Frequently reported factors for making the old skin more welcoming:

Less clutter and visual noise on the page making the page format easier to understand
Cleaner look and feel, lack of content and tools that are not important
Improved style
Better understanding of the context of an article
The limited width leading to better readability
Values function over form

Overall, users’ feelings towards the perceived welcomeness of the new skin were neutral to positive ( with an average of 5.65/10).

Overall satisfaction

Overall, users’ feelings towards the new skin were neutral to positive ( with an average of 5.15/10), with many users strongly preferring the new or old skin.

Resistance to change

The large majority of respondents with negative experiences (43 out of 60 for ease of use and 43 out of 65 for welcoming) self-reported that their opinions of the new experience were affected by the following:

Nostalgia and familiarity with the old experience
Resistance to change

Our next steps will be to focus on preparing logged-out users for this change by using banners, previews, and other onboarding methods.

Example responses:

“Again, probably for the best, but I will still miss the old clunky interface”

“I don't like change. It's sleeker, but that might not be a good thing.”

“I hate change and love the old way, but if it must change I am perfectly fine with it. It's sleek and easier to read. Best of luck!”