Reading/Web/Desktop Improvements/Features/Search/AB Test on New Search Location

In October, 2020, the Web team deployed a new, more prominent location for the search widget. The goal of this change was to make search functionality more prominent and easy to find for new and existing users. An A/B test was performed comparing the search usage of the old location versus the new location. This page presents an overview of the results of this test, as well as the full report and findings

Abstract

The Wikimedia Web team ran an A/B test from 20 October 2020 to 4 November 2020 to assess the impact of moving the search bar to a more prominent location on the top of the desktop page. The test included all logged-in users on the following early adopter wikis: Basque Wikipedia, French Wikipedia, French Wiktionary, Hebrew Wikipedia, Persian Wikipedia, and Portuguese Wikiversity. Results varied overall and by Wiki project. Estimates indicated that the new search location resulted in a higher search session completion rate overall. However, there was not sufficient evidence to definitively say that the new search location increased search sessions initiated.

Introduction

The Wikimedia Foundation’s Web team is working on researching and building out improvements to the desktop experience to make Wikimedia wikis more welcoming and to increase the utility amongst readers while maintaining utility for existing editors.

As part of this effort, the Web team deployed a new location of the search bar, moving the search bar to a more prominent location on the top of the page. The team ran an A/B test of the new location from 20 October 2020 through 4 November 2020 to assess the efficacy of this feature. The test included all logged-in users on the early adopter wikis (Basque Wikipedia, French Wikipedia, French Wiktionary, Hebrew Wikipedia, Persian Wikipedia, and Portuguese Wikiversity). In the test, 50% of logged-in users saw the search bar in the new location, while the other 50% continued to see the search bar in the previous location.

The new location (as shown in below) was also deployed as default for anonymous users on our early adopter wikis, and by preference for all other users.

The primary goal of the AB Test was to test the hypothesis that the group with the search bar in the new location will initiate more search sessions. The target was identified as 2.5% overall increase in search sessions initiated. The other primary questions we wanted to answer are:

Which group has a higher rate of search sessions completed? How does this differ per wiki?
Have any other interesting search trends emerged?
For logged-out users, are there any perceived changes in search behavior before/after the change?

Upon conclusion of the test on 4 November 2020, a total of 61,602 search sessions had been initiated across both groups. You can find more information on this change and other feature deployments on the desktop improvement project page.

Methodology

The AB test was a run on a per wiki basis and users included in the test were randomly assigned to either the control (search header in the old location) or treatment (search header in the new location) using their user ID and received the same treatment the duration of the test.

Data was collected in the SearchSatisfaction event logging table.

See the following Phabricator tickets for further details regarding the instrumentation and implementation of the AB test:

A/B test set-up T259250.
A/B test deployment T263032
SearchSatisfaction Instrumentation setup T251740 and changes T256100
AB test end T265333

Results

The complete results can be found within the complete report on this page. We saw an overall significant increase in search sessions completed, but no significant change in search sessions initiated between the old and new locations of the search. These results varied somewhat depending on the wiki tested.

On a per wiki basis, there were more completed searches (defined by total number of clicks over total number of searches) for the new search location than the old search location on Hebrew, Persian, and Portuguese Wikiversity, while the completed search rates were lower on French Wikipedia, Basque Wikipedia, and French Wiktionary.

Conclusion and Next Steps

While we did not see the expected rise in search sessions initiated, the rise in search sessions completed indicated that the new location is preferable when users are performing searches. We will continue rolling out the new location as the default on all pilot wikis within the Desktop Improvements project.