Reading/Multimedia/Media Viewer/Research 2014

This report summarizes research findings for Media Viewer, a new multimedia browser aimed at Wikimedia readers and casual editors. This tool was developed by Wikimedia Foundation's multimedia team from July 2013 to December 2014 (see project timeline).

Key findings include image views, enabled rates, activity and performance metrics, as well as to usability tests and user feedback. This research was conducted to inform product development and learn more about the tool's usage, speed and overall impact.

See companion slides for a visual presentation of the research outlined below.

Overview

Goals

Media Viewer aims to:

improve the viewing experience for readers
make it easier to preview and browse images
show basic information, link to more details
help you share, download or embed images

Features

This annotated screenshot shows key features of Media Viewer's final user interface.
Ahu-Akivi Photo by Ian Sewell licensed under CC-BY-SA-3.0 via Wikimedia Commons.

Here are some of the features developed to support goals and user needs for Media Viewer:

enlarge the image
view the image in full screen
browse through related images
read the image caption or description
learn more about the author, source or license
get more details or edit information on the file page
share, embed or download an image
close Media Viewer
disable (or enable) Media Viewer

The effectiveness of these features was tracked through a range of research tools, from behavioral metrics to usability tests, as described below.

Research

We did extensive research throughout Media Viewer's development and after its release in 2014, to learn more about the tool's usage, performance and impact.

Research methods included:

activity metrics (e.g.: image views)
enable/disable rates (e.g.: % opt-outs)
performance metrics (e.g.: load times)
design research (e.g.: usability testing)
user feedback (consultations, survey, talk pages)

Throughout the year, research metrics and user feedback guided our product development, at each step of the way. Read on for more detailed findings on each method.

Key Findings

Here are our key findings from Media Viewer research in 2014:

Media Viewer serves a lot more images than before (17M intentional views/day - see definition)
Most users keep Media Viewer enabled (99.5% enabled)
Media Viewer key features were found easy to use
Media Viewer is more useful for readers than editors

Image Views

Image Views - Top Sites

Global image views average about 23 M views per day: they fluctuate between 20M and 26M/day, with fewer views on weekends.

The top 6 sites for image views as of Dec. 7 were:

English Wikipedia (9.0M)
German Wikipedia (2.7M)
Russian Wikipedia (2.0M)
French Wikipedia (1.6M)
Spanish Wikipedia (1.0M)
Wikimedia Commons (1.2M)

Monthly image views (712M) represent about 3.4% of monthly page views (20B) on all our sites.

Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image (see next section).

Media Viewer vs. File Page

Daily image views for Media Viewer and File Page, May-Dec. 2014. Source: WMF

Global image views for Media Viewer were about 23 million per day in November 2014, versus 3.4 million for file pages, for a combined total of 27 million daily image views.

Total image views nearly doubled from May to December, 2014, with 87% coming from Media Viewer and 13% from the file page. (Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image, as described in the next section.)

During that period, file page views declined by 73%, from about 12 million in May, before Media Viewer was launched on English and German Wikipedias, down to about 3 million by the end of 2014.

Note that these metrics do not include the variance from earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons, as we did not collect all these metrics prior to May 2014.

So the actual variance is likely to be higher, but would require more research to determine.

Intentional Views

Not all Media Viewer image views can be considered 'intentional' -- here's one way to separate them from each other:

'Intentional' views (~17M/day)

thumbnail + magnifier clicks
next clicks
hash loads
file page views

'Unintentional' views (~6M/day)

previous clicks
history navigation events

On that basis, 'intentional' image views can be estimated at about 17M (versus ~6M for 'unintentional' views).

Overall, how often users look at images increased significantly with the deployment of Media Viewer - the sum of file page visits plus 'intentional' Media Viewer views, grew by about 50% after the release on English and German Wikipedias.

(This does not include the growth from the earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons as we did not track file page visits at the time. So the actual growth is likely to be higher, but would require more research to determine.)

Enabled Rates

Percent of global logged-in users with Media Viewer enabled or disabled as of Dec. 8, 2014, by user group. Source: WMF.

Media Viewer was enabled for about 99.54% of logged-in visitors to Wikimedia sites as of December 8, 2014 (versus 0.46% disabled).

Those rates are lower for advanced users:

95.57% of active contributors had Media Viewer enabled (users with 5+ edits / month)
85.91% of very active contributors had Media Viewer enabled (users w/ 100+ edits / month)

These numbers were collected a month after a very prominent enable/disable button was added to Media Viewer, to make it easier for users who didn't want the tool to opt-out. The base group for each percentage is user accounts with a user_touched date or 5 edits or 100 edits in the last 30 days. (Local accounts belonging to the same global account are not deduplicated.)

Cumulative opt-out rates for logged-in users show more active users disabled Media Viewer from Aug. to Dec. 2014. Source: WMF

Enabled/disabled rates seem to be stabilizing across all logged-in user groups.

Opt-ins & Opt-outs - Logged-in vs. Anonymous

Global opt-out events by anonymous users started in July 2014 and varied as new features were released, with a downward trend by year-end. Source: WMF

Anonymous opt-outs and opt-ins increased in early September, after File Usage was removed, making the Disable/Enable links easier to find.

After the Enlarge feature was released in late September, global anonymous opt-outs dropped by about 60%, from 2.2k opt-outs per day down to 1k. The Enlarge feature addresses one of the most frequent user requests, enabling users to zoom in on images.

After a more prominent Disable/Enable button was released in November, anonymous opt-outs events nearly tripled, from 1k back up to 3k opt-outs per day -- and opt-outs by logged-in users also increased to an average of 100 opt-outs/day.

By the end of 2014, opt-outs for both anonymous and logged-in users had dropped by half again, from a peak of 3.6k down to 1.9k total opt-outs per day.

Opt-outs by logged-in users surged briefly in late August, after the default for Wikimedia Commons was switched to opt-in for logged-in users, to facilitate curation on that multimedia file repository.

Other feature changes did not seem to impact opt-outs by logged-in users, which were steady at about 65 opt-outs/day by year-end.

Opt-ins by anonymous users seemed less impacted by feature changes, representing about one-sixth of opt-outs by the end of the year. (Note that opt-ins by logged-in users is not shown, due to a bug).

Note that for technical reasons, it is quite hard for anonymous users to re-enable Media Viewer on most wikis, unless they do so right after disabling. So the opt-in events on this chart may be due to users trying out the disable feature and choosing to re-enable it.

Activity Metrics

Here are some of the user actions tracked through EventLogging in 2014. Please refer to the image in the Features section above to see how these actions are shown in Media Viewer's final user interface.

Top 5 Actions

Here are the top 5 actions for Media Viewer, tracked by number of daily clicks:

Open/Thumbnail clicks (10.3M)
Close (9.1M)
Next (6.8M)
Previous (2.5M)
History (4M)

Every day, we log an average of 10 million clicks on thumbnails in articles -- and almost as many clicks on the Next and Previous buttons. This new browsing feature did not exist before Media Viewer. Its widespread use suggests that Media Viewer may be adding value for millions of readers, who would not be using this browsing feature if they did not find it useful.

History tracks how often users navigate within Media Viewer using their browser's history.

Enlarge & Full Screen

To address frequent requests for a 'zoom' function, a new 'Enlarge' feature lets you click on an image to view its original file in full resolution: it is now used 12x more often (1M/days) than the full screen feature (93k/day), as of December 7, 2014. We also observed a dramatic decline in opt-out rates after this new feature was introduced (as shown in a separate graph).

This data and our design research suggest that the Enlarge feature significantly improved overall user experience by enabling users to view images in detail. In usability studies, the full screen feature is still found useful, because it lets you fill the screen with the selected image -- but not as useful as zooming onto the image.

Metadata & Details

This chart compares global daily events for opening (or closing) the metadata panel, versus clicking on 'more details' to go to file description pages: the metadata panel is now used 400x more often (1.3M/day) than the 'More details' button (~67k) in Media Viewer. Note that this total includes scrolling events to open the metadata panel (1.2M/day), as well as keyboard presses on the up arrows (130k).

It's also worth noting that the metadata panel open frequency is around 7% of image view frequency, so people actually use the panel fairly often (way more than the 'More details' button to access the file description page, which is ~0.25%).

Overall, this data suggests that the metadata panel is valuable to users.

Download, Disable & Share

The Download panel is opened more often (74k / day) than the Disable/Enable panel (26k / day -- labeled 'options' in this graph) or the Share/Embed panel (20k / day -- labeled 'use this file' in this graph).

Information Links

In the information that appears below the image, users click more often on Source links (7k/day) than on Author links (3k/day).

They also click a bit more on 'View terms' (4k/day) than on License links (3k/day - e.g. 'CC-BY-SA').

Predictably, links that are below the fold (e.g. Uploader, About) get fewer clicks than above the fold, because users have to scroll down or click on an ellipsis to open the metadata panel that contains them. It's also worth noting that clicks on source and terms decreased by half after we launched the captions above the fold, perhaps because that new information in the first two lines detracted from those links on the third line.

New Features

Global clicks on 'Enlarge', 'More details' and 'Download' surged after launch in Sep.-Oct. 2014. Source: WMF

After the new Enlarge feature was released, we observed a dramatic 20x increase in clicks for this popular 'zoom' feature, one of the most frequent user requests. By clicking anywhere on the image, users can now go to the original file and use their browser's zoom tool to enlarge image details.

The release of the the big blue 'More details' button also registered a 3x increase in clicks, compared to the Commons icon that was featured before (both link to the file description page).

And giving the Download feature its own icon increased clicks by 3x as well (it used to be a tab in a single 'Use this file' panel, along with 'Share' and 'Embed').

Performance Metrics

Media Viewer vs. File Page

Global image load times for Media Viewer vs. File Pages are on par, for either median or 95th percentile, Dec. 2014. Source: WMF

This chart compares image load times between Media Viewer and the File page in Dec. 2014.

It suggests that load times are about the same, with a median load time of 1.5 seconds to 2 seconds -- even though Media Viewer typically loads larger images than the File Page.

For the 95th percentile (the longest average load for 95% of users), both viewing methods are also on par, with load times in the 8-10 seconds range. Note that File Page performance improved significantly after WMF implemented a switch to HHVM in November 2014: prior to that, Media Viewer was loading images about twice as fast as the file page.

Network Performance

This chart shows the average duration of image loads per country, in milliseconds, as a reflection of network performance. Faster load times are shown in green, slower load times in red.

Overall network performance seems related to Internet bandwith in each region, with faster load times in developed countries than in the developing world.

More metrics

Attributions & Licenses

A growing number of images shown in Media Viewer now have machine-readable data for author, source or license information.

These ratios have been increasing steadily, as a percent of image views: 97% now have license information, 86% have sources and 86% have authors.

These improvements are related to a recent drive to clean up metadata for multimedia files across Wikimedia sites.

Design Research

The Wikimedia Foundation's design research team conducted a range of design research studies and usability tests for this project:

Study 1 (July 2014)
Study 2 (August 2014)
Study 3 (September 2014)
Study 4 (November 2014)
Usability tests (Feb.-May 2014)

User Study 1

Here are key issues discovered in our first design research study in July 2014:

Hard to discover info panel
Info panel covers images & is hard to control
Hard to find 'View original file' button
Hard to find 'Download' button
Clarify meaning of viewing options

See User study 1 slides (July 2014).

User Studies 2 and 3

In studies conducted in August-September 2014, users successfully completed all tasks listed below:

Primary Tasks

Preview images
Learn about images
Get more details
Browse images
Disable feature (passed 2nd test)
Re-enable feature (passed 2nd test)

Secondary Tasks

Enlarge
Share
Download
Go back

See User Study 2 Slides (August-September 2014).

User Study 4

In a final user study conducted in November 2014, we observed that readers were able to accomplish these key tasks with no problems:

View images in Media Viewer
Navigate to next / previous image
Read a long description
Identify details about images
Get more details on file pages
Zoom in to see details
View image in full screen
Share an image with a friend
Disable Media Viewer
Re-enable Media Viewer

See User Study 4 Report (November 2014).

User Feedback

Feedback channels

User feedback came from a variety of channels:

12+ discussions (in-person, IRC, hangouts)
20+ talk pages (mw.org, dewiki, enwiki, frwiki, etc.)
25K+ beta testers (for 6 months, throughout development)
18K+ survey responses (throughout release, in 8 languages)
30+ user tests and studies (throughout development and release)
5 requests for comments (2 RfCs on enwiki, 2 on dewiki, and commons)
1 community consultation (on meta.wikimedia)

Talk Pages

Here are some key issues reported in talk pages after release:

Make it opt-in
Hard to find file page link
Show caption higher up
Can't go back quickly
Image load too slow
Issues on tablets/mobile
Show full file name
Show more license info
Show label for author
Make image clickable

See Media Viewer Talk Page.

Requests for comments (RfCs)

Three community RfCs were held from June to October 2014:

All RfCs asked that Media Viewer be disabled by default.

For the most part, the WMF did not fulfill these requests, due to concerns about the lack of representation, but committed to more improvements, based on community consultation (see next section).

Note that the Foundation partially implemented the Commons RfC request, acknowledging the differences in usage patterns between Wikimedia Commons and "reader-facing" wikis: Media Viewer's default configuration was switched to 'opt-in' for registered users on that site in August 2014.

Community Consultation

A community consultation was held on Meta, from August 28 to September 7:

the goal was identify any critical issues
it was widely promoted on community channels
community members made 130 suggestions
WMF evaluated and prioritized all suggestions
WMF selected 10 'must-have' tasks
WMF developed 13 new improvements
all improvements were released and validated by Nov. 2014
see improvements list below

See Consultation Page.

Improvements

These improvements were made from Sep. to Nov. 2014, based on community consultation:

See Improvements Page.

Surveys

Users were asked for feedback in a short survey (April-July 2014):

optional survey (subject to self-selection bias)
built-in feedback button (opened survey popup window)
8 different languages (ca, de, en, es, fr, hu, nl, pr, pt)
18K+ survey responses (satisfaction ratings, some with comments)
a majority found the tool useful (but sample may not be representative)
a lot of actionable feedback (hand-coded to identify and prioritize key issues)

See Survey Report.

Survey feedback

Here are top feature requests from surveys:

Want zoom 14%
More image sizes 8%
Want opt-out 6%
Easy way to exit 5%
Scroll down for info 5%
Info cuts off images 5%
Can't find info 4%
Show whole image 3%
More info above 3%
Disoriented by UI 3%

(% hand-coded survey requests)

See Survey feedback.

Survey results

Here are the final survey results, based on 18,199 global responses as of July 8, 2014, when all surveys ended.

Note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. Editors are vastly overrepresented (half of the respondents self-identified as editors), the volume of responses is much higher in the first few days following deployments, and smaller wikis are overrepresented since the surveys have been running for longer there. The primary purpose of the survey was to collect user comments in order to improve the tool -- not to use approval ratings as a definitive measure of success.

Keeping these caveats in mind, 56% of global respondents found the tool "useful for viewing images and learning about them", with 35% responding "not useful" and 9% "not sure", as shown on this spreadsheet.

Readers responded more positively than editors, with 65% of them finding Media Viewer useful, based on 7,582 reader survey responses in 7 languages, over a 12-week period, from 14 April to 7 July, 2014 (see linked spreadsheet).

Weekly reader approval rates are shown here for each language. The drop in non-English rates in May-June may be due to non-English readers of Enwiki and Commons: the surveys are based on interface language, not content language.

Cumulative "useful" approval ratings from all users varied greatly by language: English 36%, French 70%, Spanish 78%, Dutch 59%, Portuguese 81%, German 30%, Hungarian 63%, Catalan 71%.

Overall approval rates from English users gradually increased in the month following launch: the weekly percentage of survey respondents who found Media Viewer useful ("yes") grew from 26% in May to 52% in June 2014. Note that the survey response rate fell at the same time, as shown in these results.

Again, note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. They are only shown here for the record, as they were frequently referred to in community discussions.

Thanks

This research was collected and analyzed for over a year by many different individuals, who worked diligently, above and beyond the call of duty, to help the development team make more informed decisions. We are very grateful to everyone who made this possible!

Research contributors include: Aaron Arcos, Daisy Chen, Gilles Dubuc, Fabrice Florin, Pau Giner, Mark Holmquist, Keegan Peterzell, Abbey Ripstra, Gergo Tisza.

Research advisors include: Howie Fung, Aaron Halfaker, Rob Lanphier, Erik Moeller, Toby Negrin, Dario Taraborelli, Lila Tretikov.

This report was prepared by Fabrice Florin and Gergo Tisza, on behalf of the team, from Dec. 2014 to Jan. 2015.

Overview

Goals

Features

Research

Key Findings

Image Views

Image Views - Top Sites

Media Viewer vs. File Page

Intentional Views

Enabled Rates

Opt-ins & Opt-outs - Logged-in vs. Anonymous

Activity Metrics

Top 5 Actions

Enlarge & Full Screen

Metadata & Details

Download, Disable & Share

Information Links

New Features

Performance Metrics

Media Viewer vs. File Page

Network Performance

More metrics

Attributions & Licenses

Design Research

User Study 1

User Studies 2 and 3

User Study 4

User Feedback

Feedback channels

Talk Pages

Requests for comments (RfCs)

Community Consultation

Improvements

Surveys

Survey feedback

Survey results

Related documents

Thanks