Article feedback/Version 5/Report

Overview

A new experimental version 5 of the Article Feedback Tool (AFT5) was developed by the Wikimedia Foundation from November 2011 to June 2013. The main purpose of this experiment was to increase participation on Wikipedia by inviting readers to leave comments on article pages. To that end, we tested an updated version of this tool in three different pilots on the English, French and German Wikipedias. (see pilot results below)

Article Feedback seemed effective for engaging readers: 70% of survey respondents liked the tool; 2.7% of invited readers registered after leaving feedback -- and 3.0% of invited readers completed an edit. Over a million comments were posted during this experiment: on average, 12% of posts were marked as useful, 46% required no action, and 17% were found inappropriate by Wikipedia editors.

However, a majority of editors did not find reader comments useful enough to warrant the extra work of moderating this feedback. The French pilot just ended last month, providing further confirmation of this issue. In the final RfC we ran on the French site, about 45% of respondents wanted AFT5 removed everywhere, while 38% wanted to keep it on an opt-in basis, and 10% on help pages only; nearly everyone agreed it should not be on by default on all 40,000 pilot pages, let alone on the entire French Wikipedia. Their concerns are consistent with what we heard from editors on the English and German pilots.

On the English Wikipedia, Article Feedback is now available on an opt-in basis, and has been enabled on about 3,677 articles by editors who want reader feedback on articles they watch. On average, editors enable this tool on about 100 new articles per week -- but some editors who oppose the tool have disabled it on thousands of articles in recent months, which creates an awkward tension in our community. (Previously, the tool was enabled on over 400,000 articles, and 672k comments were collected during this pilot; but about 63% of editors in a February 2013 RfC voted against wide deployment, leading to the removal of the tool on these articles.)

In a retrospective we conducted in early February with team members involved in the development of Article feedback (see details below), the consensus was that the time has come for us to retire this tool. Most participants agreed that Flow is better positioned to give our readers a voice -- and that we should clear the way to make it a success.

Recommendation

Based on these pilot results and our team retrospective below, we recommend that Article Feedback be removed from all pilot sites, as it is clearly not welcome by a majority of editors, despite its benefits to readers.

Flow, our new discussion tool, is expected to offer a more practical alternative for reader feedback later this year. In the long-term, it is likely to provide a more effective solution than the experimental Article Feedback tool -- which was hampered by a lack of development resources and was never well received by our editor community, due to a series of ineffective trials over the years.

Though some editors expressed interest in keeping the tool on an opt-in or limited basis, Article Feedback would need significant improvements to better serve its users, and the foundation doesn't have the resources to develop it further at this time. Besides being unpopular with many in our editor community, it is also slowing down site performance -- and may require more technical maintenance that we can adequately provide.

As a result, we recommend this tool be retired from our two pilot sites, so we can focus our limited resources on Flow and other projects instead. This recommendation aligns with what we've heard from community and team members, after an extensive consultation period. Highlights of our team retrospective are also included below, for more perspectives on this experiment.

Update: The Article Feedback Tool was removed on March 3, 2014 from both the English and French Wikipedia sites, as reported here. In coming days, we will archive the feedback data in a public hub, so it can be accessed without the tool. We will post on this thread as soon as that data archive is available. Thanks again to everyone who contributed to this experiment -- we're grateful for your support of this project. To be continued. Fabrice Florin (WMF) (talk) 02:27, 4 March 2014 (UTC)[reply]

Next steps

We propose these follow-up actions for this project.

Announcement

We plan to announce this recommendation to the English and French communities on February 12, 2014.

Tool Removal

We propose to remove AFT5 entirely from both enwiki and frwiki sites at the end of February, on a date to be determined based on community feedback and team availability.

Transition Period

We propose to give editors two weeks to transfer any feedback they find useful to their article talk pages (with the ‘Discuss on talk page’ tool).

Prototype site

For reference purposes, we also plan to keep one instance of the tool running on a Labs prototype site, subject to team availability to set this up.

Other projects

If other projects wish to enable AFT5 on their sites, their communities are welcome to do this on their own, but the foundation cannot provide active support for this project, so this would have to be a purely volunteer initiative.

Discussions

Lastly, we recommend further discussions between the community and the foundation on how to give a voice to readers on our sites, with a focus on these topics:

* How can we make it easier for readers to comment on articles they read?

Casual users find talk pages and editing tools hard to use -- and their comments are not welcomed by many editors. The first issue is mostly technical and could be solved over time with new tools like Flow. But the second issue is social and may require a change in behavior to better integrate newcomers into the current Wikipedia culture.

* How can readers participate in decisions that impact them?

Typically, readers are not well represented in editor polls, RfCs and other on-wiki discussions about new features that concern them. In the case of AFT, only a few hundred editors voted to remove a tool that could have benefited hundreds of millions of readers -- and this decision was reached without any real representation for this important user group. This doesn't seem right for a site that claims to be for everyone. How could we solve this?

We look forward to discussing these important questions in coming months, so that we can better serve the needs of all our users in the free culture movement. In the meantime, we wish to thank all of the community and team members who contributed to this experiment.

Retrospective

In early February, we invited team members who participated in this project to share recommendations for next steps, as well as lessons they learned during the planning, development and pilot testing phases. The goal of this retrospective was to finalize our recommendation, identify specific strengths and weaknesses of this experiment, as well as offer suggestions that could help future projects.

Here are highlights of lessons learned by team members who contributed:

What did we learn about this product?

Commenting systems are hard to do
Nobody likes a pile of homework
The impetus to comment for readers is different than the impetus to read for editors

What did we do well?

Community engagement helped improve the tool
Many technical improvements, which are now being used elsewhere

Web analytics and engagement study tools

What did we not do so well?

Insufficient development resources slowed us down
With products that require community buy-in, simply talking is not enough
Should have experimented with the core concepts earlier

Some scope creep

How can we do better next time?

Clearly frame the problem we aim to solve

Validate our assumptions about community behavior as early as possible

Involve as many communities as possible (not just English Wikipedia)

Resist our convictions / Do not defend

How can we give readers a voice on our sites?

Provide a good survey tool
Limited feedback per editor on demand
Give readers a voice, but keep the signal-to-noise ratio high
Give editors a voice when designing feedback tools
Give visibility to the talk page

See internal retrospective document for more details.

Pros and Cons

Here are some of the pros and cons we considered when evaluating next steps for AFT5:

Pros

effective as a reader engagement tool
gives a voice to visitors who are not ready to edit yet
increases participation, converts some readers into editors
posting feedback increases registration (with a 2.7% registration rate)
increases customer satisfaction (70% of readers like the tool)
tracks article quality (how many 'found what they were looking for')

Cons

most of the feedback is not useful to editors (~12% of posts seem useful)
moderating comments creates more work for editors
no development resources to improve the tool
some maintenance required for this complex software tool
some impact on site performance
distracts product teams from priority goals

Pilot Results

The final version of AFT5 was tested on 3 pilot sites from March to December 2013. Here are key findings for each pilot.

English Wikipedia Pilot

had been enabled on 400,000 articles through March 2013
now enabled by editors on 3,677 articles, on an opt-in basis
increasing at the rate of ~100 new articles/week
feedback was disabled on thousands of articles by editors opposing this tool
672k comments

 posted as of Feb. 12 (plus 259k posts without comments, 932k total)

65k of these comments were featured or resolved (~10%)
560k posts were unreviewed, 46k required no action, 8k were found inappropriate
about 175 editors participated in February 2013 RfC
about 63% voted against wide deployment then
AFT5 discussion page includes ongoing concerns about this tool

French Wikipedia Pilot

was enabled on 40k articles until Feb. 9, 2014
165 editors participated in first pilot poll
62% voted in favor of feedback pilot
half voted for wide release on all articles
6-month pilot started in June 2013, ended in January 2014
223k comments posted as of Feb. 12 (plus 46k posts without comments, 270k total)
25k comments were featured/resolved (~11%)
56k posts were unreviewed, 124k required no action, 18k were found inappropriate
103 editors participated in December 2013 survey
60% wanted to remove AFT5, 36% wanted to keep opt-in version
91 editors participated in January 2014 poll
45% wanted AFT5 removed everywhere, 38% wanted it kept on opt-in basis, 10% on help pages only
everyone agreed it should not be on by default on all 40k pilot pages, so it was removed on most articles
a dozen projects wanted AFT5 kept on their pages, where the tool is still enabled

German Wikipedia Pilot

was enabled on 13,000 articles for 6 months
226 editors participated in poll in May 2013
65% voted against article feedback
feedback tool was removed

Moderation Stats

Average stats across all 3 pilot sites:

Useful 12%
Resolved 4%
No action needed 46%
Inappropriate 17%

Impact on Readers

70% of survey respondents liked the feedback tool (mostly readers)
2.7% of invited readers registered after leaving feedback
3.0% of invited readers completed an edit

Wikipedia Article Feedback corpus

The full dataset can be downloaded.

"The Article Feedback experiment invited readers to participate on Wikipedia by leaving comments on articles, to help editors improve them. This data set includes over 1.5 million messages posted to the English, French and German Wikipedia during the pilot."

http://datahub.io/dataset/wikipedia-article-feedback-corpus
- (or this duplicate) http://figshare.com/articles/Wikipedia_Article_Feedback_corpus/1277784

It was previously mentioned in https://blog.wikimedia.org/2015/03/13/open-data-sets/