Talk:Article feedback/Version 5/Report

Thanks for publishing this. It would need to be clearer in the text that "the time has come for us to retire this tool" means really retire. --Nemo 22:31, 12 February 2014 (UTC) P.s.: In [1] I especially liked the sections "Should have experimented with the core concepts earlier" by Pau, the comment by Matthias on how «each AFT iteration shifted in goals and implementation quite drastically» and the technical/development learnings for devs highlighted by Matthias and Ori.

My thanks as well for publishing this report. It's good work. --MZMcBride (talk) 04:36, 13 February 2014 (UTC)

One editor's practical experiences

I found the concept of an easy click to edit feedback tool very promising, but the implementation is clunky, inefficient, and ineffective.

• The tool asks if the reader has found what he or she was looking for, (a useful question) then totally fails to ask what that was, wasting the opportunity to find out what the users are looking for, which would be a useful answer.
• The questions asked while in the feedback tool are poorly designed to elicit useful feedback. the suggestion about more images is a case in point. Many suggestions for more images were totally inappropriate.
• There is no convenient way of notification of feedback on any given article. no watchlist is possible. this requires an inordinate amount of effort to find out if there is recent, unactioned, useful feedback on articles that one wishes to monitor. This is a waste of time.
• Most useful or marginal feedback can not be reliably assessed by a random editor. I have often seen feedback flagged as useful which was not, or no action when it was potentially useful. Feedback is most useful to the editors who are knowledgeable on the subject matter and able to discriminate.
• Assuming that users did not find what they were looking for because they left no feedback is misleading.
• A large portion of the feedback was insufficiently detailed to be useful. The user should be requested to be reasonably specific. It is also not possible to contact the user to request clarification. this also drastically reduces the effectiveness of the tool. An anonymising e-mail reply would help. This could be optional for people who are unwilling to leave an e-mail address, but it might not be a bad thing to eliminate the anonymous feedback as that would also eliminate almost all of the garbage feedback. Providing this as a user option for the editor who opts in for feedback would be nice.
• Some level of user control of questions for a specific article would be useful.
• What I am looking for in feedback is:
  • What did the user want to find,
  • Was the available information comprehensible
  • Are there any major aspects of the subject that have been omitted
  • Does the user have any concrete suggestions for improving the article
  • What is the background of the user in the subject of the article (only if one of the other items has been addressed)
• On the whole, I find article feedback more useful than not - marginally. It has provided feedback which has enabled me to significantly improve a few article, ath the cost of a lot of wasted time

I will add to this if I think of any additional comments. Pbsouthwood (talk) 02:38, 13 February 2014 (UTC)

Comments from Cantons-de-l'Est

It is a good idea, but the implementation left too many open doors to toxic ou useless comments. I think that Feedback should be implemented with choices. We will loose good comments, but we will dramatically reduce the time to filter out the noise.

• A comment applies to a section, not the article. In this way, the editors will have easier time to find what is to correct.
• Since the reader can only pick a choice, he cannot write toxic or hainous comment. Equally, the software can group the comments by their type. In this way, the editors will choose what is most urgent to correct (for instance, copyvios before typos).
• Possible comments :
  • The section has
    • a typo or grammatical error
    • a syntax error
  • The section
    • is OK
    • is factual and complete
    • has a copyvio
    • is defamatory
    • is hainous
    • lack illustrations (photos, diagrams, maps, etc.)
    • lack sources

We can add other bullets. However, we must not fall into vagueness (like, the section is wrong ; "wrong" is too vague : date, typo, verb ?). And, in no way, have an "others" bullet with a way to write something. This will open the door to hainous/toxic/useless/childish comments, and someone will have to filter out these, clearly a waste of time in the light of previous experiment.

We will miss many useful comments, but I prefer 60 % useful comments to 80 % hainous/toxic/useless/childish comments.

Cantons-de-l'Est (talk) 14:16, 13 February 2014 (UTC)

There should still be possible to keep some of those useful commentary by allowing "free text" comments in some of the options. Feedback classified as "the section contains a factually incorrect assertion" is useless without a way to say what assertion is incorrect.

There should be a way to provide a short comment or a URL reference for such feedback categories; and the context where that text could be entered (after selecting the type of feedback to be provided) should discourage most of the garbage that was input in the previous catch-all "write your oppinion here" text area that was the Feedback tool. After all, readers always had the possibility to add garbage comments at the Talk page and yet we are not flooded by them; so it clearly amounts to the particular way in which those comments are requested. Diego Moya (talk) 13:48, 25 February 2014 (UTC)

Thanks for your insights

Dear Nemo, MZMcBride, Pbsouthwood, Cantons-de-l'Est and Diego Moya: Thanks for your helpful insights about this project.

I appreciate your observations, many of which match comments from our own team retrospective. The Article Feedback v5 tool would have benefited from the features you describe above, which all make good sense to me: I wish we had the resources to build them for you. In my view, the biggest issue with this project is that we took on a very hard problem with insufficient resources to effectively solve it.

Our small team engaged community members extensively throughout this experiment, and we were grateful for all the good recommendations we received; but we simply did not have the capacity to build all these features with a single contract engineer. This taught us an important lesson, and we are now staffing our teams more effectively for projects of this size, such as Flow.

On the whole, I think we all gained from this project, despite its setbacks. A lot of the code and research tools we developed for Article Feedback are now being used by other projects, so this experiment is helping improve Wikipedia in more ways than one. In times like these, I am reminded of Thomas Edison's words about his own experiments: 'I have not failed. I've just found 10,000 ways that won't work.' We too have learned a lot from this exploration -- and I am very grateful for your willingness to experiment with us. I look forward to more collaborations with you in the future.

As recommended in our report, we now plan to remove the Article Feedback Tool entirely from both the English and French Wikipedia sites on Monday, March 3 at 19:00 UTC (see this Gerrit ticket and this Bugzilla report). So any editors who wish to transfer useful feedback to their article talk pages should do it this weekend, using the built-in ‘Discuss on talk page’ tool. We will also archive the feedback data in a public hub, so it may be accessed even after the tool has been disabled.

Onward! Fabrice Florin (WMF) (talk) 04:28, 1 March 2014 (UTC)