Growth/Communities/Quality of edits

Tracked in Phabricator
Task T312154

Quality of edits is actually not a well defined concept within the wikis.

“All the questions operate with the word “quality”. How do we define “quality”?” – cs:user:KPX8

As Growth received feedback from the community that the revert rate is not an accurate measure of the quality of a change, and therefore is not an accurate way for the Growth team to measure the effectiveness of our tools. With that in mind, we have some questions for communities about how we can use data other than revert rate to evaluate Growth tools.

How do you assess if a given user makes good-quality edits?
Which tools do you use to assess the quality of user edits? (This excludes gut feelings, such as "I've seen this before" or "I don't feel it.")
Besides the revert rate of a given user, which data do you use to assess the quality of their edits?
Which tools do you wish you had to assess the quality of edits? (We are unlikely to focus on this immediately, we are at the brainstorming stage and looking for inspiration).

This document is a summary of the findings, after interviewing mentors and patrollers from ar, bn, cs, es, fr Wikipedias. See T312154 for the details.

Quality according to experienced users

The definition of quality varies from one user to the other:

“Sometimes the modifications are good, but they are reverted or deleted because they simply were not well checked.” – ar:user:Bachounda

According to interviewed patrollers from the Pilot wikis, what identifies a qualitative edit is, by order of mention:

The quality of added/removed contents
Clarity of diff comments; adequation between the edit summary the the edit itself; presence of jargon
Revert rate for the user
Presence and quality of sources in the edit.
Quality of the language used
Clean user talk page and participation or other’s talk pages.
Clean contributions journal (no reverted edits, no blocks..)
Filtered or tagged edits.
Usage of templates in the edit

This list combines what is directly noticeable from the edit, but also relies on meta information about the user (messages left on article talk page, blank user talk page with no warnings, logged actions).

It is not always easy to assess the quality of each edit, even with experience. “Except in my own field, I’m unable to spot a good-quality edit myself, let alone teach a machine to do it.” – cs:Jan Spousta

IP edits perceived as lower quality is a recurring topic.

User behavior regarding low quality edits

One will accept an edit meaning well but not perfect, while another will revert it. While newcomers have to make mistakes, some experienced users don’t accept them.

“Do we care about good faith/bad faith edits? Newcomers making mistakes, but wanting to learn?” – cs:user:KPX8

Manual check of edits are an issue, as easier to check edits will be patrolled first, and the remainers left aside. This continues for the next patroller until only boring, complicated or specialized edits are left to patrol. This trend was identified for article creations patrolling by the ACTRAIL project. Therefore, an unreverted edit doesn’t mean it is a good edit. Topics edited can be a quality trigger warning for experienced users.

“The one who spends his time on Cenozoic fossils worries me less than the one who is passionate about Paris’ soccer goats” – fr:user:JohnNewton8

Tools used to check on quality of edits

“It seems difficult to verify with any tool whether a user is making good quality edits or not.” – ba:user:Aftabuzzaman

Special:Contributions can be used to see how many of the edits are the last at articles, or if there is a pattern on the edits (similar additions/removals, lots of small edits…)

X-Tools Edit Counter is used for a similar purpose, as it gives a summary of the edit types.

For the overall quality of a wiki, https://wikirank.net/ has been cited.

ORES seems to be used in Recent Changes at the wikis where it is available, but discussions around edit predictions are rare.

A blank userpage (appearing as a red link) is considered as a sign of a newcomer. Red-linked accounts are checked more carefully by some users.

The edit summary is checked to know what was added, or if the user uses wiki-slang (a sign of appartenance to the community, hence of trust, even if being experienced doesn’t mean making quality edits).

Some gadgets exist as well:

DeluxeHistory highlights the line of the edit in RCs, watchlist, etc. depending on the status of the user. IPs are by default in red.

Tools that give information on the quality of edits

ORES is the main tool to work on edit predictions.

This model is available for a few languages, and deployed at even fewer languages (mostly big wikis).
ORES requires training by humans, which means that the predictions will reflect the editing bias of the trainers.
Some wikis finished their training, but don’t have ORES available yet on the concerned wiki.

Edits marked as patrolled are useful to know if someone checked on a given edit.

Edits not marked as patrolled are highlighted by a red exclamation mark.
However, if Patrolled edits are available at all wikis, they aren’t activated by default, only by a few.
Marking an edit as patrolled is up to the user who patrols it. Someone else can disagree on it and revert.

Regarding the quality of the article that is edited, this quality is often considered as information as well. Some tools and cultural practices stat on the quality of a given content.

Wikimedia Enterprise considers to add a credibility signal to the contents they broadcast.
XTools has a gadget script that give some information about the quality of the content.
Features/Good contents are a praise for the communities.

These qualitative contents are more likely to be more patrolled, or more accurately checked.

A user who’ve been reverted by a bot is more likely to make low quality edits, as bots react to dictionaries, or previous users’ actions.