Moderator Tools/Automoderator/Multilingual testing

Automoderator

Measurement plan

Testing

Deployment

Diagram demonstrating the Automoderator software decision process.

Automoderator is currently deployed on a number of Wikimedia projects using the Language-agnostic Revert Risk model, following a round of user Testing which found that it was reliable enough to be used. One other finding of that research, confirmed by our ongoing analysis of Automoderator's behaviour, is that it doesn't currently handle a significant percentage of the patrolling workload. Because of this, we are investigating switching to the Multilingual Revert Risk model on the Wikipedias which are supported by it. Before we do that, we need more community input to test this model so that we can understand its strengths and weaknesses, and set appropriate revert thresholds.

How to test the Multilingual Revert Risk model

Screenshot of the spreadsheet, with example responses filled in.

• If you have a Google account:
  1. Use the Google Sheet link below and make a copy of it
     • You can do this by clicking File > Make a Copy ... after opening the link.
  2. After your copy has loaded, click Share in the top corner, then give any access to swaltonwikimedia.org (leaving 'Notify' checked), so that we can aggregate your responses to collect data on Automoderator's accuracy.
     • Alternatively, you can change 'General access' to 'Anyone with the link' and share a link with us directly or on-wiki.
• Alternatively, use the .ods file link to download the file to your computer.
  • After adding your decisions, please send the sheet back to us at swaltonwikimedia.org, so that we can aggregate your responses to collect data on Automoderator's accuracy.

After accessing the spreadsheet...

1. Follow the instructions in the sheet to select a random dataset, review 30 edits, and then uncover what decisions Automoderator would make for each edit.
   • Feel free to explore the full data in the 'Edit data & scores' tab.
   • If you want to review another dataset please make a new copy of the sheet to avoid conflicting data.
2. Join the discussion on the talk page.

Alternatively, you can simply dive in to the individual project tabs and start investigating the data directly.

---

Google Sheet TODO

We welcome translations of this sheet - if you would like to submit a translation please make a copy, translate the strings on the 'String translations' tab, and send it back to us at swaltonwikimedia.org.

If you want us to add data from another Wikipedia please let us know and we would be happy to do so.

About Automoderator

Automoderator’s model is trained exclusively on Wikipedia’s main namespace pages, limiting its dataset to edits made to Wikipedia articles. Additionally, this model only supports 47 languages.

Further details on Automoderator's internal configuration and caution thresholds can be found on the original testing page .

The number of reverts expected at different caution levels can be seen below:

TODO

This data can be viewed for other Wikimedia projects here (TODO).