Moderator Tools/Automoderator/Unified Activity Dashboard

Unified Automoderator Activity Dashboard

The Unified Automoderator Activity Dashboard is intended to serve both the Moderator Tools team, the community, and other interested folks, to monitor and evaluate key metrics related to Automoderator's activity. The dashboard tracks activity across all the wikis where Automoderator is deployed on.

You can view the dashboard by logging into the public Superset instance via your Wikimedia global account at
https://superset.wmcloud.org/superset/dashboard/unified-automoderator-activity-dashboard/

The metrics on the dashboard are categorized into three tabs:

Monitoring metrics: These are updated daily, to enable operational monitoring of Automoderator's activity.
Key metrics: These are updated monthly and are more reliable for evaluation and reporting purposes.
Configuration: This is updated weekly and provides an overview of current deployment status and configuration by wiki.

Monitoring vs. key metrics

Monitoring metrics are a subset of key metrics that are updated daily, while key metrics are updated monthly. The data end of the key metrics is the last day of the previous month. The monitoring metrics pipelines rely on MariaDB-replicas and run daily. To accurately calculate number of revisions reverted by a single revert action, especially in cases where multiple edits are reverted using rollback, we need to match based on the content-hash (rev_sha1) in the revision table. This is complex to implement and computationally expensive to run on a daily basis on multiple wikis, and essentially re-inventing the wheel of mediawiki_history. Many fields (especially those related to reverts) required for these metrics are already being calculated by the MediaWiki history pipeline using the notion of revision identity-revert. MediaWiki history snapshots are released on a monthly basis.

Waiting an entire month to track Automoderator's activity, take any action if needed, and then wait another month for the results won't fit our needs. So, we decided to have a subset of key metrics that we track daily. The goal is to keep the daily metrics pipelines lightweight. This is prone to a degree of error (that is acceptable for our use case)—primarily, we may be under counting if multiple edits are reverted by a single revert (rollback). So the goal of monitoring metrics is for operational monitoring only (for example, ensuring there is no major issue), while key metrics, updated monthly based on MediaWiki history, are more accurate for reporting and evaluation.

Data pipelines

Major assets

Analytics cluster

Configuration

The source truth for Automoderator deployment status as well as the localised username, which is needed fetch the edits made is InitialiseSettings.php, a PHP array. wmgUseAutoModerator for wikis where Automoderator is currently deployed on, and wgAutoModeratorUsername for localized version of the username.

Pipeline (frequency - weekly)

Airflow DAG: automoderator_config_weekly_dag.py

uses extract_automoderator_config.py

to update an internal table at wmf_product.automoderator_config (table schema; DataHub documentation)
and publish the latest snapshot to https://analytics.wikimedia.org/published/datasets/automoderator/automoderator_config.tsv

Small wiki classification

Automoderator may be enabled on small wikis by trusted global admins (T372280). The dashboard will need a filter to view activity of Automoderator on all such wikis. Although the dashboard will have a filter to select specific wikis, selecting all of them individually will be tedious. The small wikis filter is a quick access filter to select all small wikis.

Pipeline (frequency - ad hoc)

As the data is unlikely to change regularly, this will be updated on ad-hoc basis.

The initial classification is based on Special:WikiSets/7.
The classification can be updated using the instructions at automod-smallwiki-classifier.
The classification is available:

internally at wmf_product.automoderator_small_wikis_classification (table schema)
publicly at small_wikis_filter_automoderator.json

Daily monitoring snapshots

The daily monitoring snapshots are intended to help with operational monitoring of Automoderator's activity. The daily snapshots are based on data from MariaDB-replicas, use the following tables: revision, page, change_tag, change_tag_def, actor and user.

Pipeline (frequency - daily)

Airflow DAG - automoderator_daily_snapshot_dag.py, uses:

to update an internal table at wmf_product.automoderator_monitoring_snapshot_daily (table schema; DataHub documentation)
to publish latest snapshots to https://analytics.wikimedia.org/published/datasets/automoderator/daily_snaphots/
lastly purge_automoderator_daily_snapshots.py, to purge snapshots older than 30 days from the internal table.

Monthly activity snapshots

The monthly activity snapshots are intended to provide a comprehensive overview of Automoderator's activity across wikis. These snapshots are based the data from wmf.mediawiki_history.

Pipeline (frequency - monthly)

Airflow DAG - automoderator_activity_snapshot_monthly_dag.py, uses:
generate_automoderator_activity_snapshot_monthly.hql

to update an internal table at wmf_product.automoderator_activity_snapshot_monthly (table schema; DataHub documentation)

publish_automoderator_activity_snapshot_monthly.hql

to create a TSV and publish the latest snapshot to https://analytics.wikimedia.org/published/datasets/automoderator/monthly_snapshot

lastly purge_automoderator_activity_snapshot_monthly.hql, to purge snapshots older than three months from the internal table.

Potential vandalism reverted

The one of the main goals of the Automoderator project is to see if it will reduce the workload with patrolling new edits. The aggregate metrics are calculated using wmf.mediawiki_history.

Pipeline (frequency - monthly)

Airflow DAG - automoderator_potential_vandalism_reverted_monthly_dag.py, uses:

generate_automoderator_potential_vandalism_reverted.hql

to update an internal table wmf_product.automoderator_potential_vandalism_reverted (table schema; DataHub documentation)

publish_automoderator_potential_vandalism_reverted.hql

to create a TSV and publish the latest snapshot to https://analytics.wikimedia.org/published/datasets/automoderator/

lastly purge_automoderator_potential_vandalism_reverted.hql, to purge snapshots older than three months from the internal table.

Toolforge

From the datasets published, jobs based on Toolforge jobs framework update the corresponding tables in Automoderator metrics database (s56213__unified_automod_metrics_p) in ToolsDB, which can be accessed the by the public Superset instance.

Configuration - weekly

table: automoderator_config
updated weekly by update_automoderator_config_weekly.sh

using update_automoderator_config.py

Monitoring snapshots - daily

table: automoderator_monitoring_snapshot_daily
updated weekly by update_automoderator_monitoring_snapshot_daily.sh

using update_automoderator_monitoring_snapshot.py

Comprehensive activity snapshots - monthly

table: automoderator_activity_snapshot_monthly
updated weekly by update_automoderator_activity_snapshot_monthly.sh

using update_automoderator_activity_snapshot.py

Potential vandalism reverted - monthly

table: automoderator_potential_vandalism_reverted_monthly
updated weekly by update_automoderator_potential_vandalism_reverted_monthly.sh

using update_automoderator_potential_vandalism_reverted.py