Talk:Requests for comment/Proper command-line runner for maintenance tasks

For BlueSpice we have build a command line application for a variety of maintenance tasks (mainly connected to releases/builds). We are using the "symfony/console" library and are quite happy with it. Using BOX for creating a PHAR file that contains all the classes/dependencies.

Using symfony/console could be a follow-up project. This RfC makes it easier to get there, and eventually MaintenanceTask could just be a wrapper around symfony/console.

Hard to do well, as most scale based on edit throughput, but an interactive tool of "hey, this one hasn't been run in a week, it's recommended to run daily, would you like this script to try to install a cron job for you?" might be helpful for smaller third party MediaWiki sysadmins?

Hard to say; I feel like most scripts only need to be run if something's unusual (to clean up errors or old bugs, or to do batch work that was skipped by a manual database import, or changing backends etc) or are task-oriented like resetting passwords and doing imports/exports.

This is probably a little bit off topic, but yet I'd like to share:

In BlueSpice we have got some maintenance tasks that need to run periodically (e.g. "check for broken external links once a day", "create notifications for expired pages every week", "create a PDF report every month"). For this purpose we "hijack" the "runJobs.php" execution which we configure to happen every few minutes (using a cronjob). An extension can then implement a "RunJobsTriggerHandler". This handler can specify an "Interval" ("OnceADay", "OnceAWeek", ...). This Interval" is a PHP object, which therefore allows for static values (like "date of next sunday") or potentially dynamic values (that may depend on some state of the system. E.g "Send certain mails once a day, but only 100 at a time; Oh, there are 120 to sent today, so I will now send 60 and in 30 minutes send the rest. After that my next run will be tomorrow").

This way the only thing we need to setup is the cronjob that triggers "runJobs.php" at a certain frequecy. The extensions then will register their handlers and define their required intervalls.

See also

• https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation/blob/fd6d368a667c9dc060d88f36ae70a3734cac4723/src/IRunJobsTriggerHandler.php
• https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation/blob/fd6d368a667c9dc060d88f36ae70a3734cac4723/src/RunJobsTriggerHandler/Interval.php
• https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation/tree/fd6d368a667c9dc060d88f36ae70a3734cac4723/src/RunJobsTriggerHandler

Why not a general architecture to run command line tools, of which the maintenance tool is just a more specific set/scope with maybe it's own registry ?

Basically https://laravel.com/docs/5.6/artisan

which also allows you to document the command and its arguments and even to call command line programs programmatically, to run them using a scheduler or to queue the command itself as job.

Ive been enjoying using that a lot.

I see you posted the same thing at T99268#4576693. It might be better to keep discussion all in one place.

Should the 'mwscript' helper script in the production and vagrant environments be replaced by a version that calls mwcmd.php specifically? Would prefer to do "mwcmd commonswiki refreshImageMetadata" rather than "mwscript mwcmd.php commonswiki refreshImageMetadata" ...

My understanding is that mwscript is necessary because each script is its own entry point, but now we could have a single multiversion entry point that just wraps the new runner.

My understanding is that mwscript exists because WMF's setup needs to know which version's maintenance script to run. For example, mwscript eval.php --wiki=foowiki might run /srv/mediawiki-staging/php-1.32.0-wmf.19/maintenance/eval.php while mwscript eval.php --wiki=barwiki needs to run /srv/mediawiki-staging/php-1.32.0-wmf.20/maintenance/eval.php instead.

That won't go away with the changes proposed here, it just means that "mwcmd" or whatever has to decide between /srv/mediawiki-staging/php-1.32.0-wmf.19/maintenance.php and /srv/mediawiki-staging/php-1.32.0-wmf.20/maintenance.php instead.

Many command line tools have structured commands into two levels (npm access [grant|revoke] to give one example). I wonder if we should follow that to avoid a long flat list of commands.

That might of course lead to difficult questions how to determine the top level commands, but it could absorb the admin/developer split as well.

I was recently reading that until very recently, golang has avoided having any subcommands (c.f. https://go-review.googlesource.com/c/go/+/126655). I haven't yet decided on whether I agree with that idea or not.

Ideally I imagined that normal users would not even care that a sysadmin/developer split happened, because they never need to use the developer scripts.