Requests for comment/Improving extension management

Request for comment (RFC)
Improving extension dependency management and versioning
Component	General
Creation date	25 January 2015
Author(s)	Legoktm
Document status	declined 2015-02-18 IRC See Phabricator.

Tracked in Phabricator
Task T88596

This RfC proposes a system for easier extension management through:

Allowing extensions to specify their compatibility and dependencies in a standard and machine-readable method.
Providing an API to expose said metadata
Building a CLI tool that allows system administrators to install extensions, and update them.

Handling compatibility

Pain points for developers

Handling dependencies
- Depending upon another extension is typically manual and hacky with $wgExtensionFunctions to avoid load order dependencies.
Versioning
- Versioning is mainly an internal thing, not really exposed anywhere, and not really used by sysadmins. Or is it? As a dev, I typically ignore them unless someone asks me to bump it.
Compatibility with core
- We have REL_ branches, but people usually don't backport every change they make that is compatible
- Some extension maintainers keep compatibility with older versions of core, but that isn't documented anywhere in a machine readable format for tools like ExtensionDistributor.
- wfUseMW is hacky as well and until recently required a $wgExtensionFunctions.

Pain points for sysadmins

Just because an extension has a REL1_xx branch doesn't mean it actually is compatible with said version
- Just because an extension has a REL1_xx branch doesn't mean master isn't compatible with an older stable release
Upgrading tons of extensions at once (during a major version bump) is not super friendly and requires a lot of manual steps

Considerations

Note: These considerations were mainly for a web based interface. Since we're focusing on a CLI interface, it doesn't really apply anymore.

Security: This system should function in an environment where webservers cannot write executable PHP files
Ease of use: Shouldn't require a CLI interface, nearly everything should be doable from the web
- But...for people who like CLI/git, we should still support that workflow, probably with a maintenance script.

Dependencies upon MediaWiki core

Extensions can currently specify that they depend upon a minimum version of core by using wfUseMW(), however that is extremely hacky, and only allows for specifying a lower bound of support. We automatically create REL1_XX branches whenever core is branched, except that there is no actual verification that the extension works with that core version unless the maintainer or a user does so.

Instead of that, I propose that we store the versions that the extension supports in its extension.json file:

{
    "supports": ">=1.23,<1.26"
}

Any type of version contraint supported by composer should work. These constraints would also be usable for WMF deployment branches once bugzilla:65306 is fixed. These constraints would be evaluated whenever ExtensionRegistry compiles all the JSON files to cache, so not on every request (this also depends on how slow/fast the code is).

When an extension drops support for a certain release, the maintainer can create the REL1_XX branch at the last commit that did support that release. If they don't, when we create the extension branches for the newer version, we can automatically create it for the older version.

The extension distributor would be updated to read from extension.json to see which branches should be recommended for which versions.

Dependencies upon libraries

We have standardized upon using composer as the dependency manager for library dependencies, see Manual:Composer.json best practices. Extensions would continue to list their dependencies upon external libraries through composer.json files.

We would utilize the composer-merge-plugin to manage library dependencies.

Dependencies upon extensions

Many times extension dependencies simply dependencies upon common code, that really should be refactored out into a separate library. A good example of this is AbuseFilter depending upon AntiSpoof. AbuseFilter really only needs AntiSpoof's normalization functionality, not the entire extension. That common code should be refactored out into a separate library which both AbuseFilter and AntiSpoof depend upon.

However, there are some cases where extensions do actually depend upon each other. These constraints can be added in the extension.json file:

{
    "depends": {
        "Extension:FooBar": "~1.1",
        "Extension:BazBar": "~2.0"
    }
}

Extensions that use versioning are encouraged to adopt semantic versioning, and extensions that depend upon those can use version identifiers like "~1.0". For extensions that don't use versioning, "*" will get the latest version of the extension that supports your MediaWiki core version. Any operator that composer supports will be usable.

This will be processed by the registry at compile time, and exceptions will be thrown if dependencies are missing or incompatible.

Exposing information via an API

Once this information is in extension.json, we can expose it via an API. Independently I was working on improving our current ExtensionDistributor setup, and think that including this metadata with that would make sense. See Extension:ExtensionDistributor/tardist for more details.

Implementation

A maintenance script would list what extensions have updates available and allow you to update those extensions. The update process would involve downloading an updated tarball from extension distributor, unzipping it, and installing it (moving files to the right place). The composer.json would be read if it exists and dependencies will be calculated and fetched by accessing the composer functions directly.

We would utilize the code already written for composer by adding "composer/composer" as a dependency and interacting with the DependencyResolver part. (Legoktm's quick POC)

Why not use composer directly?

Regarding Requests for comment/Extension management with Composer:

As mentioned earlier, composer is what we use to manage dependencies upon libraries. It's important to note that extensions are not libraries. Libraries typically just require an autoloader, whereas extensions need an autoloader, but also need to register hooks, special pages, API modules, and much more.

composer combines the installation process (downloading tarball and unzipping it to the right place) and the enable process (require_once/wfLoadExtension) into one step. This is especially problematic for wiki-farm setups where extensions may be enabled on one wiki but not on others (see A way to disable extensions)
- We could just have composer download and install the extension, and *not* autoload the entry point.
composer is still an alpha product, ~~and not packaged by any major distros~~. This means users need to download a phar file and trust it.
- If we add composer/composer as a dependency to mediawiki/core (that would be shipped with the tarball), we wouldn't have to require the binary and could directly interact with the composer API.
- composer is now packaged for Fedora 21+ (stable) and Debian sid
What Dantman said: [1], [2].

All that said, composer's dependency resolver is pretty powerful, and makes sense for us to re-use (and not reinvent), but we can do that without having to use the rest of composer and package our extensions as composer dependencies.