MediaWiki API integration tests

Translate this page

MediaWiki performs end-to-end integration testing for the Action API and REST API using the API-Testing library. The library is implemented in JavaScript for node.js, using the SuperTest HTTP testing library, the Chai assertion library, and the Mocha testing framework. Integration tests are run as part of continuous integration for MediaWiki Core and can be added to MediaWiki extensions and Wikimedia services.

Setup

Installation

After installing node.js and npm, use npm to install Mocha and the API-Testing package.

$ npm install --save-dev api-testing mocha

For safety and convenience, it is recommended to run npm inside a fresh-node container. The instructions for running selenium tests in fresh-node apply to running API integration tests as well, see Selenium/Getting_Started/Run_tests_using_Fresh.

Configuration

To run the tests, you need access to a MediaWiki installation. We recommend using a Docker container to set up a wiki instance to test against; MediaWiki-Docker-Dev uses docker-compose to provide everything you need to run MediaWiki.

Caution:

The content of the wiki you are running the tests against will be polluted with test content! Do not run tests against a wiki with valuable content.

To allow all tests to run properly, your LocalSettings.php has to include the DevelopmentSettings.php file:

require_once "$IP/includes/DevelopmentSettings.php";

This enables experimental API endpoints and sets up caching and error reporting in a way that is suitable for testing.

To configure the API-Testing environment, copy the example into the root folder of your application, rename the file .api-testing.config.json, and add the information for your test wiki. For a MediaWiki extension, include the config file in the root of MediaWiki Core. The example config is designed to work with a default setup of MediaWiki-Docker-Dev, just add the test wiki's secret key.

For automated testing, set the API_TESTING_CONFIG_FILE environment variable to point to the correct configuration file. The configuration file is evaluated in the following order:

API_TESTING_CONFIG_FILE if set
.api-testing.config.json if it exists

Configuration example^[1]

{
	"base_uri": "http://default.web.mw.localhost:8080/w/",
	"main_page": "Main_Page",
	"root_user": {
		"name": "Admin",
		"password": "dockerpass"
	},
	"secret_key": "abcdef",
	"extra_parameters": {
		"xdebug_session": "PHPSTORM"
	}
}

Configuration schema

base_uri: Full base URI of the MediaWiki installation to target. Must end with the $wgScriptPath and a slash (/).; The domain name and port of the base_uri must match the $wgServer setting. However, if you are running the mocha tests inside fresh-node or another the docker container, you may need to replace the host name with host.docker.internal or the IP address of the host system.
main_page: Name of the wiki's main page
root_user: Login credentials for a user that has bureaucrat privileges (most importantly, the right to add users to groups to grant them privileged access)
secret_key: Replace with the value of $wgSecretKey, which can be found in the wiki's LocalSettings.php file.; If you are using Mediawiki CLI (mwcli), $wgSecretKey is set in ~/.mwcli/mwdd/default/mediawiki/MwddSettings.php.; If you are using MediaWiki-Docker-Dev (DEPRECATED - switch to mwcli), LocalSettings.php can be found under config/mediawiki/.

Caution: Be careful about running tests against a wiki that is publicly accessible. Tests should be written to use randomized passwords for all accounts they create, but there is no guarantee that no tests creates a privileged user with a known or easy to guess password. Also, if your test wiki is publicly accessible, be careful not to publish the root_user credentials, and to not use the default credentials. Even if the wiki itself doesn't have valuable content, having your test wiki compromised may open you up to attacks if it shares a domain or host with a real wiki.

Service setup

For non-MediaWiki applications, set the environment variable REST_BASE_URL to point to your service. For example:

$ REST_BASE_URL=http://localhost:3000/

Continuous integration

The Wikimedia continuous integration (CI) infrastructure supports automatic API testing for MediaWiki Core and extensions. When configured for an extension, the CI job sets up a test instance of MediaWiki, runs the MediaWiki Core tests, and then runs the extension tests. Accordingly, this integration should not be used for extensions that interfere with the behavior of Core and would cause Core API tests to fail.

To enable API tests in CI

1. Add the api-testing script to your extension's package.json file.

MyExtension/package.json example:

{
	"private": true,
	"name": "eventbus",
	"version": "0.0.0",
	"scripts": {
		"test": "grunt test",
		"api-testing": "mocha tests/api-testing"
	},
	"devDependencies": {
		"api-testing": "^1.0.4",
		"eslint-config-wikimedia": "0.16.2",
		"grunt": "1.1.0",
		"grunt-banana-checker": "0.9.0",
		"grunt-eslint": "22.0.0",
		"mocha": "^7.1.1",
		"uuid": "^3.4.0"
	}
}

2. Recommended - If you are already running ESLint linter tests for your extension make sure to include a .eslintrc.json file to your api-testing directory.

MyExtension/tests/api-testing/.eslintrc.json example:

{
	"extends": [
		"wikimedia/server",
		"wikimedia/mocha"
	],
	"rules": {
		"camelcase": "off"
	}
}

3. Enable the mediawiki-quibble-apitests-vendor-docker container in integration/config/zuul/layout.yaml.

zuul/layout.yaml example:

  - name: mediawiki/extensions/MyExtension
    template:
      - name: extension-quibble
      - name: extension-phan
      - name: extension-seccheck
      - name: extension-coverage
    test:
      - mediawiki-quibble-apitests-vendor-docker
    gate-and-submit:
      - mediawiki-quibble-apitests-vendor-docker

After you have submitted the patch, make sure it passes experimental by committing an empty change to your extension repo and run the experimental checks.

For more information about Zuul and configuring jobs, see the tutorial.

Running tests

To run the API integration tests for MediaWiki core, use the following command (preferrably inside a fresh-node container, see above).

$ npm run api-testing

If you want to run just one test file, or the tests for a specific extension, use

$ npm run mocha -- <test-file-or-dir>

To limit the test run based on the name of the test case, use

$ npm run mocha -- <test-file-or-dir> --grep <test-name>

For more information on running Mocha tests and controlling the output, see the Mocha docs.

Database Snapshots

Before running tests, it's advisable to ensure a known state of the wiki the tests run against. While tests should be written to be robust against pre-existing content, e.g. by randomizing all resource names, a known base state is useful. Also, test runs tend to pollute the wiki a lot, so a reset is bound to save space, even if not done for every test run.

The easiest way to achieve a known state of the wiki is to take a snapshot of a known state, preferably right after installation when the wiki contains just one page and one user, and then load that dump into the database before running tests. For convenience, two pairs of scripts are supplied to achieve this: one pair for use with a local MediaWiki installation and another pair for a MediaWiki-Docker-Dev environment.

Local snapshots

If you have MediaWiki installed locally, you can use:

$ node_modules/api-testing/bin/take-snapshot <name.tar> [db] [host]

This saves a snapshot of a wiki in the given tar file. The [db] parameter is the database name. If not given, "wiki" is used, which is the default name proposed by the MediaWiki installer. The [host] parameter allows the database host to be specified, in case it's not localhost.

$ node_modules/api-testing/bin/load-snapshot <name.tar> [db] [host]

This restores the snapshot in the given tar file. The tar file contains the name of the wiki database the snapshot was taken from. If the [db] parameter is not given, the dump will be loaded into that same database. The name of the database is also shown in the confirmation prompt.

Before you can use these scripts, you need to configure the location of your MediaWiki installation in bin/local.env:

MW_DIR="../../mediawiki"

Set this to something like /var/www/html/mediawiki/ or wherever you have installed MediaWiki.

MediaWiki-Docker-Dev snapshots

If you have your wiki instances managed by MediaWiki-Docker-Dev, you can use:

$ node_modules/api-testing/bin/mwdd-take-snapshot <name.tar> [db]

This saves a snapshot of a wiki in the given tar file. The [db] parameter is the database name, which is the name you gave your wiki when running the addsite script. If not given, "default" is used, which is the name of the wiki pre-installed by MediaWiki-Docker-Dev.

$ node_modules/api-testing/bin/mwdd-load-snapshot <name.tar> [db]

This restores the snapshot in the given tar file. The tar file contains the name of the wiki database the snapshot was taken from. If the [db] parameter is not given, the dump will be loaded into that same database. The name of the database is also shown in the confirmation prompt.

Before you can use these scripts, you need to configure the location of your MediaWiki-Docker-Dev installation in bin/local.env:

MWDD_DIR="../../mediawiki-docker-dev"

Set this to something like $HOME/opt/mediawiki-docker-dev/ or wherever you have installed MediaWiki-Docker-Dev.

Writing tests

In MediaWiki Core, integration test files are stored in the tests/api-testing directory. Action API tests are stored in the tests/api-testing/action directory; REST API tests are stored in the tests/api-testing/REST directory. For MediaWiki extensions, add tests to the tests/api-testing directory of the extension.

Each test file corresponds to a test suite covering an area of functionality. A test suite is defined by a top-level describe() function that takes two parameters:

a string describing the feature being tested
a function containing the test code

Within the describe() function, you can use the before() hook to set up preconditions and the it() function to create individual test cases. You can break up a long, complex test suite by nesting additional describe() functions under the top-level function.

Within a test suite, each test case is executed as an async function. These asynchronous functions contain await expressions to execute test steps in sequence and assert expressions to evaluate responses. Assertions can use any methods supported by the Chai assert interface, such as assert.equal, assert.match (using a regular expression), and assert.include.

An await expression pauses the execution of an async function until the expression can return a resolved Promise. Because of this, await expressions are only valid inside async functions. For more information about Promises, see MDN's guide to using Promises.

Here's a template that shows the main sections of a test suite:

'use strict';

// Load required modules
const { assert, action, utils } = require( 'api-testing' );

// Define a test suite
describe( 'The feature being tested', function () {
	// Define global variables
	...

	// Set up preconditions
	before( async () => {
		await ...
	} );

	// Define a test case
	it( 'should perform the action being tested', async () => {
		// Execute test steps
		await ...
		// Validate output using assertions
		assert ...
	} );
} );

Generating random strings

When writing tests, use random values whenever possible. This allows tests to function when run against a wiki that already contains content from previous test runs. To generate a random string for use in a test, use the uniq() function.

// Generates a unique, 20-character string of random alphanumeric characters
// Defaults to 10 characters
utils.uniq( 20 );

Handling page titles

The API testing tool includes functions to help you manage wiki page titles.

// Returns a random, 10-character, alphanumeric page title
utils.title();

// Returns a random page title with the prefix 'Test:'
utils.title( 'Test:' );

// Returns the provided title with spaces replaced with underscores
// This example returns 'My_Wiki_Page'
utils.dbkey( 'My Wiki Page' );

// Returns true if the provided titles are equal
assert.sameTitle( 'Test Page 1', 'Test Page 1' );

Creating accounts and logging in

Fixtures

To manage accounts and sessions, the API testing tool provides convenient fixtures that you can reuse across tests.

fixture name	account type
`alice`	user
`bob`	user
`robby`	bot
`mindy`	admin

To open a wiki session and log in using a fixture, instantiate the account asynchronously using a fixtures function.

// Defines a global variable to use in the test suite
let alice;

// Opens a session and logs in as alice
before( async () => {
	alice = await action.alice();
} );

Custom accounts

If you're planning to make a permanent change to an account (for example: blocking an account) or if you need an account with a custom prefix, you can create a custom account instead of using a fixture. To open a wiki session and log in using a custom account, define a session using the getAnon() function, and log in using the account method. Applying a prefix to the username is optional.

// Defines global variables for two custom account sessions
const fiona = action.getAnon();
const franky = action.getAnon();

// Logs in and applies the Fiona_ and Franky_ prefixes to the usernames
before( async () => {
	await Promise.all( [
		fiona.account( 'Fiona_' ),
		franky.account( 'Franky_' )
	] );
} );

Account properties

Once logged in, you can access information about the account using the username, userid, and password properties.

parameter name	example	description
`username`	`alice.username` `myUser.username`	Randomly generated username tied to the account. Usernames for accounts created using a fixture are prefixed with the name of the fixture (for example: alice_dRUET7xhKQ). Usernames for custom accounts can have an optional prefix; for example, `myUser.account('User1_')` results in a username in the format User1_nD9EUfYXgR.
`userid`	`alice.userid` `myUser.userid`	User ID tied to the account
`password`	`alice.password` `myUser.password`	Randomly generated password tied to the account

Anonymous users

To create an anonymous user, define the account using the getAnon() function, but omit the account function. This opens a new session without logging in, resulting in an anonymous user.

// Creates a session for an anonymous user
const anonymousUser = action.getAnon();

Working with wiki pages

The API testing tool provides helpful methods for interacting with wiki pages, including editing a page, exploring page history, and retrieving page HTML. To create and edit a page, use the title() function to generate a random title and the edit() method to edit the page. To validate the edit, you can use the getHtml() method to get the HTML of the page and the assert.include() method to check for the edited text.

describe( 'Page editing', function () {
	let alice;
	// Generates a random page title
	const title = utils.title();

	before( async () => {
		alice = await action.alice();
	} );

	it( 'should edit a page', async () => {
		// Has alice edit the page with "Hello, world!"
		const editPage = await alice.edit( title, { text: 'Hello, world!' } );

		// Returns the HTML of the page
		const pageHtml = await alice.getHtml( title );

		// Validates whether the text is present in the HTML
		assert.include( pageHtml, 'Hello, world!' );
	} );
} );

Edit a page

`edit()`
arguments	Page title (string) API:Edit parameters (object)
response	API:Edit#Response response You can also access edit properties using the `param_user`, `param_text`, and `param_summary` parameters.
example	edit( title, { text: 'Hello, world!' } )

Return a revision record for a page

`getRevision()`
arguments	Page title (string) Revision ID (optional) - If revision ID is 0 or not provided, returns the latest revision. API:Revisions parameters (object)
response	API:Revisions#Response response
example	getRevision( title )

Return HTML for a page with comments stripped

`getHtml()`
arguments	Page title (string)
response	API:Parsing_wikitext#Response response
example	getHtml( title )

Return the most recent changes entry matching the given parameters

`getChangeEntry()`
arguments	API:RecentChanges parameters (object)
response	API:RecentChanges#Response response
example	getChangeEntry( { rctitle: page } )

Return the newest log entry matching the given parameters

`getLogEntry()`
arguments	API:Logevents parameters (object)
response	API:Logevents#Response response
example	getLogEntry( { letype: 'delete', letitle: title } )

Calling the Action API

The Action API list, meta, and prop modules provide access to information about wiki pages and users. The testing tool provides methods that let you make GET requests to these modules within tests.

The List API

`list()`
description	GET request to list items that match select criteria. See the Lists API docs for available submodules.
arguments	API submodule (string) Submodule-specific parameters
response	See the Lists API docs for submodules responses.
example	list( 'usercontribs', { ucuser: `${fiona.username}\|${franky.username}`, ucprop: 'ids\|user\|comment\|timestamp' } )

The Meta API

`meta()`
description	GET request to fetch information which is not associated with pages(metadata). See the Meta API docs for available submodules.
arguments	API submodule (string) Submodule-specific parameters
response	See the Meta API docs for submodules responses.
example	meta( 'userinfo', { uiprop: 'options' } )

The Properties API

`prop()`
description	GET request to list properties of selected pages. See the Properties API docs for available submodules.
arguments	API submodule (string) Page title (string) Submodule-specific parameters
response	See the Properties API docs for submodules responses.
example	prop( 'links', pageX, { plnamespace: 0 } )

The Upload API

`upload()`
description	POST request to upload a file. See the API:Upload API docs for available submodules.
arguments	Submodule-specific parameters (object) file (string)
response	See the Upload API docs for submodules responses.
example	upload( { filename: 'file_1.jpg', token: await mindy.token() }, '~/file.jpg' )

Other Action API calls

To call any module in the Action API, use the action() method.

`action()`
description	Executes an HTTP request to the Action API and returns the parsed response body. This method fails if the response contains an error code. See the API docs for available actions.
arguments	Action name (string) Submodule-specific parameters POST (for POST requests)
response	See the API docs for action responses.
example	action( 'parse', { page: pageTitle } )

Some Action API calls require a token. See individual API action docs for token requirements. For example, to patrol a page, make a POST request to the patrol action. In the API:Patrol docs, we can see that this call requires a token, which we can get using the token() method.

const result = await mindy.action(
    'patrol',
    {
        title: pageTitle,
        revid: edit.newrevid,
        token: await mindy.token('patrol')
    },
    'POST',
)

The action() method fails if the response contains an error code. To test for expected errors, you can use the actionError() method.

`actionError()`
description	Executes an HTTP request to the Action API and returns the error stanza of the response body. This method fails if there is no error stanza.
arguments	Action name (string) Submodule-specific parameters POST (for POST requests)
response	See the API docs for error responses.
example	actionError( 'query', { list: 'recentchanges', rctitle: pageTitle, rcprop: 'ids\|flags\|patrolled' } )

Action API example test

The Recent Changes test suite contains two tests cases that validate a user's ability to access recent changes for a page.

'use strict';

// Loads required modules
const { action, assert, utils } = require( 'api-testing' );

// Defines a test suite called 'Recent Changes'
describe( 'Recent Changes', function () {
	// Defines a random page title prefixed with 'Recent_Changes_'
	const title = utils.title( 'Recent_Changes_' );
	// Defines the account we'll use in this test suite
	let alice;

	// Logs in to a wiki session using the alice fixture
	before( async () => {
		alice = await action.alice();
	} );

	// Defines the first test case
	it( 'should create page and get new page recent changes', async () => {
		// Has alice add the text 'Recent changes testing' to the randomly
		// defined page title, only if it does not already exist
		const edit = await alice.edit( title, { text: 'Recent changes testing', createonly: true } );

		// Has alice request the most recent changes for the same page
		const results = await alice.list( 'recentchanges', { rctype: 'new', rctitle: title } );

		// Validates that the most recent change creates a new page
		assert.equal( results[ 0 ].type, 'new' );

		// Validates that the recent change has the same titled as the edited page
		assert.sameTitle( results[ 0 ].title, title );

		// Validates that the page ID in the recent change is the same as the page ID that was edited
		assert.equal( results[ 0 ].pageid, edit.pageid );

		// Validates that the revision ID in the recent change is the same as the revision ID that was edited
		assert.equal( results[ 0 ].revid, edit.newrevid );
	} );

	// Defines a second test case
	it( 'should edit page and get most recent edit changes', async () => {
		// Has alice make a second edit to the page with the text 'Recent changes testing..R1'
		const rev1 = await alice.edit( title, { text: 'Recent changes testing..R1' } );
		// Has alice request the most recent changes for that page
		const results = await alice.list( 'recentchanges', { rctype: 'edit', rctitle: title } );

		// Validates the same data points as the previous test case
		assert.equal( results[ 0 ].type, 'edit' );
		assert.sameTitle( results[ 0 ].title, title );
		assert.equal( results[ 0 ].pageid, rev1.pageid );
		assert.equal( results[ 0 ].revid, rev1.newrevid );
	} );
} );

Calling the REST API

The testing tool provides methods that let you make requests to the MediaWiki REST API. To open a REST API session (usually within the top level describe() function or a before() hook), call the REST() function for an anonymous session, or pass an account object into clientFactory.getRESTClient().

To make a request, use the corresponding method for the HTTP request method:

get(): Make an HTTP GET request to the REST API
post(): Make an HTTP POST request to the REST API
put(): Make an HTTP PUT request to the REST API
del(): Make an HTTP DELETE request to the REST API

These methods can take up to three arguments:

The endpoint path as a string, starting after the version number (Example: /revision/${revId1}/compare/${revId2} for the compare revisions endpoint)
If supported by the endpoint, a request body or parameters as an object
If supported by the endpoint, headers as an object. POST and PUT requests have a default content-type of application/json

See the REST API docs for endpoint paths and request body requirements.

REST API example test

Here's an example test suite with two tests for the get revision endpoint.

'use strict';

// Loads required modules
const { action, assert, REST, clientFactory, utils } = require( 'api-testing' );

// Defines a test suite called 'Get revision'
describe( 'Get revision', () => {
	// Defines the account we'll use in this test suite
	let mindy;
	// Defines the REST API session we'll use in this test suite
	let client;

	// Logs in to a wiki session using the mindy fixture
	before( async () => {
		mindy = await action.mindy();
		// anonymous REST API session
		client = new REST();
		// alternatively, authenticated REST API session:
		// client = clientFactory( 'rest.php/v1', mindy );
	} );

	// Defines a successful test case
	it( 'should successfully get information about revision', async () => {
		// Defines a random page title prefixed with 'Revision'
		const page = utils.title( 'Revision' );
		// Has mindy add the text 'Hello World' to the randomly defined page title
		const { newrevid, pageid, param_summary } = await mindy.edit( page, { text: 'Hello World' } );
		// Makes a request to the REST API to get information about mindy's edit
		const { status, body } = await client.get( `/revision/${newrevid}/bare` );

		// Validates the API response properties
		assert.strictEqual( status, 200 );
		assert.strictEqual( body.id, newrevid );
		assert.deepEqual( body.page, { id: pageid, title: page } );
		assert.nestedPropertyVal( body, 'user.name', mindy.username );
	} );

	// Defines a failed test case
	it( 'should return 404 for revision that does not exist', async () => {
		// Makes a request to the REST API with a known invalid parameter
		const { status } = await client.get( '/revision/99999999/bare' );

		// Validates that the API returns a 404
		assert.strictEqual( status, 404 );
	} );
} );

Contributing

The API tests are hosted on Gerrit and mirrored on GitHub. To open a patch request, see the guide to using Gerrit for Wikimedia projects.

To review open tasks or file a bug report, visit Phabricator.

Sharing feedback

To share your feedback about this page or to ask a question about API tests, leave a comment on the talk page.

↑ configs/example.json

[1] s/example.json

[1]