Continuous integration/Browser tests
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. See Selenium instead. |
This page describes how to have Jenkins execute a subset of your Selenium tests for every patchset submitted to Gerrit.
What the job does
[edit]Reusing much of the JJB configuration originally created for running QUnit tests, it:
- Checks out master branches of core and the extension dependencies along with the current Gerrit patchset
- Creates a fresh database and installation of MediaWiki to run on localhost
- Sets all the necessary MediaWiki-Selenium environment variables to point to the local MediaWiki installation
- Executes all scenarios in the test suite tagged with
@integration - MediaWiki-Selenium will start a local headless Xvfb display and execute the tests using the local browser you've configured in
environments.yml - Cleans up
Requirements
[edit]Your test suite must be using MediaWiki-Selenium 1.4 or greater (1.5 for video recording support), as it contains a few important fixes to Cucumber logging and browser startup, and a user factory system that creates new accounts for each scenario as they're needed.
Check your version of MediaWiki-Selenium by running bundle show mediawiki_selenium in the project directory. If you're already using a 1.x version, the upgrade should be trivial. Just increment the version in Gemfile and run bundle install (or bundle update if you run into dependency issues).
If you're running an older version like 0.4.1, the upgrade process will be a little more involved. Start by following the upgrade documentation, and reach out to #wikimedia-releng connect if you need help.
Setting up your test suite to run in CI
[edit]Once you're using MediaWiki-Selenium 1.4 or greater, add an integration section to the environments.yml file in your tests directory that will tell the CI builder how to run the tests. It really just needs to enable the user factory, and specify a browser (chrome or firefox).
integration:
browser: chrome
user_factory: true
# mediawiki_url: JENKINS WILL SET THIS
Next, tag some scenarios with @integration.
The general strategy here should be to choose scenarios that cover the most core functionality of your project without increasing the build time too much. Remember that these kinds of end-to-end tests are inherently slow and sometimes fragile; long build times and blocking false failures could be quite frustrating to your fellow developers. In general, try to achieve a decent looking test pyramid, not a cupcake.
Configuring JJB for your extension
[edit]Schedule the tests to run upon each commit to Gerrit by modifying the JJB configuration for your extension. You might want to start by adding or modifying the experimental pipeline, which you can manually trigger to run by adding a review comment check experimental to a patch in Gerrit. After you've vetted your set of tests for stability in this environment, move the job to the test and gate-and-submit pipelines. Again, hit up #wikimedia-releng connect if you need help.
In addition to scheduling the job itself, make sure Jenkins/Zuul knows which dependencies to set up for your extension by adding them to zuul/ext_dependencies.py in the integration/config project.
Video recordings and screenshots
[edit]MediaWiki-Selenium (>= 1.5) supports video recording of headless Xvfb based browser sessions. Recordings and screenshots for all failed scenarios will be kept as Jenkins build artifacts for up to 3 days. Look under "Build Artifacts" when viewing the build result in Jenkins—Gerrit links to the full console output so you'll need to follow the "Status" link in the left-hand menu to see the main result page.
Limitations
[edit]- Jobs are run with local browser for performance reasons, so only Chrome 43 and Firefox 39 are available
- The fresh MediaWiki installation only sets up one wiki, so you can't test interwiki features