Selenium/Ruby/Running tests

TL;DR

gem install bundler
git clone <project>
cd <project>/tests/browser
bundle install
bundle exec cucumber [features/some.feature[:line]]

Setup

You may not need to run browser tests at all. If our Continuous Integration reports that an automated test is failing, you can view its code, manually go through its steps in your browser, and if it fails for you, file a bug against the MediaWiki component that has the problem.

However, if you're writing a test or investigating a test, it helps if you can run the browser tests yourself. You don't necessarily need your own local MediaWiki instance, as the tests can load web pages on some MediaWiki host such as test2.wikipedia.org or en.wikipedia.beta.wmflabs.org.

You do need:

• a browser that supports the test framework
• the code of the tests
• the cucumber test program and all its supporting bits and pieces

Browser support

Recent Firefox and Chrome/Chromium have the WebDriver automated testing support built-in. So too does PhantomJS, the "headless" WebKit browser.

Getting browser test code

For the most part, browser tests reside in the particular MediaWiki extensions being tested, such as VisualEditor and Flow, in a subdirectory named tests/browser. All these projects are on gerrit.wikimedia.org and are mirrored to Github.

Use git clone to get the relevant repository, for example

$  git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend

or

$  git clone ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor

(Note that VisualEditor is the only extension with modules, so find the browser tests under /Visualeditor/modules/ve-mw/tests/browser/)

Common code used by all the browser tests in all the extension repositories is maintained in a Ruby gem called mediawiki-selenium gem.

The mediawiki_selenium gem contains shared steps code such as

• login
• visit RandomPage
• reset Preferences
• check for ResourceLoader errors

and manages common aspects of running tests such as

• starting browsers both local and on Sauce Labs
• managing browser capabilities
• running headless tests
• running browser tests in Jenkins
• managing browser behavior according to environment variables such as KEEP_BROWSER_OPEN and REUSE_BROWSER

Getting the necessary software

Ruby software for browser testing

You can either

• follow the instructions in mediawiki-selenium to configure Ruby and install cucumber and supporting code for your system
  • Quality Assurance/Browser testing/Setup instructions has detailed guidance on installing Ruby, git, etc.

or

• install the MediaWiki-Vagrant virtual machine, and in it enable the browsertests role which will install all the necessary code.

The first takes some work, it's not too bad if you are running Linux or Mac OS X.

The second may be more than you need if you are going to run tests targeting a public MediaWiki instance, since it configures an entire local MediaWiki installation.

Browser support

A browser must implement the WebDriver interface so that tests can "drive" it. This is built into Firefox (the default browser assumed by our browser tests); for Chrome/chromium you need to install it, e.g. the package chromium-chromedriver. See the chromedriver page for more information.

To run a regular browser without it drawing windows on your screen, you need to install xvfb so it can draw to an invisible "X virtual framebufffer". In Ubuntu or Debian, something like sudo apt-get install --no-install-recommends xvfb

You can also run tests in the PhantomJS headless WebKit browser. Recent versions have built-in WebDriver support.

Running the tests

For each test run you must tell Cucumber the wiki to visit, supply some additional information if the tests log in, and name the features to test.

• Specify the target wiki that the browser will visit with export MEDIAWIKI_URL=URL/to/target/wiki/ , e.g. http://127.0.0.1:8080/wiki/ for MediaWiki-Vagrant.
• Tests that use the MediaWiki action API need a export MEDIAWIKI_API_URL=URL/to/api.php ,e.g. http://en.wikipedia.beta.wmflabs.org/w/api.php for Betalabs.
• For tests that login:
  • create a user named Selenium_user on the target wiki
  • specify the Selenium_user password on the target wiki with export MEDIAWIKI_PASSWORD=Selenium_user_password
    • the Selenium_user passwords on some of our test environments where we run Continuous integration tests are stored in Jenkins configuration.
• In the browsertests/tests/browser directory, type bundle exec cucumber to run all tests.
• To ignore tests tagged with @wip (works in progress), run cucumber with the -w flag. E.g. bundle exec cucumber -w
• To run tests tagged for a particular browser or host computer, run cucumber with the --tags @some_annotation flag. E.g. bundle exec cucumber --tags @firefox --tags @en.wikipedia.beta.wmflabs.org. Cucumber tags explains these annotations in more detail.
  • • the tags for each of our Continuous integration browser test jobs are set in its configuration script, click "XXX"

• To run a single test, in the browsertests/tests/browser directory enter bundle exec cucumber features/foo.feature:NN , where foo.feature is the name of the feature test and NN is any line within the particular scenario you want to test.
• The browser by default is Firefox. You can change the browser by setting the environment variable BROWSER, other choices include chrome and phantomjs. PhantomJS is a headless WebKit browser.
• The browser specified by the BROWSER opens windows on your screen (unless it is phantomjs). On Linux with X11 you can run browser tests in a headless mode by setting the environment variable HEADLESS to true. This makes the browser windows appear on an invisible Xvfb virtual framebuffer.
• By default, the browser will close itself at the end of every scenario. You can keep the browser windows around by setting the environment variable KEEP_BROWSER_OPEN to true. (This makes no sense for an invisible run using HEADLESS or phantomjs.)
• You can get screenshots of failures by setting the environment variable SCREENSHOT_FAILURES to true. This will capture the screen after errors and save to a screenshots subdirectory.

You can export these environment variables, or put all these variables on the command line before invoking bundle exec cucumber, thus

$  HEADLESS=true \
   SCREENSHOT_FAILURES=true \
   BROWSER=chrome
   MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ \
   MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php \
   MEDIAWIKI_USER=Selenium_user \
   MEDIAWIKI_PASSWORD=SomePassword \
        bundle exec cucumber features/search.feature        

Running browser tests at Sauce Labs

Sauce Labs gives you access to a variety of browsers and platforms for testing. For a full list of browsers and OSes supported by Sauce Labs, see their platforms page (the specific Linux variant used by Sauce Labs is Ubuntu LTS). If you don't already have a Sauce Labs account, you can get a free limited account at the Sauce Labs signup page (choose either a Free account or an Open Sauce account).

Before running a browser test via Sauce Labs, you'll need to define five environment variable values:

1. BROWSER
2. VERSION
3. PLATFORM
4. SAUCE_ONDEMAND_USERNAME
5. SAUCE_ONDEMAND_ACCESS_KEY

The first three values are selected from the set of possible values defined on the Sauce Labs platforms page. The last two values are your Sauce Labs user credentials.

For example, to run a browser test via Sauce Labs with Windows XP and Firefox version 27, you would define environment variable values as follows.

Linux/Unix/Mac:

export BROWSER=firefox
export VERSION=27
export PLATFORM="Windows XP"
export SAUCE_ONDEMAND_USERNAME=(username)
export SAUCE_ONDEMAND_ACCESS_KEY=(key)

Windows:

set BROWSER=firefox
set VERSION=27
set PLATFORM="Windows XP"
set SAUCE_ONDEMAND_USERNAME=(username)
set SAUCE_ONDEMAND_ACCESS_KEY=(key)

Replace (username) and (key) with your Sauce Labs username and key. You can get the key at https://saucelabs.com/account at the bottom-left part of the screen.

Then if (for example) you want to run the Go to Log in page scenario, you can execute the test with the usual Cucumber command syntax:

 bundle exec cucumber features/login.feature:18

Running VisualEditor browser tests via Vagrant

Some of the browser tests have been re-located to the specific code repositories for which they are most relevant. An example is browser tests for Visual Editor, which can be found in the VisualEditor GitHub repo. To run these tests in the MediaWiki-Vagrant virtual environment, a couple of extra steps are necessary in order to make the Cucumber script itself findable. Here are specific steps that should do the trick:

(in the directory where you have cloned the vagrant repo):

$ vagrant enable-role visualeditor

$ vagrant up
$ vagrant provision

$ vagrant ssh -- -X
$ cd /srv/VisualEditor/VisualEditor/modules/ve-mw/test/browser
$ export MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/
$ export MEDIAWIKI_USER=<username>
$ export MEDIAWIKI_PASSWORD=<password>
$ export PATH=$PATH:/home/vagrant/.gem/ruby/1.9.1/gems/cucumber-1.3.8/bin
$ bundle exec cucumber features/<featureName.feature

Enabling the visualeditor role not only gives you the VE tests, it configures the Vagrant virtual machine to run a local MediaWiki instance with the VisualEditor extension, so you can manually exercise VisualEditor on your local version. To verify the local MediaWiki instance is running successfully, visit http://127.0.0.1:8080/wiki/Main_Page

Running Math extension browser tests via labs-vagrant

Similar to running visual editor browser tests on vagrant, browser tests for the Math extension can be run on labs-vagrant in the following way.

Once you are logged into your instance all you need to do is:

cd /vagrant/mediawiki/extensions/Math/tests/browser/
bundle install
HEADLESS=true bundle exec cucumber features/*.feature

Extending timeouts for slow tests

Tests have a default 5 second timeout. This should be adequate for most operations against production or the beta cluster. If you regularly get errors running tests on a slower test wiki, you can locally modify steps. Any "when_present" clause will take an arg in seconds, e.g. when_present(10) or wait_until(10).

See also

• Writing browser test code
• Debugging browser tests
• Continuous integration/Browser tests to automatically run browser tests on each commit