Selenium/Ruby/Browser testing
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. See Selenium instead. |
|
Browser testing
Automated testing of MediaWiki within various browsers to improve quality
|
Rationale
[edit]Why automate tests at the browser level? They find bugs!
Browser tests are most useful for testing:
- JavaScript: different browsers interpret JS in different ways.
- Features requiring navigation: unit tests don't usually navigate.
- Fragile Features: if something breaks often, we want to test it regularly.
- Critical Features: features that must work all the time.
- Beta features: checking that they work in all browsers.
It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.
- We run periodic browser tests via Jenkins on SauceLabs to test the latest master code on the beta cluster in multiple browsers
- Some projects run browser tests on each check-in to gerrit as part of Continuous integration, see Continuous integration/Browser tests.
We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.
How to contribute
[edit]Investigate a broken issue
[edit]- Analyze failing tests in Jenkins browser test runs.
- Investigate a recent test that's failing
- Drop in to #wikimedia-releng connect and ask if it's a known issue.
- Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
- Watch the screencast of the failing test and/or view the test's code.
- If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.
This doesn't require any local code at all, it's all on the web.
Fixing a broken issue
[edit]The next level is to fix broken tests and write new ones.
- Follow the browser testing setup instructions
- Run browser tests yourself, pointing to some MediaWiki install
- If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
- If you find an error in a test, file a bug.
- Going further, contribute a fix for it in gerrit.
- You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.
None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.
Resources
[edit]- Current test results.
- We use given-when-then testing format.
Subpages
[edit]
External links
[edit]- Cross-browser testing is a resource often use for specific testing