Reading/Web/QA

We use manual testing as well as automated testing to QA our products.

Our manual testing frequently occurs as part of our sprint process. After code has gone through engineers and design we hand it over for QA.

Manual handover

First it is important to check we have instructions on how and where to test the change in the description, given we have many test environments identifying where testing can take place.

We have 2 environmentsː

The beta cluster, which updates frequently as code is merged
reading-web-staging.wmflabs.org, which is specialized for testing article content

Depending on what is being tested usually one is more suitable than the other. It's extremely important to clarify where to test to avoid testing false positives.

staging

Staging can be updated by ssh'ing into the machine (readingwebstaging.eqiad.wmflabs). Always run sudo su mwvagrant before doing anything else so that the logs of what you are doing can be seen by other users.

Updates usually involve either of these commands:

cd /srv/mediawiki-vagrant/
vagrant git-update
git fetch && git checkout origin/master

If this doesn't work, it may be because a change has been cherry picked (see below). In this case, you should either rebase the cherry pick onto master or revert back to master if the change is no longer needed.

cherry picking changes

Sometimes for risky code we may cherry pick changes to staging. Right now we don't have a good process for managing this kind of change other than commenting on the tickets that need these changes.

Changes are last in first out so there is always a risk staging is left in an unexpected state. It is up to the tech-lead to keep track of these changes.

When cherry picking changes, always refer to the "design sign off" and "QA" columns of the kanban board and when done leave a comment.

LocalSettings

It may also be necessary to update LocalSettings.php (either to disable/enable the content provider) or to turn on new feature flags.

On staging, there is a file /vagrant/mediawiki/Staging.php where changes should be added to.

When making a breaking change (e.g. switching language) we should mail the team mailing list. Recently, I've been exploring making use of wgSiteNotice to inform users of the wiki without need for email. You'll find some commented out lines down the bottom of the Staging.php file.

Automated testing

Browser tests

What are browser tests? Quality_Assurance/Browser_testing

Node.js

Selenium/Node.js

What are browser tests for?

A browser test should ideally be written to test behaviour. Wherever possible, favor unit tests. Usually a browser test is warranted in one of the following scenarios"

You are testing behaviour of a workflow dependent on one or more events e.g. "When I click the browser back button the overlay closes"
The thing being tested is dependent on configuration either on user or site level e.g. "When I have disabled page previews I do not see page previews"
The behaviour of the thing being tested is subject to change by another extension "e.g. "When VisualEditor is installed, the default editor is the wikitext editor."

Running browser tests on the beta cluster

Set the following variables, using an existing account for username and password:

export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
export MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php
export MEDIAWIKI_ENVIRONMENT=beta
export MEDIAWIKI_USER=YourUsername
export MEDIAWIKI_PASSWORD=***

Note that certain tests may require your username to have certain permissions set.

Simulating browser test run on an unmerged patchset.

The way we test such change is to create a copy of the existing Jenkins job, disable all notifications and then fetch and checkout the patchset.

Steps:

For example, you want to test 298959
Log in to Jenkins
Click New Item link in the left sidebar
A form would show
- Item Name: selenium-(repository)-(gerrit-patch-number), so in this example selenium-Wikidata-298959
- Copy existing Item: selenium-(repository), so in this example selenium-Wikidata
- Save form
Go to the job configuration
- Click Source Code Management > Git > Advanced
  - Change Refspec from +refs/heads/*:refs/remotes/origin/* to refs/changes/59/298959/1
    - You can get refs/changes... from Gerrit patch
      - Go to the patch 298959
      - click Download
      - URL in Pull will be git pull https://zfilipin@gerrit.wikimedia.org/r/wikidata/browsertests refs/changes/59/298959/1
      - You only need the last section, refs/changes/59/298959/1
  - Change Branch Specifier (blank for 'any') from master to FETCH_HEAD
- Uncheck Build periodically
- Delete the sections:
  - Execute shell (only one that starts with # Attempt to figure out MediaWiki branch being used and fetch it out)
  - Editable Email Notification
  - IRC Notification
Save and trigger a build of that item-name to run browser tests.

Troubleshooting "flakey tests"

Sometimes our beta cluster integration jobs fail, even though they pass when we merge code and a manual test shows that functionality is working as expected. This is often due to problems with the beta cluster's availability or performance and how the tests are written not the thing they are testing.

Use you can point Selenium jobs to unmerged patches, to test changes to browser tests by cloning an existing job by clicking "New Item" on the dashboard and using "copy from". This results in a duplicate job with a different name e.g. https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue-BugFix/

You can use this job to point run tests from another code path:

Click configure
Find on page "gerrit". In build "Execute shell" you'll see an explicit checkout command. You can update this line to check out any code for that repository.
Run the build (using "Build now")

Current browser test results

Dashboard