Selenium/PHP/Test framework deployment

Group:	Release Engineering
Start:	2011-07-01
End:	2013-12
Team members:	Markus Glaser, Ryan Lane, Rob Lanphier, Mark A. Hershberger, Priyanka Dhanda, Dan Nessett

This is a project to get an integrated testing framework in place.

Markus Glaser added initial Selenium testing support to the repository,
Ryan Lane has been building a cluster,
Priyanka Dhanda has been refining the Selenium configuration
Mark Hershberger integrating with CruiseControl to run both Selenium and PHPUnit tests
Michelle Knight has been recording Selenium tests

Status: Permanent hiatus

Update 2013-12-31: Browser testing has been revived using cucumber: More details. So this page about test framework deployment has only archival function. --Mglaser (talk) 13:25, 31 December 2013 (UTC)[reply]

Update 2011-07-01: We sort of set up Selenium Grid, which automatically runs Selenium tests in different virtual environments (each with a different browser/OS etc.).

But most contributors ignored Selenium; they don't find its output useful. And Selenium Grid is an ops headache (it's a chore to maintain the VMs, and sometimes a Selenium Grid test failure actually indicates a Grid or virtual machine problem rather than a broken test!). Also, Selenium Grid's scope was limited: It remotely controls browsers in the VMs, so unless Selenium supported an OS and browser, it couldn't be part of the 'grid'.

WMF shut down their Selenium Grid VMs a few weeks ago in favor of running QUnit with Jenkins. Sumanah 18:56, 1 July 2011 (UTC)[reply]

Feature justification

Currently, we do not have a lot of formal testing of MediaWiki, except for some usability work that Wikimedia Foundation has worked with Calcey to provide. Automated testing has also been lacking.

Selenium provides a means of doing automated system testing. This work would be complementary to work on Apache Continuum, which can provide us the means of continuously running Selenium tests, and with PHPUnit, which provides us with a framework for unit testing.

User requirements

See Test framework deployment/Requirements

Specification

Documentation about the existing Selenium Framework (in progress)

Software design document

Documentation about the existing Selenium Framework (in progress)

Test plan

Documentation plan

User interface design docs

Schedule

Milestones correspond to status meetings:

Sept 17 - send out proposed configuration (with prototype code)
Sept 24 - more or less working dynamic configuration, with a test or two actually using it
Oct 1 - dynamic config problem solved, and a couple of extension tests, initial test runs on the grid
Oct 8 - running better on the grid
Oct 15 - running smoothly on the grid
Oct 22 - finished

Task management

(e.g. link to relevant Bugzilla queries)

Release management plan

Community management plan

Status updates and notes

Automation work done by the Calcey team

Calcey has worked on basic automation of Wikimedia usability, using Selenium.

Test scripts are recorded using Selenium IDE 1.0.7 and converted to PHP. Test scripts were implemented such that data and code can be maintained individually and are reusable.

The automation suite can be found on the <<svn+ssh://svn.wikimedia.org/svnroot/mediawiki/trunk/testing/selenium/UsabilityInitiative/WikiAutomationTC/testCases>> location. Recorded test scripts can be executed using the NetBeans IDE 6.8 which was integrated with PHP Unit and Selenium module for PHP.

Automated scripts can be run individually as separate scripts or complete test suite can be run at once. Test results will be generated with status of passed or failed for each and every test scenario available in the test scripts.

Functionality automated using Selenium.

1. Search functionality

Search for a wiki page and verify search results directed to the page
Search for a text and verify the search results contain the required text

2. Watch page

Mark a page as watch and verify the My Watch list
Mark a page as watch and then unwatch and verify the My Watch list
Mark a page as watch on page edit and verify the My Watch list

3. Adding header levels

Add header level 2, 3, 4 and 5 and verify header outputs
Add header levels 2 & 3 and verify header output
Add header levels 2, 3 & 4 and verify header output
Add header levels 2, 3,4 & 5 and verify header output

4. Editor

Add a internal link and verify the link on wiki preview
Add a internal link with different display text and verify the link on wiki preview
Add a internal link with blank display text and verify the link on wiki preview
Add a external link and verify the link on wiki preview
Add a external link with different display text and verify the link on wiki preview
Add a external link with blank text and verify the link on wiki preview
Add a table and verify the table
Add a table only with header row and verify the table
Add a table only with borders and verify the table
Add a table without headers, borders and sort rows and verify the table
Add a table with headers, borders and sort rows and verify the table
Verify the search and replace function
Verify Match Case option with ‘Replace All’ option
Verify regular expression option with ‘Replace All‘ option

5. Toolbar

Verify media wizard function and verify the output
Add reference file function and verify the output
Add picture gallery function and verify the output

6. Text format

Add a bold text and verify the output on the wiki preview
Add a italic text and verify the output on the wiki preview
Add both italic and bold text and verify the output on the wiki preview
Add bulleted items and verify the output on the wiki preview
Add numbered items and verify the output on the wiki preview
Add a no-wiki text and verify the output on the wiki preview
Add a line break and verify the output on the wiki preview
Add a big text and verify the output on the wiki preview
Add a small text and verify the output on the wiki preview
Add a superscript text and verify the output on the wiki preview
Add a subscript text and verify the output on the wiki preview