Language Testing Plan
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
Automatic Browser Tests
[edit]The Language Engineering team is collaborating with the QA team to build automatic browser tests for its extensions and features. The scenarios are written in cucumber and implemented in ruby. The following section lists the test scenarios for several feature, which can then be moved to gerrit and github.
Test Scenarios
[edit]Manual Integration Tests
[edit]Bug Triage
[edit]The team hosts open bug triage sessions on the 4th Wednesday of every month, at 1700UTC on #mediawiki-i18n connect.
Announcements are made over mailing lists and also marked in the Project:Calendar
The team also carries out regular bug triages whenever required. Notes from these sessions can be found in the following section.
Notes from past sessions
[edit]- 28 August, 2013 - Right-to-Left language bugs
- 29 May, 2013 - Internationalization bugs
- 24 April, 2013 - Translate (TUX) bugs
- 22 March, 2013 - Narayam, Webfonts and Universal Language Selector
- 12 March, 2013 - Narayam, Webfonts and Universal Language Selector
- 22 August, 2012 - i18n bugs in Mediawiki extensions
- 9 May, 2012 - Translate feature bugs
- 18 January, 2012 - Webfonts
- 14 September, 2011 - Miscellaneous i18n bugs
Bug Stats
[edit]- List of all open bugs for the i18n components
- New bugs over the past week
- Bug Closed over the past week
Test days - Plan overview
[edit]THE FOLLOWING SECTION IS ONLY A DRAFT
This section outlines the plan for a pilot series of testing events of tools developed by Wikimedia’s Language Engineering team. The plan is based upon the current development release and the testing framework may change depending upon the focus of each release cycle and the feedback about the previous testing cycle.
Pre-Event Tasks
[edit]- Mark completion stage within a Release Cycle (i.e. mapped to the Mingle-based sprints) for testing. The test events in the series are:
- One big test day approximately half-way into the release (in translatewiki.net or in a deplyment in WMF Labs)
- Another big test day before the final bugs-based sprint, on a dark-launched version of the feature
- One more before the public launch (optional)
- Set up a lab/Virtual environment with the released version of the component(s) to be tested
- Prepare test scenarios for the components and build test cases for each
- Prepare complete walkthrough instructions for the test-cases
- A wiki page (or any other better option) with the test cases + instructions + link to record results to be set up
- Results should have options to provide (a Google spreadsheet+form may be useful):
- V and X results for pass and fail
- Additional Notes
- Screenshots
- Bug number
- Add an option to file a bug (through a template) for an issue (but don’t make it mandatory for the testers)
- Known issues to be listed to avoid testers spending too much time on them
- Set a 7-10 day window for the event to be open (Timezone and volunteer time is of concern)
- A dry run of the test event is done by a team member (QA person)
- Event is opened and emails sent to:
- appropriate mailing lists <To-Be-Decided>
- social media
- etc
- Sent out communication should include:
- Links to testing event instructions
- Google hangout/IRC where discussion related to the testing event will be done
- Ways to get information if no one is around to provide live help
Tasks during the Event
[edit]- Test organizers (or people with enough information to help testers) must be present on Google Hangout/IRC
- Someone with knowledge on how to fix the lab installation if it’s broken, to be around
Post-Event Tasks
[edit]- Go through all the feedback received from the test results
- File bugs/issues for inclusion into the development cycles
- Prepare event reports
Notes:
- Types of Testing
- A development release cycle based test event can target ‘Regression’ and ‘New Features’ tests
- A general test event can be used for smaller general updates (such as changes in Message or Language class, jqueryMsg, cldr, etc. as these happen quite often)
- Frequency
- A development release cycle can have at least 2 test events
- The second one before final bugs-based sprint to catch any potential blockers
- A general test event can be done less frequently
- Test Cases
- These are set based upon the type of test event
- Previously written testing instructions can be reused for regression testing
- Deployment requirements for the testing environment
- Scripts to automatically deploy MediaWiki:
- In the needed version (Git checksum)
- With the right extensions
- In multiple languages (all or a defined subset for testing)
- With at least some content in those languages
- With relevant interface customizations from the respective Wikipedia (or another project): gadgets. common.css, common.js etc.
- Pony features
- A test dashboard that will list:
- Upcoming test days
- Link to testing environment
- Link to start test that leads to the test instructions, and recording the test results
- Benefit: This can be used as the go-to place for anything related to Language testing and only one URL can be handed out when communicating about test events