Jump to content

User talk:Cmcmahon

Add topic
From mediawiki.org
Latest comment: 12 years ago by Cmcmahon


Cmcmahon 20:29, 7 February 2012 (UTC)Reply

Community Testing for WMF

[edit]

Community Testing/QA

[edit]

It would benefit WMF to have interested parties test various aspects of Mediawiki, Wikipedia, and related projects in the user interface, with various browsers, before they are released to the general public.

As I see it, the issue to be addressed is not to persuade more core/Extensions/Gadgets people to do more testing. Nor is the issue to attract more FLOSS community members to Mediawiki/Wikipedia testing.

Instead, the issue is to attract more people from the general software testing community to the effort to test Mediawiki/Wikipedia, particularly at a per-feature and per-release level. And, while doing so, to keep the amount of noise (NOTANISSUE, WONTFIX, etc.) to a minimum.

Two reasonable approaches for doing this suggest themselves.

Feature Testing by Exploratory Testers

[edit]

For a number of features, it is desirable to have as many people as possible investigate the feature, looking for issues and problems in various browser/OS/data/etc. combinations before such features are deployed to production.

An excellent example of such a potential testing project would be the Timed Media Handler (TMH). TMH has many parameters, ranging from the display of media files in various browsers, to the interaction of media files on the wiki(s) itself, to the nature of the files being uploaded.

However, it is not feasible that a community tester (especially Windows users) would be in a position to set up a local test environment for TMH. A shared test environment for features such as TMH is highly desirable.

Here is an excellent example from Mozilla: https://quality.mozilla.org/2012/03/web-qa-testday-for-the-affiliates-project-march-29-2012/ They have in place a dedicated test environment, an etherpad with test charters, and Bugzilla all set up to handle bug reports. Mozilla has "Testdays" of several different kinds: https://quality.mozilla.org/category/events/month

Note: Weekend Testing http://weekendtesting.com/ is a group of about 5000 Exploratory Testers in chapters in India, Americas, Europe, and Australia/NZ who gather on the first Saturday of every month (Americas at least) to do ET on some sort of publicly available software. They would be an excellent resource both to do feature/release testing and also to publicize the WMF community QA effort.

Automated Browser/UI/Mobile Testing

[edit]

WMF has no automated browser testing in place right now. This is actually an advantage, because the general environment for doing such testing has seen a surge of innovation in recent times, and those involved are forming consensus on the best approaches to doing so.

Selenium is far and away the tool of choice for doing browser automation. The Selenium bindings are maintained institutionally for four languages: Java, C#, Python, and Ruby. Java and C# are poor choices for WMF.

Between Python and Ruby, I would strongly recommend Ruby, for a number of reasons.

  • The culture of testing is strong in Ruby, less so for Python.
  • The Ruby/Selenium community is in strong agreement as to optimal approaches to browser automation: Page Objects, RSpec, rake, etc. the "gems" involved are all well-known and well-understood by the Ruby testing community.
  • There is a strong culture of philanthropy in the Ruby community. I've already had an offer to do a 2-day "GiveCamp" to get the WMF browser automation project underway, or to improve it once it exists.
  • Mozilla has the attention of the Python browser testing community. No project as famous as Wikipedia has tapped the Ruby testing community, which is potentially much larger and more expert than the equivalent Python community.
  • The way Page Objects are implemented in Ruby also supports Mobile testing. http://watirwebdriver.com/mobile-devices/ (Note: Watir today is a higher-level API that wraps Selenium 2.0)

Job #1: Reduce Barriers to Entry

[edit]

Whether for investigation of features before release, or for an ongoing browser automation effort, WMF should make a strong effort to reduce the barriers to entry for anyone in the community that wants to contribute to the testing effort.


  • Good shared test environment(s)
    • make Labs useful to strangers
      • Have the correct version of MW in place.
      • Have the test wiki(s) configured properly (e.g. currently there is no working link to upload a file on beta Commons, one has to know the URL for UploadWizard.)
    • Create test envs other than labs beta?
    • encapsulate Mediawiki for download e.g. Bitnami? (nice to have)
  • Prepare
    • Relevant documentation (test plan, etc.) in place before start of testing
    • Modern, attractive frameworks to automate with
    • Support for testers via IRC, mail, etc. (i.e. Mozilla)
    • Have Bugzilla set up for Community QA reporting, with proper categories and a published procedure for reporting issues found

Selenium/Ruby project

[edit]

Prequisites for Awesome Selenium Testing

Framework

  • Ruby
  • selenium-webdriver
  • watir-webdriver
  • page-object
  • rspec
  • rake

See https://github.com/chrismcmahon/Page-Object-WMF-spike for reference implementation.

WMF host

After doing some research on the apt vs. gems issue, it seems like the approach most of the Ruby community takes is to keep apt and Ruby completely separate. In practice this means getting RVM (Ruby Version Manager) from github, installing it, and then using RVM to install and manage Ruby versions and gems. The link above describes the practice in general. I tested this out on a labs instance and it worked perfectly. RVM also solves the issue of maintaining Ruby infrastructure without sudo; and since RVM contains everything within user space, not root, it's more secure.

I polled some Rubyists in the testing community: Convio has RVM/gems on their prod servers. Groupon has RVM/gems in their CI and staging envs, and for prod they have abandoned apt altogether and use their own package manager.

Test Environment

Test Data Management

  • Need strategy for test user with publicly available user/pass who can log in on labs beta wikis
  • Possibly need scheme to reset test data, dbs, etc.

Jenkins stuff

  • Deploy from trunk or release candidate to labs beta wikis
  • Kick off rake task, gather results

success factors for automated browser testing

[edit]

Everything happens in the browser

[edit]
  • Don't manipulate db tables
  • Don't manipulate files on disk

No hacks

[edit]
  • Every aspect of the test framework should Just Work

Let the user write tests out of the box

[edit]
  • get the framework/scaffolding out of the way, out of sight

Defined test environments

[edit]
  • Test suite for clean install
  • Test suite for maintained shared test env. e.g. Beta Labs

Dynamic test data

[edit]
  • Get test data at runtime
    • Extensions can help
    • If necessary create REST API exposing XML for browser to read
      • Note: data must be represented as an 'attribute' of the XML node(s) because IE is retarded

Issues

[edit]

1.20wmf2 research

[edit]

ahead of official RELEASE NOTES:

in wmf2 not in wmf1

  • Added GitViewers hook for extensions using external git repositories to have a web-based repository viewer linked to from Special:Version.
  • Memcached debug logs can now be sent to their own file logs by setting $wgDebugLogFile['memcached'] to some filepath.
  • (bug 35685) api.php URL and other entry point URLs are now listed on Special:Version
  • Edit notices can now be translated.
  • (bug 22887) Add warning and tracking category for preprocessor errors
  • (bug 34428) Fixed incorrect hash mismatch errors in the DiffHistoryBlob
  • (bug 34929) Show the correct diff when a section edit is rejected by the spam
  • jQuery UI upgraded to 1.8.19.
  • (bug 35572) Blocks appear to succeed even if query fails due to wrong DB structure
  • (bug 31757) Add a word-separator between help-messages in HTMLForm
  • (bug 30410) Removed deprecated $wgFilterCallback and the 'filtered' API error.
  • (bug 36012) Space in $separatorTransformTable should be non-breaking in Portuguese, Esperanto and Udmurt.

1.20wmf2 Feature Test Charters: DRAFT IN PROGRESS

  • Investigate diff displays
    • Trigger spam on edits; check diffs (spam: ALL CAPS, drug names, obscenity, etc. USE SANDBOX OR USER TALK PAGES! )
  • Validate Special:Version display and data
  • Validate appearance and content of Help messages in various locations
  • Investigate edit notices in languages other than English
    • Portuguese, Esperanto and Udmurt of particular interest

things to be checked upon deploy (automatically or otherwise)

[edit]

ProofreadPage (wikisource) UploadWizard (commons)

Retrieved from "https://www.mediawiki.org/w/index.php?title=User_talk:Cmcmahon&oldid=547962"