User:Jdlrobson/Experiences porting Ruby browser tests to NodeJs

Recently the Release engineering team dropped support for browser tests written in Ruby. As one of the biggest clients of this codebase, this has created a large amount of work for the reading web team. This page aims to collect our experiences and request support in making this rewrite easier.

Recommendation: Provide Browser test specific Node.js library to speed up stub creation

The Ruby framework was written over a period of 5 years and has led to the creation of common utility libraries such as mediawiki-selenium.

In Node.js the nodemw library is provided. This library is a low level general purpose module for interacting with the MediaWiki API and thus does not provide the sort of libraries we commonly need when writing browser tests. For instance, there is no helper method for common actions, which require multiple steps and are cumbersome to achieve such as:

The nodemw library makes token management very difficult as it requires a good understanding of how the API works and different queries work together.
- There are no helper methods for watching a page
- There are no helper methods for unwatching a page
The nodemw library does not return JavaScript Promise's which leads to difficult to read code and inability to use the await operator
User creation for the purpose of running tests is tricky. Creating and making use of users with different characteristics could be easier e.g.:
- creating a user with Echo notifications
- creating a user and then obtaining the user's new username
- creating a new user with an edit count of 5 and then logging in as that user inside the test environment
- creating a brand new user and then logging in as that user inside the test environment
- ensuring you are a user with admin permissions and protecting a page
Article editing could be easier
- creating stubs with certain text (login, then create page)
- batch edits (e.g. create a page with 51 edits)

In addition to this there are common actions that we need to perform in browser tests that are not API specific. For instance

checking if certain ResourceLoader code is loaded. This should be a helper method on ArticlePage.
Selecting elements should be quicker and easier (a single line of code using a PageObject)

It would be even more useful if a common library existed that provided various stub pages with different characteristics. Most of our test writing involves setting up stubs and some tests are resource extensive - for instance a page with 51 edits could be shared across all tests rather than created on every test that needs it.

Recommendation: Cleanup import paths by creating and publishing a mw-selenium-page-objects library

1) Importing pages is a pain as it requires knowledge of the current folder and where the core folder is. The use of `..` leads to very unreadable code e.g.

const UserLogin = require( '../../../../../tests/selenium/pageobjects/userlogin.page' )

Could we please package these into a library to make this easier?

npm install mw-selenium-page-objects
const UserLogin = require( 'mw-selenium-page-objects' ).UserLogin

Recommendation: Make it easier to run tests

Previously it was possible to run browser tests without an instance of core. We would install a gem and then run the tests locally. Since we have moved to Node we have lost this and now there are multiple steps which makes setting up and running tests cumbersome - we have to open multiple terminal tabs to start chromedriver and then execute the tests from the mediawiki/core folder. This is even more painful when repos do not follow the expected directory structure - for instance if an extension is actually included via symlink and thus it is not possible to cd up into core two levels.

We would benefit from this to be a local job that can be invoked from package.json inside the repo the browser tests belong to.

Recommendation: Provide better debugging tools

Previously, when running tests against Jenkins we got screenshots and a video. The video was essential for debugging various issues. It seems we have lost this in the move to Node and when rewriting browser tests for RelatedArticles it was extremely hard to debug issues which worked locally but not against Jenkins.

Recommendation: Provide low level methods on ArticlePage.

Creating page objects for every single page is very cumbersome (it was in Ruby as well, but we built those tests over several years rather than in 1-2 sprints). Especially, given the majority of tests just need to click or verify the existence of certain elements.

It would be useful to have some helpers on ArticlePage for common tests e.g. check element exists, clickElement, seesElement.

When working on https://gerrit.wikimedia.org/r/373902 this was much more friendly then having to setup a class for every type of page, but I suspect this goes against the PageObject model....

Recommendation: Clarify in documentation use of PageObject model

Using the PageObject model has caused us much confusion as various tests use pages with different qualities. For instance, consider a test which requires a protected page and watches it on the mobile version of a page (and then again on a desktop version of the same page). It's not clear in such a situation if we need a MobilePage, ProtectedPage, MobileProtectedPage, MobileWatchedPage, WatchedPage and a ProtectedWatchPage and MobileProtectedWatchPage or there is some better way of doing such a thing.

It's not clear how to mix and match pages with different properties without utilising multiple inheritance and we've run into some meta conversations and confusion within our team and with release engineering.

Please provide guidance.

Recommendation: Provide some example code

Most of the Ruby code was written by reading through existing code examples and copying much of the patterns. In working on the porting of MobileFrontend and Minerva we have been advised to follow the page object model. Since the Node.js stack is new, very few examples exist.

We recommend that https://www.mediawiki.org/wiki/Selenium/Node.js should have an FAQ detailing common code snippets to make porting easier. Right now no such page exists. This lack of guiding principles causes lots of tension in code review.

Such a page would be useful if it included information detailing:

all the above difficulty points
how to create a page
how to use the browser back button
how to disable JavaScript
how to switch between resolutions e.g. mobile (320px), tablet (720px), desktop (1000px)
how to login (as the default user or as a brand new user)
how to get the current user and visit their user page
how to apply a query string parameter to a Page
selecting elements by text e.g. how would you mimic the `element(:name, "textcontent")` selector pattern in JS?
how to exclude a certain test from a certain environment e.g. "do not run tests on commit" but do run them against the beta cluster.
how to visit a page with a hash fragment e.g. https://en.m.wikipedia.beta.wmflabs.org/wiki/Selenium%20section%20test%20page#Section_2A