Selenium/Ruby/Environment abstraction layer

Project overview

The Environment Abstraction Layer component of the mediawiki_selenium framework was started as part of a effort to: 1) improve test determinism by enforcing an outside-in read-only environment configuration; 2) provide constructs around common MediaWiki-related resources that will simply test patterns; 3) better isolate test implementation that requires multiple sessions; and 4) allow for custom configuration of browser settings such as language, proxies, etc. without having to add support to the core framework.

Environmental contract

We face limitations in the "given" steps of test implementation when it comes to setting up initial application state. Lacking the database access required for test fixtures we must, instead, rely on other methods of resource provisioning such as the MW API or Selenium-driven configuration via the UI. For resources that cannot be setup using either method—such as instances of wikis themselves and additional users (on account of CAPTCHA)—we rely on a handful of prerequisite environments that already have those resources in place: Beta Cluster, Wikipedia Test2, and local instances of MediaWiki-Vagrant.

Because these environments can differ in the way they're setup, it's important the we be able to articulate to our test suite a consistent set of assumptions about the state of the resources therein. For instance, "this is the base wiki URL," "this is the sandboxed test user," "this is where you can make API calls," etc. In more advanced use cases, this can include things like "this is a new user, you can assume it has no edit history." In a way, these bits of configuration can be looked at as a contract between the environment and the test suite.

Configuration is currently accomplished using a simple set of environment variables that are referenced in code using Ruby's ENV global. This is sufficient for many use cases, but it doesn't quite have the contract-like qualities necessary to ensure deterministic test behavior. For one, there's no guarantee of failure when a requested variable isn't found and, secondly, the values of ENV variables are, well ... variable.

The EAL addresses these two issues directly by enforcing a strict configuration lookup and preventing any changes to its state at runtime.

Strict lookup

Ruby's ENV global is a simple Hash that provides no guarantee or expectation of failure when a requested variable is not found. A developer can implement their own checks, but that must be done for each and every case, and many times leads to redundant nil/empty string checks.

Before do
  raise "you must provide a CENTRALAUTH_URL" if ENV["CENTRALAUTH_URL"].to_s.empty?
  raise "you must provide a ..." unless ...
end

If a strict check for the configuration is not enforced, it typically leads to unexpected/obscure failure down the line.

When(/^I navigate to the centralauth wiki$/) do
  @browser.goto ENV["CENTRALAUTH_URL"] # if this is empty, Selenium throws a very strange error
end

The EAL solves both of these problems by enforcing a uniformly strict policy on configuration lookup unless a default value is explicitly provided.

When(/^I navigate to the centralauth wiki$/) do
  @browser.goto env[:centralauth_url] # will always raise a <code>ConfigurationError</code> if empty
end

Immutable state

Mutation of the global ENV object is permitted by Ruby, but it's generally not a good idea as it can lead to race conditions and other unpredictable behavior, especially in evented contexts such as Cucumber's.

To illustrate the complications, consider how we (for security reasons) configure the test-user's password in CI by way of an alternative variable (corresponding to MEDIAWIKI_PASSWORD_VARIABLE).

Before("@login") do
  ENV["MEDIAWIKI_PASSWORD"] = ENV[ENV["MEDIAWIKI_PASSWORD_VARIABLE"]] if ENV["MEDIAWIKI_PASSWORD_VARIABLE"]
  # ...
end

Now imagine this same global variable is referenced somewhere else, say in another Before hook that's defined somewhere in the project's test-suite.

Before("@login") do
  visit(HomePage).login_with(ENV["MEDIAWIKI_USER"], ENV["MEDIAWIKI_PASSWORD"])
end

It's entirely possible that, depending on how the various support files were imported, this second hook would end up executing before the first, an example of a simple race condition.

The EAL prevents these conditions by providing a one-way, read-only path for environment configuration; configuration is sourced at the beginning of each run, may be read by step definitions, but is never defined or redefined at runtime.

Before("@login") do
  env[:mediawiki_password] = ... # would raise a NoMethodError
end

Configuration defaults

So the EAL's configuration can be considered a strict contract, but who is the authority on its correctness? We could leave it up the environment's test runner to define all the necessary configuration, but then we risk incorrect assumptions or changes in the environment affecting the outcome of our tests—this is exactly the kind of indeterministic behavior the EAL is meant to reduce.

If we consider environmental configuration to be as important to test determinism as the resources created in "given" context by calls to the API or by simulating UI interaction, we should probably allow the test suite to dictate at least a correct default configuration for different environments.

Via cucumber profiles

One way to to provide defaults for environment variables would be through Cucumber profiles. They are typically used to specify default cucumber parameters, but can also specify environment variables. Leveraging this feature, we could allow test suites to define their own defaults for each environment that's expected to run it.

---
default: --profile mw-vagrant
mw-vagrant-host: >-
  MEDIAWIKI_URL=http://127.0.0.1:8080/wiki/
  MEDIAWIKI_URL_ALTERNATE=http://centralauthtest.wiki.local.wmftest.net:8080/wiki/
  MEDIAWIKI_URL_LOGIN=http://login.wiki.local.wmftest.net:8080/wiki/
  MEDIAWIKI_USER=Selenium_user
  MEDIAWIKI_PASSWORD=vagrant
mw-vagrant-guest: >-
  MEDIAWIKI_URL=http://127.0.0.1/wiki/
  MEDIAWIKI_URL_ALTERNATE=http://centralauthtest.wiki.local.wmftest.net/wiki/
  MEDIAWIKI_URL_LOGIN=http://login.wiki.local.wmftest.net/wiki/
  MEDIAWIKI_USER=Selenium_user
  MEDIAWIKI_PASSWORD=vagrant
integration-beta: >-
  MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/
  # ...
integration-test2: >-
  MEDIAWIKI_URL=http://test2.wikipedia.org/wiki/
  # ...

Via a native feature

The approach using Cucumber profiles may have some downsides, however. For one, there's no way to specify a different set of defaults for different environments; in non-default environments, the user would have to explicitly provide a profile on the command line. Secondly, there's no way to allow overwriting of the variable values; profiles don't support any kind of variable substitution like MEDIAWIKI_URL=${MEDIAWIKI_URL:-...}.

A native approach might provide more flexibility. Change Ifb358ec20 provides a mechanism by which you can define these environmental defaults in a file named environments.yml within browser-test directory. It's similar to the Cucumber profiles configuration in that it's a YAML hash where each key corresponds to a environment name.

mw-vagrant-host:
  mediawiki_url: http://127.0.0.1:8080/wiki/
  mediawiki_url_alternate: http://centralauthtest.wiki.local.wmftest.net:8080/wiki/
  mediawiki_url_login: http://login.wiki.local.wmftest.net:8080/wiki/
  mediawiki_user: Selenium_user
  mediawiki_password: vagrant
mw-vagrant-guest:
  mediawiki_url: http://127.0.0.1/wiki/
  mediawiki_url_alternate: http://centralauthtest.wiki.local.wmftest.net/wiki/
  mediawiki_url_login: http://login.wiki.local.wmftest.net/wiki/
  mediawiki_user: Selenium_user
  mediawiki_password: vagrant
integration-beta:
  mediawiki_url: http://en.wikipedia.beta.wmflabs.org/wiki/
  # ...
integration-test2:
  mediawiki_url: http://test2.wikipedia.org/wiki/
  # ...

The set of defaults that should be used is specified as a single MEDIAWIKI_ENVIRONMENT variable.

Some of the key (seemingly advantageous) differences here are: 1) the format of environments.yml is a bit more congruent with how variables are referenced in the step implementation; 2) the values can be overwritten by variables defined in the shell; and 3) there's no extra invocation option necessary as long as the user has MEDIAWIKI_ENVIRONMENT correctly set in the shell.

Rich environment resources

Beyond just codifying the way that environment configuration is defined and read, the EAL provides additional constructs around common MW resources so that more expressive and readable test implementation can be written.

It allows for steps like the following:

Given(/^I am logged in to the centralauth domain$/) do
  @browser.goto ENV["MEDIAWIKI_CENTRALAUTH_LOGINWIKI_URL"]
  visit(LoginPage).login_with(ENV["MEDIAWIKI_USER"], ENV["MEDIAWIKI_PASSWORD"])
end

To be rewritten as:

Given(/^I am logged in to the centralauth domain$/) do
  on_wiki(:centralauth) { visit(LoginPage).login_with(user, password) }
end

A whole new MediaWiki world

Every Cucumber step implementation runs in the context of a World instance which is created anew for each scenario. (In Ruby speak, the step-definition block is evaluated with self as the Cucumber World object, probably using instance_exec.) This construct of a "world" object fits nicely with the idea of an "environment" interface and, fortunately, Cucumber allows us to customize the world object used.

World { MediawikiSelenium::Environment.new(ENV) }

When(/.../) do
  self.class # => MediawikiSelenium::Environment
end

Helper methods

By setting up our own world object, we're not only neatly encapsulating our configuration at the beginning of each scenario, we're also making available all the instance methods of the Environment class. This allows us to provide some nice constructs that make for more readable and flexible tests. The most important of these helper methods are:

user

Current wiki user name

user_label

Current wiki user name (with underscores replaced)

password

Current wiki user's password (taking into account MEDIAWIKI_PASSWORD_VARIABLE)

wiki_url

Current wiki URL

on_wiki(id) { ... }

Perform actions on another wiki

as_user(id) { ... }

Perform actions as another user

in_browser(id, options = {}) { ... }

Perform actions in an isolated (and optionally customized) browser session

browser

Current browser instance (started on demand)

browser_factory

Access to the browser factory for setting up custom settings that last over the duration of the scenario

Switching between resource alternatives

As you can see, some of the helper methods allow for performing actions on "other" wikis or as "other" users. Indeed, the EAL has a construct for switching between "alternative" configurations. This serves test cases where one has to simulate interactions between multiple users, across multiple wiki, etc. (The Echo and Flow extensions require these kinds of scenarios, for example.)

At first this may seem to contradict the idea that configuration should be static from start to finish, but the EAL actually accomplishes the temporary re-configuration in a very isolated way: by cloning the Environment object, overwriting its config with the alternative values, and evaluating the given block within the new object's context. In other words, the new environment only affects the scope of the block and the changes don't amount to a change in global state that would threaten us with race conditions and other unpredictable behavior—this is actually a fairly common pattern used in functional programming.

When(/.../) do
  self # => #<Environment:1>

  with_alternative(:mediawiki_url, :b) do |url|
    self # => #<Environment:2>
  end
end

Alternative configuration values are looked up using the base name (e.g. :mediawiki_url) appended with the alternative ID (e.g. :b).

For example, given the following configuration:

MEDIAWIKI_URL=http://an.example/wiki/
MEDIAWIKI_URL_B=http://another.example/wiki/

When the following step is evaluated:

Given(/^I am logged in to wiki B$/) do
  with_alternative(:mediawiki_url, :b) do |url|
    env[:mediawiki_url]
    # ...
  end
end

I expect url and env[:mediawiki_url] within the block to be "http://another.example/wiki/".

Note that while the above example illustrates how alternative configuration is resolved and overwritten, a more realistic scenario involving two wikis would probably use on_wiki (which shares implementation with with_alternative).

Given(/^I am logged in to wiki B$/) do
  on_wiki(:b) do |url, api_url|
    # ...
  end
end

Reducing coupling with names over values

One of the aims of the EAL is to reduce environment coupling, like in this scenario:

Given I am logged in as "Selenium_user"
When I click on "Preferences" in the menu
Then I see my preferences page

Scenario text (and user stories in general) is most useful as acceptance criteria when it includes only what is relevant to the feature's requirements. In other words, is it important in this case we're logged in as "Selenium_user"? No, not really. Is it important that we're clicking the "Preferences" link? Yes. Therefore, we'd probably want to refactor this scenario to read.

Given I am logged in
When I click on "Preferences" in the menu
Then I see my preferences page

Without the EAL, you would probably implement the first step using the value for the MEDIAWIKI_USER environment variable (directly using ENV), which is simple enough. However, without the EAL, this sort of refactoring doesn't work in cases where we're describing more than one of the same kind of resource, for example the interaction between two users.

Given I am logged in as "Selenium_user"
  And "Selenium_user2" has mentioned me

In order to reduce coupling in cases like this, one can make use of the EAL's named alternatives. The above example could be refactored as the following scenario text and step definitions.

Given I am logged in as user A
  And user B has mentioned me

Given(/^I am logged in as user A$/) do
  as_user(:a) { |user, password| visit(LoginPage).login_with(user, password) }
end

Given(/^user B has mentioned me$/) do
  as_user(:b) { api.create_page("Talk:Page", "Hello, [[User:#{user(:a)}]]") }
end

As you can see, the actual name of the user is factored out which reduces coupling and, furthermore, the logic between scenario steps and implementation remains clear due to the use of descriptive variables.

Isolated browser sessions

MediaWiki developers of user-to-user features like Echo and Flow often need to implement scenarios that simulate multiple user sessions, which can be problematic for a number of reasons. Most notably, logging out (before logging back in as someone else) destroys all sessions of the current user, including those of concurrently running tests.

The EAL attempts to solve issues like these by providing an easy way to instantiate multiple browsers within the context of a single scenario.

Given I am logged in
  And have created a new flow topic
  And user B has responded to it
When I revisit the wiki
Then I see a new notification
  ...

As noted previously, whenever possible, you typically want to provision the initial state described by the Given clauses using the API. Assuming that's not an option here—ostensibly, Flow topics cannot be created via the API, but if that's not actually the case, just pretend it is for the sake of this example—we'd have to resort to driving the UI to create the initial topic; we'd want to be logged in as the default user to create the initial topic, then as user "B" to respond to it, then again as the default user to perform the When and measure the Then.

This would be quite cumbersome in the current framework, requiring some sort of tag, a before hook, and an after hook to close out the extra browser session. Use of page objects also becomes problematic: they expect a @browser instance variable to be set, so you might have to repeatedly overwrite it.

With the EAL's in_browser helper method (in combination with other helpers like as_user), we can easily perform any number of operations within the context of a new session, which is started on demand, cached, and closed automatically at the end of the scenario.

Given(/^I am logged in$/) do
  # log in as primary user ...
end

Given(/^I have created a new flow topic$/) do
  # create the topic ...
end

Given(/^user B has responded to it$/) do
  in_browser(:b) do
    as_user(:b) do |username, password|
      visit(LoginPage).login_with(username, password)
      # respond to topic ...
    end
  end
end

When(/^I revisit the wiki$/) do
  visit(ArticlePage)
end

Then(/^I see a new notification$/) do
  on(ArticlePage) do |page|
    expect(page.notifications).to match("1 notification")
  end
end

In the example above, we've opened up a new browser session, and named it "b", to perform authentication and subsequent actions of user "b". If we wanted to perform additional actions in separate steps using this same browser, we could again reference it using in_browser(:b) { ... }.

Another useful aspect of in_browser is that custom browser settings can be given as the second argument.

When(/^I view the same article with my browser in Spanish$/) do
  in_browser(:spanish, language: "es") do
    visit(ArticlePage)
  end
end

More on the implications of customizing or changing browser settings in browser factories.

Browser factories

In the current framework a browser is instantiated before each scenario using a simple Before hook. While this is adequate for simple use cases that require neither custom browser settings nor multiple sessions, it makes implementing more advanced cases difficult. The EAL tries to address a few of these difficulties.

Customization

An often requested feature is the ability to easily customize browser settings, for example, set a proxy, customize the user agent, use a different default language, etc.

For settings that are supported by the framework (such as the latter two), it's easy enough to set the corresponding environment variables. However, if support isn't there yet, you'd have to first tag the scenario with @custom-browser, then implement your own Before hook that instantiates a Watir::Browser with all the right Selenium options and such, essentially repeating all of the complex configuration logic that the framework already implements. You'd also have to worry about possibly breaking remote browser support altogether.

The EAL attempts to encapsulate most of the rather cryptic Selenium driver setup for you, and provides a simpler interface for supporting new browser configuration across the project, or making arbitrary customizations for the duration of a scenario.

For example, you might want to support some new project-wide optional configuration that specifies a proxy to use for any browser that's started during your tests—a browser proxy, not a Selenium proxy, though the latter is also possible. It would look something like the following.

Before do
  browser_factory.bind(:http_proxy) do |proxy, options|
    options[:desired_capabilities].proxy = { http: proxy }
  end
end

First we access the factory for the currently configured browser via browser_factory—if we only wanted to implement support for Firefox, we'd specify it as an argument, i.e. browser_factory(:firefox).

Next, we call bind(:http_proxy) to "bind" any given value for the new :http_proxy variable using the given block.

The block is passed the value of :browser_proxy as it's configured for the current environment, and is expected to modify the second options parameter according to its needs. If there's no proxy configured, the block never executes. See the documentation for MediawikiSelenium::BrowserFactory::Base and the browser-specific subclass for the initial value of options.

In this example, we set the proxy attribute of Selenium::WebDriver::Remote::Capabilities. So, yes, you still need to know a little about Selenium's options, but it still simplifies things quite a bit.

Note that you don't have to specify an environment variable name at all. You can make arbitrary project-wide customizations via bind as well, simply by omitting the arguments.

Before("@spanish") do
  browser_factory(:firefox).bind do |options|
    options[:profile]["intl.accept_languages"] = "es"
  end
end

Instantiation and management

Another useful feature of the EAL is that browsers are started on-demand (lazily), upon the first call to browser—which is typically done indirectly via page objects, etc. This allows you to perform operations in steps, and test those steps, without having to wait for the browser to start up and close each time.

Given(/^article "(.+)" exists$/) do |title|
  api.create_page(title, ...)
  # no browser has been started yet
end

Given(/^it contains "(.+)"$/) do |text|
  api.edit(...)
  # still no browser
end

When(/^I visit the wiki$/) do
  visit(ArticlePage) # a browser is born
end

This lazy instantiation also has important benefits when it comes to customization. You can safely implement browser customization in Before hooks, or even in Given clauses, without having to start up a new browser session. (Warning, contrived example below ...)

Given(/^I am using proxy "(.+)"$/) do |proxy|
  # browser hasn't started yet, so we can continue to do customizations
  browser_factory.bind do |options|
    options[:desired_capabilities].proxy = { http: proxy }
  end
end

Every browser that's started (directly or indirectly) via browser is cached by the current factory, uniquely indexed by its configuration, and automatically closed the end of each scenario.

Caveat

One important side effect to caching according to configuration is that changes to any of the "bound" variables will result in a new browser being opened. This caveat is especially important to consider when passing the second overrides argument to in_browser.

When(/^I view the same article with by browser preferring Spanish$/) do
  in_browser(:b, language: "es") { ... }
end

Then(/^I see options to view the Spanish version$/) do
  in_browser(:b) { ... }
end

In the above example, browser operations made in the Then clause would actually result in a new browser session, separate from the one used in the When. This is because the effective configuration used to index the browser cache would be different between the two contexts.