Mobile/QA Test Audit March 2014

The purpose of this page is to audit MobileFrontend's browser tests for all features that are currently in stable and check what needs additional coverage.

The page lists attempts to list all actions a user can make on the stable version of the MobileFrontend site in stable. If a test is covered the action is in bold. Actions which are covered but indirectly as a consequence of another test are marked in italic.

This document highlights that various things could use tests and that the tests could be improved with better wording and organisation.

Article page features

• This page has issues button opens a list of issues
• Clicking browser back button closes the this page has issues overlay
• Clicking close button closes the this page has issues overlay
• Editing
  • Clicking edit button when anonymous shows CTA [covered in header.feature]
  • Clicking edit when logged in opens the wikitext editor. [covered in wikitext_editor.feature]
  • Clicking close button closes the wikitext editor [covered in wikitext_editor.feature]
  • Clicking browser back button closes the wikitext editor [covered in wikitext_editor.feature]
  • Clicking back button from preview takes you back to editor
  • Hitting submit makes an edit to the lead section [Covered indirectly by the diff.feature]
    • and editing reloads the page with my change. [Covered indirectly by the diff.feature]
  • I can edit a section which is not the lead section
• Uploading
  • Upload button is disabled on pages which do not need a lead photo [covered indirectly by uploads_lead_tutorial.feature as we have a test to check it is enabled]
  • I see a tutorial when I am a new user [covered in uploads_lead_tutorial.feature]
    • and when the tutorial finishes I can upload an image
  • I can upload an image to a page that needs a lead photo.
    • Trying to upload a lead image to a page which doesn't need one results in error message [covered in badly named header_logged_in.feature]
    • Clicking upload when anonymous shows CTA [covered in header.feature]
• Watching
  • Clicking watch star when anonymous shows CTA [covered in header.feature]
  • Clicking watch star adds page to watchlist [covered by watchlist.feature "Add an article to the watchlist" - however this test could be better - it doesn't actually check the item appears on Special:Watchlist and only applies to pages in the main namespace]
  • Clicking a watch star on a watched page removes page from watchlist [covered by watchlist.feature but see notes above]
• Clicking language button shows list of available languages [covered twice by language.feature and language_logged_in.feature]
  • • ...and I can see variants where available
    • and I can close the language overlay [covered in language.feature]
• Searching
  • Searching shows me results [covered in search.feature but relies on Barack Obama]
  • I can do a full text search
  • I can watch articles from search
  • And I can exit the search via the browser back button [covered in search.feature]
    • ... or the close button [covered in search.feature]
• Left nav
  • I am able to open the left navigation menu [covered indirectly by watchlist.feature as you have to open it to get to the link]
  • When I click on a link that requires me to be logged in I am taken to the login page with a redirect to the feature [covered in watchlist.feature]
  • I am able to close the left navigation menu
• Notifications
  • When I click the notification icon I see my notifications [covered in notifications.feature]
  • ... and I can close the notification overlay using back button [covered in notifications.feature]
    • ... or using the close button [covered in notifications.feature]
• Switching to desktop
• Clicking a reference shows the reference overlay.
• Toggling a section
  • I can toggle open a section [Covered indirectly by external_links.feature]
  • I can close a section

Special pages

• Nearby page in vector skin
  • When I give my location I see articles near me
• Nearby page in mobile skin
  • When I give my location I see articles near me [covered in nearby.feature]
  • I can refresh the list of results
• Clicking random button takes me to a random page [covered in a strange way by random_ua.feature]
• Watchlist page
  • I can switch to modified view
  • I can switch to list view
  • Clicking the more link shows me more pages on my watchlist
  • I can apply a namespace filter on modified view to limit feed to a certain namespace e.g. talk
• Page shows latest diff [Covered by diff.feature]
• Diff page renders correctly [Covered by diff.feature]
  • I can thank the user for their contribution
• Uploads page
  • When anon takes me to login page [covered by login_required_uploads.feature]
  • As a new user when I click contribute image I see a tutorial [covered in uploads_page_tutorial.feature]
  • When I scroll to the bottom of the page new older images load (Infinite scroll)
  • I can upload an image to the uploads page [covered in uploads_page.feature]
  • When I fail an upload I see an error message. [covered in I56ac12dff9566496dadcde8470022a41ce2ebc77 which has not been merged yet]
• Settings page
  • I can opt into beta [indirectly covered in tests for beta/alpha features - we may want to make this more explicit]
  • I can opt into alpha [indirectly covered in tests for beta/alpha features - we may want to make this more explicit]
  • When I turn on disable images, images are disabled.
• Login
  • I see message when I am already logged in [covered by login.feature]
  • Errors on logging in:
    • Bad username / password combination results in error message
• Account creation
  • I can refresh captcha [covered by "Can refresh captcha" in create_account_failure_messages.feature]
  • I am able to create an account [covered indirectly by keepgoing.feature]
  • I get an error when I have an issue creating an account
    • Invalid captcha [Covered in create_account_failure_messages.feature]
    • password mismatch [covered in create_account_failure_messages.feature]
    • I try to register with an existing username
• I am able to logout

Hygiene

create_article.feature

... doesn't seem to test anything. o_O

nearby.feature

Only works on Firefox

toggling

Tests for toggling although they exist should be more explicit. Maybe repurpose external_links ?

watchlist.feature

"Add an article to the watchlist" and "Remove an article from the watchlist" use the random feature to find a page to watch. This will not always pass - as the tests run more there will be less pages that are unwatched

The statement "I select Watchlist" actually means "I click the watchstar on the homepage"

The tests "Add an article to the watchlist" and "Remove an article from the watchlist" cannot be run independently. The second test tears down the first. It would be good if this was more explicit.

pages/home_page.rb and pages/random_page.rb

Big mess which duplicates functionality/steps in article_page - should inherit from ArticlePage. random_page should probably be deleted in favour of article_page since they are essentially the same.

login_required_watchlist.feature

Given statement seems a bit too much :) - Given that I am using Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

header.feature

Badly named should probably rename to anonctas.feature or even better split into watchstar, edit and upload features.

header_logged_in.feature

Badly named. Scenario "Uploading Image" is confusing. It actually checks you cannot upload lead photos to the home page.

random_ua.feature

The tests are written strangely and use user agents for unknown purposes. It seems to test that after clicking a random page that UI elements such as watchlist and links in the left navigation are present. Very weird.

external_links.feature

It's not clear why we would want to check that an external link goes to the right place. This also creates an unnecessary dependency on having a Barack Obama page with a specific link.

footer.feature

Not clear to me why we want to check that links in the footer resolve. Checking for the existence of these links seems like a better test?

image_resolves.feature

Again not clear why we need to check that clicking an image takes you to it's file page. Also adds this dependency on the Barack Obama page which seems unnecessary.

language_logged_in.feature

Doesn't seem to add any value on top of language.feature - should probably be collapsed into language.feature

menu_opens_page.feature

Again .. why we need to check that these URL's all resolve to the correct place? Seems redundant. Also the tests aren't really true "my URL should be set to the Watchlist Page" for example resolves to true when you are on the login page with a redirect set to the watchlist page.