Wikimedia Release Engineering Team/20180320 Selenium Retrospective
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
Retrospective: Selenium Ruby to NodeJS project
Attendees: Jon Robson, Željko Filipin, Jean-Rene Branaa, Erik Bernhardson, Elena Tonkovidova, Timo Tijhof, Jan Drewniak, Joaquin Oltra Hernandez
Agenda
[edit]- Brief overview of project
- Purpose/Objectives
- Zeljko: Were using old Ruby framework from the onset. Worked for some, not for others. Framework was perhaps overly complicated. Additional abstraction with Cucumber turned out not to be needed by most.
Roundtable
[edit]What went well
[edit]- Timo: Worked from master from the start and happened on every commit. Works locally and doesn't make assumptions.
- Jon: some repos were doing this: MobileFrontend and Minerva
- Erik: Also because phantomJS is a giant pain.
- Erik: cirrus was also doing per-commit, although external from Jenkins.
- Timo: industry was moving forward faster than we could.
- Jan: Headless browser support also positive.
- Zeljko: SauceLabs was a nice feature for Ruby, providing Windows/IE support, but in the end was also causing daily failures, and the Windows support wasn't as important (most issues were found in Firefox and/or Chrome as well).
- Jon: Communication was pretty good during the switchover. It was clear what was happening and when.
- Elena: +1 to Jon. Zeljko is super helpful with any questions and all explanations of what and how works
- Timo: Feels like the new framework is easier to use. The new framework allow you to use both BDD and TDD. It feels more naturual to JS developers.
- Elena: Any experience with Rspec?
- Timo: I've not, but it is very similar to Mocha.
- Zeljko: Stuck with simple builtin assertion library to keep things simple.
- Erik: we ended up using chai.js assertions library, probably similar to mocha.
- Timo: QUnit library
- Erik: Not sure why we used Chai vs Mocha, but it may have been the error messages.
- Elena: Chai assertions is more sophisticated.
What could have gone better
[edit]- Jon: Strategically it made totally sense to switch from Ruby to Node, but our team has failed to port them over and is stuck to see an easy way to do so. I wrote up my experiences here > https://www.mediawiki.org/wiki/User:Jdlrobson/Experiences_porting_Ruby_browser_tests_to_NodeJs
- Jon: Team has felt an additional stress now that Ruby is not supported and we're dependent on it. 13+ tests in MobileFrontend, 75+ in Minerva.
- Jon: We've stopped writing tests :/ Jon: Webdriver has a bit of a learning curve even for those who know JS (sync nature is not obvious)
- Erik: There was a lot of effort involved in the porting of tests. Hopefully we'll see some benefit over time.
- Zeljko: Ruby as a language was perhaps a barrier to entry (learning new language, in addition to learning Selenium and the PageObject pattern)
- Zeljko: Seemed like some teams waited until last warning to migrate.
- Elena: When migrating the tests did you (Erik) review the tests themselves or just migrate them?
- Erik: just migrate. Pretty confident in our tests as they stood.
- Jon: Not everyone had a plan for the port. As a result new test development stopped. Not sure if Ruby and NodeJS could coexist. Extensions and libraries that they depend on in Ruby aren't necessarily available on NodeJS.
- Zeljko: There's no technical reason that he can think of that you couldn't port tests one at a time. So they should be able to coexist.
- Jon: Also motivation for migrating is low as the Ruby tests are working.
- Zeljko: willing to pair and help migrate. However, wants to have each team responsible for their own migration in order to insure teams are comfortable with maintaining them moving forward.
- Erik: If using Cucumber, they were able to migrate one feature file at a time.
- Jon: Didn't know that was an option (Cucumber) as he thought we are abandoning Cucumber.
- Zeljko: that was the plan. Level of abstraction didn't seem worth it for developers.
- Elena: Even with an understanding of the code, Cucumber is still valuable for communication/overview.
- Zeljko: Are there things that are missing in the NodeJS framework that I've not already captured?
- Jon: Multi-user logins (as user a do x as user b do y).
- Zeljko: Dan did a lot of work with the helper files, it wasn't clear how much was there and who was using what. As a result not everything was ported. It was developed over 5 years.
- Jon: It might be too much to ask to complete the transition over a few months (maybe need to let the NodeJS evolve over time too).
- Jan: Adding more abstraction capabilities as Jon mentioned.
Tips
[edit]- Erik: Built-in assert() wasn't providing useful error messages. Chai provided these. Mocha would work as well.
- Joaquin: https://github.com/power-assert-js/power-assert
- Timo: Built-in assert() is indeed quite basic. It does have additional methods with better messages though - https://nodejs.org/api/assert.html , but nothing as sophisticated as Mocha or Chai indeed. There's also https://www.npmjs.com/package/assert-diff
Actionables
[edit]- Zeljko: Current activities (selenium improvements) - https://phabricator.wikimedia.org/T182421
- Zeljko: needs help with code reviews and such to wrap up project in next couple of days.
- Timo: Willing to help with code reviews.
- Joaquin: Sharing experiences on a regular basis would be helpful.
- Erik: Willing to show/review how cirrus integrated cucumber/node
- JR/Zeljko: Summary, actionables, writeup by EOW
- JR: Reduce future retrospectives to 60 min.
- JR/Zeljko: next update/sync-up meeting 50min, then down to 20min.
Chat Transcription
[edit]Jon Robson 8:09 AM we did but only for some repos
Erik Bernhardson 8:10 AM also because phantomjs is a giant pain
Jon Robson 8:10 AM we are using Ruby on MobileFrontend and Minerva per commit
Erik Bernhardson 8:11 AM cirrus was similarly doing ruby per-commit, although external from jenkins
Jon Robson 8:17 AM What is the format of this meeting Jean-Rene ? Sounds like we're going over things that went well?
Jean-Rene Branaa 8:17 AM yup
Timo Tijhof 8:17 AM https://etherpad.wikimedia.org/p/20180320-SeleniumRetrospective
Jon Robson 8:21 AM i can expand on my points yeh sorry for dumping in etherpad just not sure how and when to jump in
Elena Tonkovidova 8:24 AM Jan: makes sense
Jon Robson 8:29 AM i guess also the motivation is low for us - the Ruby tests are working and finding bugs 1) I think setting up those repos with Node and Ruby browser tests in parallel would be a good starting point to incrementally porting tests 2) I'd love to get some insight into experiences using cucumber and just porting the step definitions. Particularly around how you got round the Selenium wikimedia library (e.g. log_in is a method in Ruby - I think that's a lot trickier in Node.js?)
Elena Tonkovidova 8:36 AM we should have Tips on etherpad, so the above can be added
Jean-Rene Branaa 8:36 AM I'll capture the chat at the end and put it in the meeting notes
Jon Robson 8:36 AM Thanks JR!
Jean-Rene Branaa 8:36 AM NP
Elena Tonkovidova 8:38 AM Jon: re: 2) - it depends on how Cucumber files are actually structured. Sometimes they are so elaborate that porting them to not-so-familiar framework is quite cumbersome
Erik Bernhardson 8:42 AM worth noting cirrus tests are more "integration" than browser, primarily testing api and end-to-end integration
Elena Tonkovidova 8:45 AM Erik:exactly. I looked at your tests - unfortunately, historically our tests are elaborate and involved user scenarios Erik Bernhardson
8:48 AM we ended up using chai.js assertions library, probabily similar to mocha
Elena Tonkovidova 8:51 AM chai assertion is more sophisticated than mocha assertions
Joaquin Oltra Hernandez 8:53 AM this could be useful https://github.com/power-assert-js/power-assert improves node's assert messages
Željko Filipin 8:53 AM https://phabricator.wikimedia.org/T182421
Joaquin Oltra Hernandez 9:02 AM testing support group
Jean-Rene Branaa 9:02 AM yup
Elena Tonkovidova 9:02 AM sounds great! SIG
Jon Robson 9:07 AM Write up experiences!
Joaquin Oltra Hernandez 9:07 AM quarterly sounds good
Elena Tonkovidova 9:07 AM Successes and failures
Joaquin Oltra Hernandez 9:08 AM etherpad died it's like we never met
Jean-Rene Branaa 9:09 AM lol
Joaquin Oltra Hernandez 9:16 AM 1h makes more sense IMO
Timo Tijhof 9:16 AM the etherpad has returned