Selenium Conference 2015 Wrap-up Notes

This year, I set out to attend Selenium Conference 2015 with some specific goals:

1. See which JavaScript framework(s)/librar(y|ies) are being used and recommended for “full-stack” testing (as a potential complement or replacement to our Python stack).  Check out an earlier post on InternJS for our goals and challenges.
2. See which mobile-testing strategies + tooling are being used.
3. Look into what folks are using Docker for, and how we might apply it.
4. Cloud, cloud, cloud (and, more specifically, for us) + AWS – what are companies and organizations of all sizes and needs using it for, testing-wise, and what are the considerations and gotchas, etc., as well as how testing considerations change in the DevOps model that often accompanies cloud.
5. Which open-source tools can we use + hopefully integrate, to get performance + accessibility testing, along with our functional tests?  (Particularly, as close to Real User Monitoring as we can get, without a third-party, expensive, hosted solution – or in addition.)
6. Catch up on what the core committers of the Selenium project are doing, thinking, and where the project is next headed.
7. General information-seeking + evangelizing, particularly around open-source offerings, both in terms of what we could potentially use, but also sharing what we’ve done, are working on, and can share with the community.

Talks I attended and particularly enjoyed were:

1. Simon Stewart’s keynote, entitled: Selenium: State of the Union – video (no slides available)
2. Curing Imposter Syndrome – video, slides
3. Mobile End-to-End Testing at Scale: Stable, Useful, Easy – Pick Three – video, slides
4. Distributed Automation using Selenium Grid/AWS/Auto-scaling – video, slides
5. Docker Selenium: Getting Started – video, slides

A few key takeaways:

1. Very common among many companies for monitoring and continuous deployment were: statsd, graphite, Jenkins CI, and Travis CI.
2. If not optimized, AWS can be _very_ expensive, and you can very easily end up wasting paid-for computing cycles/resources.
3. Docker looks very useful if you only need to test (using Selenium Grid) on Linux, primarily – it’s less clear with Windows + Mac.  The particular talk on this used Ruby as its language, although it shouldn’t be too much work to do in Python.
4. All the core committers I spoke with, and other industry peers as well, recommended “just using” WebDriverJS if we’re going to continue experimenting with JavaScript-based, developer-integration level testing.  Yahoo! has also used Protractor (for Angular apps) + CasperJS, along with WebDriverJS as its core, with good results.
5. Lots of discussion around mobile testing, and while most of the folks testing mobile do so using Appium (as do we), the needs vary widely, as with good monitoring and responsive design, there isn’t always a need to do full end-to-end tests across all devices.  This also depends on whether you’re testing a mobile website or a native app on Android or iOS, of course.  Appium 1.5 is coming soon, we learned, and will be a nice, clean release with a lot of fixes + features.
6. Yahoo! has used sitespeed.io to drive webpagetest.org; it looks really useful, and thankfully seems kept up-to-date.

Action Items:

1. Catch-up with the remainder of the Selenium Conference 2015 talks I wasn’t able to attend.
2. Look further into the potential use-case(s) for Docker, test-infrastructure wise.
3. Investigate integrating a Selenium proxy (such as browsermob-proxy) with HAR files and potentially Apache/Nginx server logs, to augment our HTML Reports with richer, more timely information to help better-report and debug problems (heavily re-inspired by the Facebook talk!)
4. Look into the possibilities of scaling our tests + increasing performance using AWS/cloud infra, while balancing cost considerations.
5. Look into evaluating and potentially integrating visual-comparison testing tools, such as Applitools Eyes.
6. Evangelize – through blog posts, meetups, etc., more of what Web QA and Mozilla are up to – particularly for our peers and community’s benefit.

For more wrap-up posts on last week’s separate, but follow-on Web QA team meetings, look here, as well: https://blog.mozilla.org/webqa/category/meetings/

Keep an eye on this blog for future posts; we’ll be posting a little more often than we have in recent months!

– Stephen