Selenium Conference, Austin 2017 Wrap-up Notes

We went to Selenium Conference (Austin, Texas, 2017) looking for a few key things:

1. The latest news and perspectives from the Selenium project and its core committers, particularly around formalizing the WebDriver specification
2. How are teams integrating Selenium tests into their CI/CD workflows?
3. Has the nature of/use-cases for Selenium tests changed?  Are teams writing more, or fewer?  What are the considerations?
4. The latest in automated security, performance, and accessibility testing.
5. What are the major trends among our peers – where are they going, or hoping to go with, with testing?
6. And finally, though not least of all: what are the experiences and challenges in testing with Firefox, in the geckodriver-based, Marionette-backed WebDriver world?

Major themes throughout the conference:

1. Quality belongs to all
2. Test establishes baseline for tooling/apps/practices – we are responsible for advising others on best practices
3. Security
4. Diversity

Here are a few of our highlights and takeaways:

Future of Headless-Browser Testing

• PhantomJS and other headless-browser testing frameworks were brought up, with the question of “who here uses a headless browser to browser the Web?” directed at the audience.  Instead of using a third-party framework, as some do now, browser-vendor supplied testing modes should be used, once available.  Bug 1338004 tracks Firefox’s headless-browsing mode.

Team and Testing Culture

• in many of the talks we attended, and in conversations we had, QA/testing teams and individuals were no longer structured, organizationally-wise, or at least considered as, “separate.”  Many testers were either directly integrated into their project teams, or, commonly as well, in “advisory” roles: bringing their battle-earned test-coverage/approach recommendations, and adapting them with and for their team(s).
• likewise, test teams are moving much closer to controlling, or at least being partly responsible for, and having input on, the test environment, and development workflows.

Tooling

• many teams are using Jenkins for their main continuous-integration server, as part of, typically a continuous deployment workflow; few, however, have yet migrated to or even looked at Jenkins Pipeline.  Those which have, have found it useful, but have had to file bugs and use workarounds (as have we), to have a working setup.
• also quite a few Travis CI users, as well, with the use-cases being largely similar to ours: running linters, subsets of test-suites, and/or specialized, pre-build testing.
• somewhat surprisingly, adoption of (and tooling for) accessibility, performance, and security testing still seems relegated to a few automated tools.  Unsurprisingly, though, these are nuanced areas which are either better (or at least primarily) served by focused teams, and/or difficult to reliably test in an automated manner.

“Best Practices”

• “less (fewer Selenium tests) is more:” Selenium is a great tool, but use it appropriately.  Find ways and tools (unit tests, monitoring, etc.) to augment your testing strategy, so it’s faster, more reliable, and less maintenance.
• config as code
  • test your tests, and their test infrastructure
  • goal: repeatable, disposable, and scalable test+deploy infrastructure
• (prepare to) do more with your test data
  • even if you don’t think you need, or can use, now, your test data, if you can publish and store test artifacts, you might find a use for them in the near future
  • Dave Hunt has started exploring this with ActiveData and Treeherder, using our team’s projects
• directly from core committers, and a recommendation of ours since at least 2012: use explicit waits as often as possible (at least over implicit waits).  Many agreed they would remove implicit waits from the codebase/support, if it weren’t for the fact that many still have strategies, and appear to heavily rely on them.

Geckodriver and Marionette

• in his Selenium: State of the Union keynote, Simon covered WebDriver’s new status as a Candidate Recommendation, acknowledging some of the community’s frustrations at the “speed at which Geckodriver has been moving” and asking if anyone in the audience had been “bitten by Firefox.”  However, after catching the audience up briefly on Marionette and the lead-up to what is now the “probably the first spec compliant WebDriver implementation,” calling testing with the specific combination of the latest Selenium + Geckodriver + Firefox nightly “a pretty sweet combination.”
• outside of the presenters’ talks, we fielded a few comments and questions about Geckodriver, but with rare exception, most folks hadn’t-yet tried the above combination, or had primarily hit issues with Firefox 47.0 (fixed in 47.0.1), or had tried using Firefox 48+ without Geckodriver.

Core Committers’ Panel, and Selenium’s Future

• one of the key topics in the Closing Keynote: Q&A With the Selenium Committers panel was their hope that once WebDriver’s specification has been finalized, browser vendors take the brunt of development and maintenance, helping the project move faster, yet be stable and continue to add features testers need.

Final Thoughts

Rebecca Billings and Stephen Donner both really enjoyed Selenium Conference 2017 in Austin. It was well organized and offered a lot of interesting talks to attend. Not to mention that it was well attended by many industry peers, which provided valuable networking. We both look forward to attending a future event.