Web QA publishes automated test results

I am delighted to announce that Web QA has migrated to a new continuous integration server instance that is publicly accessible! This means you can see all of our test results without needing to be on the private network or using our VPN! This is a massive step forward for the team as for many years our results have been hidden, causing a barrier for getting our awesome community to help us with managing failures.

We’re using Jenkins CI for our continuous testing, and further details can be found on our wiki page. Head on over to https://webqa-ci.mozilla.com/ to have a look at the results for our various projects. You should see something similar to the following screenshot (click for larger version).

Main Jenkins Dashboard

Jenkins consists of job and builds. On the main dashboard you’ll see all jobs listed, however you can reduce these to a subset by clicking on a project tab near the top of the page. Each project has several jobs to cover a variety of environments and configurations. If you click on a job name you’ll be taken to the job page (see below).

Job Page

From here you can see the project owner(s), build history, and test result trend. A single test run is referred to as a build. Note that there’s also a link to the project’s GitHub page – handy for raising issues for new test failures. Each of the items in the build history items is a link through to the build results page (below).

Build Page

The build page shows a summary of the test failures, but mostly serves as the entry point for the various methods of investigating test failures. The first is the HTML report:

Custom HTML report

This is a custom report generated by our py.test plugin that is often the first place to go when investigating failures. It shows the traceback from the console, a screenshot at the point of the failure, as well as the page source and URL. It can also be useful to look at the entire console output:

Console Output

Finally, the test result page allows you to view the failures alongside their stability. If a test has failed all of the recent test runs it means it’s likely to be easily reproduced, however tests that fail intermittently can take a little more digging:

Test Results

I hope you’ve enjoyed this little introduction to Web QA’s automated test results. If you’re interested in helping the team please feel welcome to look into the current or intermittent failures. I would recommend contacting the project owner(s) before doing making any changes to the tests though in case the failure is already being worked on.

Also, please consider joining our mailing list or find us on IRC in #mozwebqa on irc.mozilla.org.