{"id":395,"date":"2015-03-05T21:18:54","date_gmt":"2015-03-05T21:18:54","guid":{"rendered":"http:\/\/blog.mozilla.org\/webqa\/?p=395"},"modified":"2015-03-05T21:18:54","modified_gmt":"2015-03-05T21:18:54","slug":"web-qa-publishes-automated-test-results","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/fxtesteng\/2015\/03\/05\/web-qa-publishes-automated-test-results\/","title":{"rendered":"Web QA publishes automated test results"},"content":{"rendered":"

I am delighted to announce that Web QA has migrated to a new continuous integration server instance that is publicly accessible<\/a>! This means you can see all of our test results without needing to be on the private network or using our VPN! This is a massive step forward for the team as for many years our results have been hidden, causing a barrier for getting our awesome community to help us with managing failures.<\/p>\n

We’re using Jenkins CI<\/a> for our continuous testing, and further details can be found on our wiki page<\/a>. Head on over to https:\/\/webqa-ci.mozilla.com\/<\/a> to have a look at the results for our various projects. You should see something similar to the following screenshot (click for larger version).<\/p>\n

\"Web<\/a>

Main Jenkins Dashboard<\/p><\/div>\n

Jenkins consists of job and builds. On the main dashboard you’ll see all jobs listed, however you can reduce these to a subset by clicking on a project tab near the top of the page. Each project has several jobs to cover a variety of environments and configurations. If you click on a job name you’ll be taken to the job page (see below).<\/p>\n

\"Web<\/a>

Job Page<\/p><\/div>\n

From here you can see the project owner(s), build history, and test result trend. A single test run is referred to as a build. Note that there’s also a link to the project’s GitHub page – handy for raising issues for new test failures. Each of the items in the build history items is a link through to the build results page (below).<\/p>\n

\"Web<\/a>

Build Page<\/p><\/div>\n

The build page shows a summary of the test failures, but mostly serves as the entry point for the various methods of investigating test failures. The first is the HTML report:<\/p>\n

\"Web<\/a>

Custom HTML report<\/p><\/div>\n

This is a custom report generated by our py.test plugin<\/a> that is often the first place to go when investigating failures. It shows the traceback from the console, a screenshot at the point of the failure, as well as the page source and URL. It can also be useful to look at the entire console output:<\/p>\n

\"Console<\/a>

Console Output<\/p><\/div>\n

Finally, the test result page allows you to view the failures alongside their stability. If a test has failed all of the recent test runs it means it’s likely to be easily reproduced, however tests that fail intermittently can take a little more digging:<\/p>\n

\"Test<\/a>

Test Results<\/p><\/div>\n

I hope you’ve enjoyed this little introduction to Web QA’s automated test results. If you’re interested in helping the team please feel welcome to look into the current or intermittent failures. I would recommend contacting the project owner(s) before doing making any changes to the tests though in case the failure is already being worked on.<\/p>\n

Also, please consider joining our mailing list<\/a> or find us on IRC in #mozwebqa on irc.mozilla.org<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

I am delighted to announce that Web QA has migrated to a new continuous integration server instance that is publicly accessible! This means you can see all of our test … Read more<\/a><\/p>\n","protected":false},"author":513,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[228],"tags":[228,31343,604,31342,101],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts\/395"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/users\/513"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/comments?post=395"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts\/395\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/media?parent=395"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/categories?post=395"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/tags?post=395"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}