{"id":416,"date":"2014-03-04T21:39:52","date_gmt":"2014-03-04T21:39:52","guid":{"rendered":"http:\/\/blog.mozilla.org\/webqa\/?p=416"},"modified":"2014-03-04T21:39:52","modified_gmt":"2014-03-04T21:39:52","slug":"automated-testing-analyzing-results","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/fxtesteng\/2014\/03\/04\/automated-testing-analyzing-results\/","title":{"rendered":"Automated testing: Analyzing results"},"content":{"rendered":"
\n

The step that comes after Step 6 in Running WebQA Automated tests<\/a> is Analyzing Results! The last article was all about how to get set up for automation, and getting the tests running. This post is all about what to do with the information you get from the results.<\/p>\n

Next Steps<\/h2>\n

Step 1: What are your results?<\/h3>\n

If all of the tests passed, then you can move on to running more tests. Congratulations!<\/p>\n

If you have failures or other test results, read on.<\/p>\n

\u00a0Step 2: XFail or xPass<\/h3>\n

Did you get test results that match \u201cxFail\u201d or \u201cxPass\u201d? You may be unfamiliar with those terms. Tests that have been x-Failed, or xPassed, have been manually set to automatically fail or pass. *<\/p>\n

We set tests to xFail or xPass when we want the test to fail or pass on purpose. We would do this if, for example, there is a known bug which hasn\u2019t been fixed yet. If the test failed every day, we would spend time looking into the reason why. Setting it to \u2018xFail\u2019 means that we know the test is not correct, and will not be until the bug has been resolved fixed.<\/p>\n

[* To read more, search on \u2018xFail\u2019, or read articles like this:\u00a0https:\/\/quality.mozilla.org\/docs\/webqa\/managing-web-qa-xfails\/<\/a>]<\/p>\n

You should look at our Xfail Dashboard tool to see the status of test cases:<\/p>\n

http:\/\/mozilla.github.io\/mozwebqa-dashboard\/#\/xfails<\/a><\/p>\n

There you can see a list, by project, of tests that are failing and which bugs are assigned to the failure. If bugs are still open, the xFail is still valid. If you have questions regarding your specific test results, please contact us<\/a>!<\/p>\n

BONUS<\/strong><\/em>: One of the best ways to get involved with testing is to submit a pull request on an xFail. For each xFailed test, check the corresponding bug. Has the bug been fixed? First verify the bug, after it has been fixed. At that point the xFail may be removed. Take out the code relating to the xFail, and submit it as your pull request! Once your code has been merged, you will be eligible for one of our badges<\/a>.<\/p>\n

Step 3: Test Failure Analysis<\/h2>\n

What should you do if a test fails? There are a few steps to determine what you do with a test failure.<\/p>\n

a) Bug search: Search through Bugzilla<\/a> to see if the bug was already entered. Enter keywords related to the failure to help your search.<\/p>\n

b) Manual testing: When you get a failure, it\u2019s always a good idea to see if the test passes manually following the same steps as the automated test.<\/p>\n

c) Determine if it is a problem with the test. Are the locators up to date? Did something change in the code that needs to be updated in the test? If the failure is due to the test needing an update you can either submit a pull request fixing the test, or add a GitHub Issue under the GitHub project<\/p>\n

d) I\u2019m just not sure! If you test it and check it in Bugzilla, and you\u2019re still not sure if it\u2019s a real bug, then click this link<\/a> to hop on IRC and ask us!<\/p>\n

Step 4: It\u2019s a bug<\/h2>\n

If you have determined that the test is failing because of a bug, the next step is to file a bug!<\/p>\n

Here are a couple of documents explaining bug writing:<\/p>\n

https:\/\/developer.mozilla.org\/en-US\/docs\/Mozilla\/QA\/Bug_writing_guidelines<\/a><\/p>\n

https:\/\/quality.mozilla.org\/docs\/bugzilla\/starter-kit\/how-to-write-a-proper-bug\/<\/a><\/p>\n

That\u2019s it! Thanks for helping us run automation, and for trouble-shooting when problems arise. This is a complicated process at first, but these are the steps each of our team contributors goes through each time a test starts failing. Any information you can provide regarding the when\/where\/how of it saves lots of time.<\/p>\n

As always, when in doubt you can ask in channel. Click the IRC link <\/a>to join us and tell us what you are seeing. Thanks for your help!<\/p>\n<\/div>\n