{"id":343,"date":"2015-01-15T23:39:05","date_gmt":"2015-01-15T23:39:05","guid":{"rendered":"http:\/\/blog.mozilla.org\/webqa\/?p=343"},"modified":"2015-01-15T23:39:05","modified_gmt":"2015-01-15T23:39:05","slug":"make-the-intern-do-it","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/fxtesteng\/2015\/01\/15\/make-the-intern-do-it\/","title":{"rendered":"Make the intern do it"},"content":{"rendered":"<h1><a href=\"http:\/\/blog.mozilla.org\/fxtesteng\/files\/2015\/01\/mbrandt.jpg\"><img decoding=\"async\" loading=\"lazy\" class=\" size-medium wp-image-347 alignleft\" src=\"http:\/\/blog.mozilla.org\/fxtesteng\/files\/2015\/01\/mbrandt-252x168.jpg\" alt=\"mbrandt\" width=\"252\" height=\"168\" \/><\/a><\/h1>\n<p>Hi, I&#8217;m Matt Brandt [:mbrandt in irc]. I&#8217;m a core contributor on Mozilla&#8217;s Web QA team. I work for the dino.<\/p>\n<h1>A shared experiment<\/h1>\n<p>As a QA team we had a difficult and challenging quarter to end 2014 on; that said, the work with the <a href=\"https:\/\/developer.mozilla.org\/\" target=\"_blank\">MDN<\/a> team was a personal highlight for me. A fulfilling way to cap off the end of the year.<\/p>\n<p>The goal was to implement an experiment: convert several Python-based tests into JavaScript. Place these tests into the build process that the developers use and see what it nets.<\/p>\n<p><a href=\"http:\/\/davidwalsh.name\/\" target=\"_blank\">David Walsh<\/a> and I evaluated <a href=\"http:\/\/theintern.io\/\" target=\"_blank\">theIntern.io<\/a> as an alternative to our Python based<br \/>\n<a href=\"https:\/\/github.com\/mozilla\/mdn-tests\/\" target=\"_blank\">suite of tests<\/a> that use WebDriver, Pytest, and <a href=\"https:\/\/github.com\/mozilla\/pytest-mozwebqa\" target=\"_blank\">pytest-mozwebqa<\/a>. The developers very much want to be involved in end-to-end testing using WebDriver as a browser automation tool but found the context switching between JavaScript and Python a distraction. <a href=\"https:\/\/quality.mozilla.org\/teams\/web-qa\/\" target=\"_blank\">Web QA<\/a> wanted our test cases to be more visible to the developers and block deployment if regressions were introduced into the system.<\/p>\n<h3>The problem(s) we wanted to address:<\/h3>\n<ul>\n<li>Scalability of test coverage &#8211; joint ownership of creation and maintenance of test cases.<\/li>\n<li>Moving test runs further upstream in the build process &#8211; providing<br \/>\nquicker feedback.<\/li>\n<li>Test runs and reports are hooked into a build process that the dev team uses.<\/li>\n<\/ul>\n<h2>Present<\/h2>\n<p>Presently, Web QA\u2019s MDN tests are Python based &#8212; <a href=\"https:\/\/github.com\/mozilla\/mdn-tests\/\" target=\"_blank\">suite of tests<\/a> &#8212; and run on a <a href=\"http:\/\/jenkins-ci.org\/\" target=\"_blank\">Jenkins<\/a> instance which is fire-walled from the public. Both of these components make accessing and monitoring our test runs complex and an event that is separate from the developer workflow.<\/p>\n<p>Developer-level unit tests run on a <a href=\"https:\/\/travis-ci.org\/mozilla\/kuma\" target=\"_blank\">travis-ci instance<\/a> and presently there is no hook that runs the Python based end-to-end tests if their build passes. The end-to-end tests run on a cron job and can also be started by an <a href=\"https:\/\/github.com\/mozilla\/mozwebqa-bot\" target=\"_blank\">irc bot<\/a>.<\/p>\n<p>Additionally, our Python-based tests live outside of the developers project: <a href=\"https:\/\/github.com\/mozilla\/mdn-tests\/\" target=\"_blank\">github.com\/mozilla\/mdn-tests<\/a> and <a href=\"https:\/\/github.com\/mozilla\/kuma\/pull\/2947\/files\" target=\"_blank\">github.com\/mozilla\/kuma<\/a>. Adding an additional hurdle to maintainability and visibility.<\/p>\n<h2>The New<\/h2>\n<p>Working closely with the MDN team I helped identify areas that an end-to-end testing framework needed to take into account and solve. Between David\u2019s and my experiences with testing we were able to narrow our search to a concise set of requirements.<\/p>\n<h3>Needs included:<\/h3>\n<ul>\n<li>Uses real browsers &#8212; both on a test developer\u2019s computer as well as the ability to use external services such as Sauce Labs.<\/li>\n<li>Generate actionable reports that quickly identify why and where a regression occurred<\/li>\n<li>An automatic mechanism that re-runs a test that fails due to a hiccup; such as network problems, etc.<\/li>\n<li>Choose an open source project that has a vibrant and active community<\/li>\n<li>Choose a framework and harness that is fun to use and provides quick results<\/li>\n<li>Hook the end-to-end tests into the build process<\/li>\n<li>Ability to run a single test case when developing a new testcase<\/li>\n<li>Ability to multithread and run n tests concurrently<\/li>\n<\/ul>\n<p>David Walsh suggested theIntern.io and after evaluating it we quickly realized that it hit all of our prerequites. I was quickly able to configure the <a href=\"https:\/\/github.com\/mozilla\/kuma\/pull\/2947\/files\" target=\"_blank\">MDN travis-ci instance<\/a> to use <a href=\"https:\/\/docs.saucelabs.com\/ci-integrations\/travis-ci\/\" target=\"_blank\">SauceLabs (configuring travis-ci)<\/a>. Using the included <a href=\"https:\/\/theintern.github.io\/digdug\/index.html\" target=\"_blank\">DigDug<\/a> library for tunneling, tests were quickly able to run on Sauce Lab\u2019s infrastructure.<\/p>\n<h3>Outstanding issues<\/h3>\n<ul>\n<li>The important <a href=\"https:\/\/theintern.github.io\/leadfoot\/Command.html#isDisplayed\" target=\"_blank\">isDisplayed method<\/a> of Intern&#8217;s <a href=\"https:\/\/theintern.github.io\/leadfoot\/index.html\" target=\"_blank\">Leadfoot API<\/a>, paired with the polling function, returns inconsistent results. The cause may be as simple as we\u2019re misusing isDisplayed(), polling, or the isDisplayed() has a bug. This will take more investigation.<\/li>\n<li>Due to security restrictions Travis-CI cannot run end-to-end Webdriver based tests on a <a href=\"http:\/\/docs.travis-ci.com\/user\/pull-requests\/#Security-Restrictions-when-testing-Pull-Requests\" target=\"_blank\">per pull request basis<\/a>. Test jobs are limited to merges to master only. We need to update the <a href=\"https:\/\/github.com\/mozilla\/kuma\/pull\/2947\/files\" target=\"_blank\">.travis.yaml <\/a>file to exclude the end-to-end tests when a pull request is made.<\/li>\n<li>Refactor the current tests to use the Page Object Model. This was a fairly large deliverable to piece together and get working, thus David and I decided to not worry about best coding practices when trying to get a minimally viable example working.<\/li>\n<\/ul>\n<p>We\u2019re well-positioned to see if Javascript-based end-to-end tests that block a build will provide positive benefits for a project.<\/p>\n<h2>Next Steps<\/h2>\n<p>Before the the team is ready to trust using Leadfoot and Digdug we need to<br \/>\nsolve the outstanding issues. If you are intestested in gettin involved,<br \/>\nplease let us know.<\/p>\n<ul>\n<li>Remove the sleep statements from the code base &#8212; this likely will involve better understanding how to use Promises.<\/li>\n<li>Get an understanding why some tests fail when run against SauceLabs &#8212; the failures may be legitimate, we may be misusing the api, or their is a defect somewhere within the test harness.<\/li>\n<li>Refactor tests to use the Page Object Model.<\/li>\n<\/ul>\n<p>After these few problems are solved, the team can include end-to-end into their<br \/>\nbuild process. This will allow timely and relevant per feature merge feedback<br \/>\nto developers on the project.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi, I&#8217;m Matt Brandt [:mbrandt in irc]. I&#8217;m a core contributor on Mozilla&#8217;s Web QA team. I work for the dino. A shared experiment As a QA team we had &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/fxtesteng\/2015\/01\/15\/make-the-intern-do-it\/\">Read more<\/a><\/p>\n","protected":false},"author":1359,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[228,25193],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts\/343"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/users\/1359"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/comments?post=343"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/posts\/343\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/media?parent=343"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/categories?post=343"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/fxtesteng\/wp-json\/wp\/v2\/tags?post=343"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}