Web QA: 2014 – Year in Review

(I love this team so much — and we’re not all represented, here, even!)

Shamelessly stealing a page from Laura Thomson’s “2014: Engineering Operations Year in Review” blogpost, Web QA would also like to take this slightly belated opportunity to highlight many — but certainly not all — team accomplishments:

Infrastructure:

• started the process of creating a new, publicly open Jenkins instance for all our automation
• added Windows virtual machines in SCL3
• moved both Bouncer and Snippets test automation into the public, Webdev Jenkins (ci.mozilla.org)
• built out and maintained a 40+ device Firefox OS test-automation lab

Marketplace:

• got Marketplace Payments tests (UI flow, using Marionette) running on a Firefox OS Flame device using Zippy
• switched Android tests over to Appium, using Sauce Labs
• shipped Marketplace on low-spec Tarako devices/branch
• shipped serverless in-app payments
• supported (both manually and with automation) the split of AMO and Marketplace
• shipped update notifications and feature detection for Hello and WMW
• shipped Operator Dashboard
• shipped Curation tools
• shipped Feed!
• shipped Bango dev-portal integration into DevHub
• shipped single-page webpay
• migrated from Persona to FxA (and supported both, through manual testing and automation, throughout the transition)
• shipped support for pre-installed apps
• shipped support for hidden apps
• shipped recommended apps
• shipped support for Newsletters
• shipped communication dashboard for  interaction between reviewers and developers
• enabled payments in regions like Mexico and worked extensively on getting payments working on Bangladesh

MDN:

• theintern.io – Collaboration on an experiment/prototype between teams: Web QA and MDN
• audited JS WebDriver frameworks and decided to use theintern.io
• created a representative subset of our Python-based tests using JS
• configured the build — travis-ci — to run the end-to-end tests using Sauce Labs

Mozilla.org:

• tested and shipped 10-year anniversary pages/tours
• tested and shipped Yahoo! tour
• tested shipped Developer Edition pages/tours
• tested and shipped a Basket rearchitecture
• tested and shipped revamped Get Involved flow
• added homepage coverage through automation:
  • https://github.com/mozilla/mcom-tests/pull/343
• streamlined a bunch of tests by removing is_visible() checks, where they were redundant or unreliable:
  • https://github.com/mozilla/mcom-tests/commit/369a17902d50e9c18193db83be1091a99ab51b52

Mozillians:

• shipped vouch refactor
• shipped geolocation
• moved the project’s development environment to Docker
• successfully transitioned the project to a community QA Lead

MozTrap:

• Acting PM – created a prioritized list of triaged feature needs
• Began to solve scaling problems in MozTrap to reduce pain for QA – unresponsive UX, long running queries, data integrity (more to come in 2015!)

Socorro:

• Migrated crash-stats storage off HBase and into S3
• Crash-stats reporting for B2G

OWASP ZAP:

• pytest-zap work to support deeper integration with OWASP ZAP + pytest-mozwebqa

Web QA Dashboard:

• Published the Web QA dashboard, which initially included information on xfails and skips for all of our projects
  • http://mozilla.github.io/mozwebqa-dashboard/#/xfails
• Added the Web QA Issues dashboard, which displays all open issues and pull requests for our GitHub projects
  • http://mozilla.github.io/mozwebqa-dashboard/#/issues
• Added the Marketplace tests dashboard, which provides a quick reference for the tests we run for Marketplace and shows which are passing, failing, and skipped
  • http://mozilla.github.io/mozwebqa-dashboard/#/marketplace

One and Done:

• Improved admin system for creating and editing tasks
• Added admin views for Task Activity and Metrics
• Ability to create “one time only” tasks
• Ability to create tasks from Bugzilla bugs
• Publicly viewable profiles, with a customizable URL
• User notifications for expired and/or closed tasks
• updated privacy protocols
• ability to interact with users who submit feedback
• One and Done test days
• Community highlights

QMO:

• shipped a redesigned (and long-awaited) quality.mozilla.org blog platform:
  • http://quality.mozilla.org/2015/01/announcing-a-redesign-of-quality.mozilla.org/

Open Badges:

• https://badges.mozilla.org/en-US/profiles/profile/webqa.badges
  • we added many new badges:
    • http://quality.mozilla.org/2014/04/announcement-web-qa-team-badges-2/

Presentations & Meetups:

• Python Development Standards:
  • http://secretmustache.com/presentations/2014/11/python_development_standards
•  Building a 40+ real-phone test-automation lab:
  • http://www.meetup.com/seleniumsanfrancisco/events/211095862/
• Engagement offsite – two presentations: 1) Continuous Delivery and how to integrate community, 2) What does QA do — an overview of what comprises a test plan
• Continuously detecting web security vulnerabilities:
  • http://people.mozilla.org/~dhunt/talks/seunconf2014/
• started an “I am MozWebQA” blog series:
  • https://blog.mozilla.org/webqa/2014/11/21/i-am-mozilla-web-qa-meet-stephen-donner/
• Intro to Web testing test days [multiple] + SF meetup
• Summer project internships
• Test automation clean up test days [multiple]
• Two OPW internships
• Xfail test days

A huge thanks yet again to the entire team for really coming together and making  our efforts in 2014 really pay off — here’s to 2015!