SUMO is Deploying Continuously

As of a couple of weeks ago, the Support Engineering team is deploying changes to SUMO continuously. This is the result of over a year of work by developers, sysadmins, QA, and more, and I couldn’t be prouder of everyone involved.

Continuous deployment is a complicated term, but at its core it means the engineers have the ability to update the software that runs SUMO at any time.

Not only do we have the ability, but we’re using it. Instead of pushing changes once per week, or when a feature was “complete,” we’re deploying changes as often as we can. That means bug fixes and new features will make it to our users and contributors much, much faster. The first week we pushed seven code updates in four days with zero downtime.

An example of a mid-day performance improvement.

Beyond getting fixes, features, and improvements out to users faster, we have made a number of improvements on the road to getting here.

• Simplified and automated our deployment process.
• Automated test coverage of the client-side parts of the code.
• Made our tests run as close to check-in time as currently possible.
• Added the ability to dark launch and use feature flags to integrate continuously.
• Started graphing hundreds of application metrics in real-time.
• Made our staging servers consistent and up-to-date environments.
• Improved our planning process for predictability.
• And a lot more…

Those are all necessary for us to deploy continuously, but they are also great things to have by themselves. The road to get here has been long, but filled with positive changes along the way.

So what will be different? How will this affect our users and contributors?

Primarily, our users will have a better experience, as issues we discover won’t be around very long. More subtly, during this process we have eliminated downtime from all but the rarest and most complex of code changes, we’ve dramatically improved our confidence in the code we push, and we’re able to objectively monitor and improve in ways we haven’t been able to before.

Again, I can’t express enough how proud I am of everyone involved for reaching this point, and how happy I am that the transition was an utter non-event—just like our deploys are now.

We will continue to improve this process, and refine our planning around it. And we will share and export everything we’ve learned on the way, both to other web projects within Mozilla and to the web development and sysadmin communities at large.