{"id":504,"date":"2013-02-15T08:26:39","date_gmt":"2013-02-15T15:26:39","guid":{"rendered":"http:\/\/blog.mozilla.org\/axel\/?p=504"},"modified":"2013-02-15T08:26:39","modified_gmt":"2013-02-15T15:26:39","slug":"risk-management-for-releases-at-scale","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/axel\/2013\/02\/15\/risk-management-for-releases-at-scale\/","title":{"rendered":"Risk management for releases at scale"},"content":{"rendered":"

Let me share some recent revelations I had. It all started with the infamous Berlin airport. Not the nice one in Tegel, but the BBI<\/abbr> desaster. The one we’ve thought we’d open last year, and now we don’t know which year.<\/p>\n

Part of the newscoverage here in Germany was all about how they didn’t do any risk analysis, and are doomed, and how that other project for the Olympics in London did do risk analysis, and got in under budget, ahead of time.<\/p>\n

So what’s good for the Olympics can’t be bad for Firefox, and I started figuring out the math<\/a> behind our risk to ship Firefox, at a given time, with loads of localizations. How likely is it that we’ll make it?<\/em><\/p>\n

Interestingly enough, the same algorithm can also be applied to a set of features that are scheduled for a particular Firefox release. Locales, features, blockers, product-managers, developers, all the same thing :-). Any bucket of N things trying to make a single deadline have similar risks. And the same cure. So bear with me. I’ll sprinkle graphs as we go to illustrate. They’ll link to a site<\/a> that I’ve set up to play with the numbers, reproducing the shown graphs.<\/p>\n

The setup is like this: Every single item (localization, for exampe) has a risk, and I’m assuming the same risk across the board. I’m trying to do that N times, and I’m interested in how likely I’ll get all of them. And then I evaluate the impact of different amounts of freeze cycles. If you’re like me, and don’t believe any statistics unless they’re done by throwing dices, check out the dices demo<\/a>.<\/p>\n

Anyway, let’s start with 20% risk per locale, no freeze, and up to 100 locales.<\/p>\n

\"80%,<\/a><\/p>\n

Ouch. We’re crossing 50-50 at 3 items already, and anything at scale is a pretty flat zero-chance. Why’s that? What we’re seeing is an exponential decay, the base being 80%, and the power being how often we do that. This is revelation one<\/strong> I had this week.<\/p>\n

How can we help this? If only our teams would fail less often? Feel free to play with the numbers, like setting the successrate from 80% to 90%. Better, but the system at large still doesn’t scale. To fight an exponential risk, we need a cure that’s exponential.<\/p>\n

Turns out freezes are just that. And that’d be revelation two<\/strong> I had this week. Let’s add some 5 additional frozen development cycles.<\/p>\n

\"80%,<\/a><\/p>\n

Oh hai. At small scales, even just one frozen cycle kills risks. Three features without freeze have a 50-50 chance, but with just one freeze cycle we’re already at 88%, which is better than the risk of each individual feature. At large scales like we’re having in l10n, 2 freezes control the risk to mostly linear, 3 freezes being pretty solid. If I’m less confident and go down to 70% per locale, 4 or 5 cycles create a winning strategy. In other words, for a base risk of 20-30%, 4-5 freeze cycles make the problem for a localized release scale.<\/p>\n

It’s actually intuitive that freezes are (kinda) exponentially good. The math is a tad more complicated, but simplified, if your per-item success rate is 70%, you only have to solve your problem for 30% of your items in the next cycle, and for 9% in the second cycle. Thus, you’re fighting scale with scale. You can see this in action on the dices demo<\/a>, which plays through this each time you “throw” the dices.<\/p>\n

Now onwards to my third revelation<\/strong> while looking at this data. Features and blockers are just like localizations. Going in to the rapid release cycle with Firefox 5 etc, we’ve made two rules:<\/p>\n