Let’s start with the punchline: By making a few minor tweaks to our top landing pages, we can drive an additional 60,000,000 Firefox downloads per year.
60 million is a big number, so let’s add a little context. Looking back at our record breaking download day during the launch of Fx3 (June 2008), a total of 8,002,530 people downloaded Firefox. Adding 60 million downloads is akin to running 10 Download Day campaigns per year.
What are the tweaks?
It comes down just one factor — speed. As it stands, our landing pages can be painfully slow. Take a look:
The entire Chrome landing page loads before our header appears. Worse yet, our download button doesn’t appear for 7 seconds! While this video does not reflect the experience of every IE.html visitor, nearly a quarter of our visitors wait at least this long.
After implementing these changes in an A/B test, we saw impressive results. Previously, we predicted a 1 second reduction in page load speed would improve download conversions by 2.7%. In reality, our optimized experimental variation shaved 2.2 seconds off the average page load time and increased download conversions by 15.4%!
With 275,000 daily visitors, a 15% improvement on this single English page translates to 10.28 million additional downloads per year. And, if we’re able to achieve a similar performance boost across our other top landing pages, we’ll drive in excess of 60 million yearly Firefox downloads.
15% looks great, but how confident are we?
Running a one-sided Student’s t-test with a means difference of 14%, our experimental data yields a P value of 0.000051. This means that there is only a 0.0051% chance that we would obtain a 14% (or greater) improvement if the real effect wasn’t at least this large.
We’re excited by our initial results, but are only just getting started. We have a bunch more optimizations to try and have some ideas of our own. For example, we want to test a landing page that loads the download button before all other page elements. Later on, we plan to test more elements of Firefox’s adoption funnel, including the download and First Run experiences.
Many thanks go out to Ryan, Fred, Laura, and Matthew, who suggested running performance tests and has already made impressive performance improvements on addons.mozilla.org.