We recently ran a week long A/B optimization test at www.mozilla.com where we rotated in a few different colors of the Firefox download button. (You may recall that we’ve previously conducted one prior multivariate test on the Firefox product/download page… check it out for some historical context.) For this current test, here was our existing page and download button:
Within this page (the green download button was our “control” color), we randomly rotated three other colors of the download button – blue, purple, and orange/yellow:
The results are below. The difference between a 76.5% conversion rate and a 77.3% conversion rate is a 1% lift, i.e., the percentage change between those two percentages. So, what does this translate to in terms of Firefox usage? Assuming we see about 500K daily downloads at www.mozilla.com, a 1% lift translates to nearly 2 million marginal, or incremental, downloads annually (i.e., 2 million potential new Firefox users we would have otherwise lost).
We’re already running the winning button color, so we’re now looking forward to future optimization testing (e.g., a download button quadrupled in size, an octagon shaped download button, a button centered within the page, etc.) to see if we can actually gain a 1% lift (or ideally a 5%+ lift), and in the process, improve our site experience for millions of new Firefox users.
Lastly, I want to thank John Slater, Laura Mesa, Monique Johnson, Stephen Donner, Neil Lee, Jeremy Orem, and Alex Buchanan for making this initiative happen.