As previously introduced, and subsequently rolled out in an initial test, we recently wrapped up our evaluation of a multivariate website optimization tool with a more complex test on the main Firefox product page – www.mozilla.com/firefox. We considered several variations of three components on the page – the headline, sub-headline, and text within the download button. To accomplish this, the tool had us create 16 different variations, or recipes, of our page. We then exposed these recipes to a small percentage of IE visitors to our site.
What did we find?
Some good news and some not so significant news.
The first thing we noticed was that no recipe performed significantly better than our existing Firefox product page (the control page during our test) in a statistically significant way. That said, there was one strong and valid finding. The four recipes shown below all performed significantly worse than our existing page in a statistically significant way. What’s the common thread among these four recipes?
These are the four recipes using “Try Now!” as the text within the download button. The other two variations of text used within the button were “Download” and “Free Download”.
How does this change our business/creative decision making?
As some of us already guessed, it appears that it’s the download button itself that matters most here. Thinking about the size, shape, color, and placement (among other characteristics), our findings indicate that future testing could reveal surprising – and positive findings – based on changes to the download button. In other words, it wouldn’t be a stretch to imagine that future tests, that adjust such characterisitcs of the download button, could easily translate to a 2% lift (or much greater), and in turn, a substantially improved experience for millions of new Firefox users each year.
For those interested, the full results from our test are below. The “winner” is highlighted in green and the four recipes discussed above are highlighted in red.