Multivariate Testing Coming to www.mozilla.com

Mozilla will soon be rolling out some optimization testing to one or two pages at www.mozilla.com, and before we get to the point of implementation, we wanted to take a minute to highlight our objectives and goals.

What are we planning to do?

As part of our existing web analytics infrastructure, we’re planning to deploy an A/B and multivariate solution for www.mozilla.com.  Initially, this will be a one-time test on one or two pages – e.g., www.mozilla.com and/or www.mozilla.com/en-US/firefox/ – but depending on the results, we could consider rolling out the solution to other pages or on an ongoing basis.  Multivariate testing allows us to figure out which design and creative elements within a web page have an optimal effect for the visitors to that page.  In our case, we’ll be looking at improving the Fx download rate (also known as “conversion rate”, this rate is determined by the equation downloads/visitors).

Why are we planning to do this testing?

We feel we have a moral imperative to make the Firefox user experience as good as possible at every turn.  When a new user conceives the idea in their mind that they’d like to try Firefox and then he/she goes about the process of actually downloading and installing the browser, we feel these are important steps within the broader Fx user experience.  For example, through testing, if we find that by adjusting some element of the design of the Fx product page (e.g., changing the download button from green to blue) that the download rate increases by 1%, that’s 1% of new community members we’ve previously been losing each day due to the simple fact that a single web page was not optimized.

Furthermore, multivariate testing has the potential for improving the Firefox user experience in many other ways.  For example, we’ll be better able to answer the question: is a user able to optimally find what he/she is looking for at www.mozilla.com, whether that be Firefox support help or a particular Firefox add-on?

Next Steps

While we look forward to sharing the results and findings in the not too distant future, we’re also interested in hearing feedback/suggestions from those within Mozilla and the Mozilla community.  Please don’t hesitate to leave comments below.