A couple months back we started a discussion around our plans for performing some multivariate optimization tests within the mozilla.com site. We started with one simple A/B test on a single web page and we now have some findings to share. At the Firefox customize page — www.mozilla.com/firefox/customize/ — we made a singular text change: the green download button could either include our default text “Download Now – Free” or our test text “Try Firefox 3”.
The results from the test are below. Please keep in mind that the primary purpose of this initial test was to evaluate a possible long-term solution/tool for implementing much more robust tests in the future. That said, our findings provide a small amount of insight. During the course of two weeks, this page saw about 300K visitors and nearly 30K clicks to download Firefox. The conversion rate for “Download Now – Free” was 10.07% versus 9.73% for “Try Firefox 3”, which translates to a 3.44% decrease in optimization when using “Try Firefox 3”, with 99.85% confidence in these results (confidence = likelihood that this version of the page performs significantly different than our default page).
If we’ve piqued your interest, please note that we’ll soon have some exciting findings related to a currently ongoing multivariate test at the main Firefox product page (www.mozilla.com/firefox).