We have 36 precious hours left in the campaign, and that also means we’ve come to the end of testing and optimization on our donation flow. Everything we have learned in our testing over the past 1.5 months has brought us to this: Our very best performing version of the snippet — the one that donors respond to most. This post is about our final round of tests and how we arrived at the ultimate “winner” that will go live during our critical last few hours.

After several rounds of testing Adam already posted about, we settled on this version of the snippet as our high performer, which was shown to the most traffic December 9 – 28 (the screenshot is from the Firefox Developer Edition, hence the blue logo):

Part of the challenge of the snippet is that there is a predictable pattern of click fatigue on the same design if presented without change for several days. This not only requires us to “freshen” the snippet every few days to keep the click through rate (CTR) from going to the floor, but we have to factor this fatigue into our testing strategy and results interpretation.

We wanted to test different variations of the copy, and to help offset click fatigue we also varied the highlight color. This was not a structured test to find an optimal text highlight color. We were following a hunch that varying the color from time to time is more effective than repeating whatever single color might perform best in a given test. This would be interesting to test in a more structured way one day, but we didn’t have time for that test this year.

Here are the copy variants we tested, along with varying highlight colors:

Once these new text variations had been through the necessary internal checks, we set them up to be seen by a small percentage of the overall audience, while our existing ‘best performer’ continued to be seen by the majority of the audience.

Then over the course of a few days we watched how each of these text variations performed comparing them on end-to-end impression-to-donation metrics ($ / 1000 impressions). As the data came in we shifted the allocation of audience percentages based on which variations of the text performed the best.

And we were excited to find a new best performer. Which version do you think yielded the most donations?

Key:

Our new winner, by a significant margin, is the longest one — in fact the longest snippet text we’ve ever tested. This version that begins “Hello there” almost didn’t make it into the testing as we only planned to try four variants and it was an extra idea added at the last minute. We were also concerned because it is so long that the red “Donate now” button was bumped down quite a bit.

But in the end, users really responded to the “Hello there” version! It’s always tempting to make up explanations for why a variation works well after you’ve seen the results, but in truth there are lots of things about this which could account for the positive effect. We don’t know for sure what the key to the success of this ask is, but some things we should continue to explore in the future:

• Talking about the connection and the differences between Firefox as a product, and Mozilla as a non-profit organization;
• Simply telling people about some of the awesome things Mozilla does (and that their gift can help enable us to do);
• Longer text is visually disruptive because it’s not usual on that page;
• Longer text allows us to tell a more complete appeal story (albeit still a short one);
• It’s more friendly and direct and honestly written than a lot of fundraising appeals people see;
• “Hello there” is an interesting opener (email campaigners have seen similar results with subject lines). This would likely be something people tire of quickly, but it may have worked in this case.

We found our winning text. However, to turn up the visual “volume” in our final hours of the campaign, we tested alternate background colors paired with the winning text variant (dark grey and yellow). Because it is such a departure from the about:home normal design, this is a strategy we reserve for only the most critical final hours. Testing showed the yellow background color (red in the chart) performed better than grey (orange):

After many many test variants over 1.5 months, we arrive at our very best performing snippet:

This one will be shown to 100% of snippet viewers until midnight December 31st. Thanks to testing, we are able to present a snippet we know users respond more to (though we can’t say precisely why) and this will result in higher response, and increased donations.

Thanks very much to Adam Lofting who conducted the analysis and contributed to this blog post.