While our End of Year (EOY) fundraising campaign is finished, we still have a few updates to share with you. This post documents one of the A/B tests we ran during the campaign.

Should we put payment provider options directly on the snippet?

This was a test of the content on the Firefox about:home Snippet; our highest visibility call to action during the EOY campaign. In the past, fundraising snippets have been a simple icon, message and link to a donation page but this year we tested putting donation amount options and a donation button directly into the space on the snippet. Getting this ‘action’ content into the snippet was more effective than we’d hoped for, and was one of the factors that helped us raise so much more this year than last.

However, even with this new snippet design performing so well, we wanted to test adding payment provider options in for four reasons:

• All Snippet content suffers ‘ fatigue’ so changing the design keeps it fresh
• Making PayPal a clearer option on our donation form encouraged what looks like a new type of donor to support the campaign – we thought this could happen on the Snippet too
• The Snippet is our biggest audience by far; small % changes here have significant impact on our overall fundraising campaign
• For donors choosing PayPal (~%75 of them) it would cut another step out of their donation flow

This was our champion/control Snippet at that point in the campaign:

And this is the test variation we ran (the challenger):

It’s important to note that quite a few things changed in this variation. We also had less data to compare because a significant percentage of the users responding to the snippet were sent directly to the PayPal donation flow without touching our web pages, so we don’t even have impression data. We are comparing these snippets on a strictly end-to-end basis. How many impressions of each snippet vs how many donations were made.

Results

Blue = The control / champion
Red = The new test snippet (challenger)

The test variant fatigued quickly even though it was only seeing a small percentage of the snippet traffic in relation to the ‘Select Amount Snippet’ which had been live for a while. From experience testing lots of snippets if we scale up the percentage of the audience to see a variant, performance will fatigue even faster, meaning the ‘at scale’ results would be worse than these testing results.

In short, this new design would have a significant negative impact on fundraising income if we were to use it instead of the version we have been using before. So we didn’t scale this up.

But, we don’t have a clear answer on why this was the case (and we never will unless we isolate and test some of these ideas further).

These are some of the things that could have caused the reduction in donations:

1. The icon (fox in a box)
   1. Adding an icon might be problematic
   2. Adding this particular icon might be problematic (prior snippet testing shows icon selecting is important for driving click-through-rate, and finding the right icon requires testing)
2. The text highlight (or lack of)
   1. We may need to highlight the same text in the new snippet design as we did in the control
3. The relative complexity
   1. Decision complexity: This form may require too many decisions from the donor too quickly
   2. Visual complexity: Too many elements to the design might cause some people to switch off
4. Sending a user straight to PayPal skips some of the Mozilla brand experience / loyalty / reassurance part of the donation process which may reduce conversion
   1. This one is harder to test as we don’t get analytics data from the PayPal funnel

In conclusion:

Should we put payment provider options directly on the snippet?

Not in this format. There may be an alternative design that does work really well, but this is not it. We should keep testing in the future, but keep this data point in the back of our minds as we prioritize future work.