We ran this A/B test because conversion rates from the Firefox Snippet seem to be much better than conversion rates from many other places where we ask people to donate. And in some cases, we would expect the opposite to be true, so we are testing our assumptions.

Context for this test:

Most of the donations for our end of year appeal come from the promotional feature in the Firefox about:home snippet. The experience for these donors looks mostly like this.

But we also promote the appeal across other Mozilla ‘channels’ and ‘properties’. For example, we have a banner on the mozilla.org homepage.

Along with many other asks in less prominent places like the ‘About Firefox’ window in your Firefox browser.

Our assumption/hunch is that people on the mozilla.org site are one step closer to the Mozilla Mission than people who use Firefox but may not know about Mozilla and the fact that we are a non-profit. In marketing terms, the people closer to the Mission would be ‘warmer’ to a fundraising appeal. We would expect visitors responding to a fundraising ask on mozilla.org to have a higher conversion rate than we see in other channels.

However, while we see excellent conversion rates for many channels, the conversion rates for our ‘warmer’ channels are often much lower than we expect. We can’t make a direct comparison though, because of the differences in the user experience and the point at which we think of someone as signalling interested in donating.

On the snippet flow shown above. A user only makes it to the donation form after actively engaing with a decision to make a donation (ie selecting a donation amount and clicking a Donate Now button). The audience arriving on the donate form from the snippet are pre-filtered because we’ve pushed the donation ask further up in the flow. So these conversion rates look great in isolation.

In many other flows, the user might click a small donate link, or a donate banner without much context and then land directly on the donation form. These users are not converting as well (by comparison with the pre-filtered snippet traffic, and also by benchmarks from our team’s experience working on other fundraising campaigns).

So that’s the context.

The test we ran

If you look closely at the two donation flows shown above, one of the most important differences (at least in theory!) is easy to overlook when you’re close to Mozilla, and the campaign. In the second example, there is no fundraising ‘ask’. There is no ‘reason to give’.

So our hyptohesis was that giving these users more information about who we are and why we need your donations would increase our conversion rate.

So for users arriving from mozilla.org, or clicking our Firefox directory-tile (which was a simple donate graphic) we A/B tested these two form variations.

And the result was…

…no impact!

So is our hypothosis wrong? Perhaps.

But, there are too many variables in play to give up on the current line of testing yet. So we’re going to test this page further.

Next steps

The first variable we are going to play with on this new ‘additional info’ form is the language of the fundraising ask. Words are powerful (see this for example) and the language used on this first iteration had not been tested previously. Making a donation is largely an emotional decision and those particular words might not motivate people to give.

So we’ve started a follow up test to use the most effective language from our snippet testing in 2014.

As always, we’ll share what we learn here.