A/B Test: Additional ‘ask’ for some channels

We ran this A/B test because conversion rates from the Firefox Snippet seem to be much better than conversion rates from many other places where we ask people to donate. And in some cases, we would expect the opposite to be true, so we are testing our assumptions.

Context for this test:

Most of the donations for our end of year appeal come from the promotional feature in the Firefox about:home snippet. The experience for these donors looks mostly like this.

snippetflowBut we also promote the appeal across other Mozilla ‘channels’ and ‘properties’. For example, we have a banner on the mozilla.org homepage.

otherflowAlong with many other asks in less prominent places like the ‘About Firefox’ window in your Firefox browser.

Our assumption/hunch is that people on the mozilla.org site are one step closer to the Mozilla Mission than people who use Firefox but may not know about Mozilla and the fact that we are a non-profit. In marketing terms, the people closer to the Mission would be ‘warmer’ to a fundraising appeal. We would expect visitors responding to a fundraising ask on mozilla.org to have a higher conversion rate than we see in other channels.

However, while we see excellent conversion rates for many channels, the conversion rates for our ‘warmer’ channels are often much lower than we expect. We can’t make a direct comparison though, because of the differences in the user experience and the point at which we think of someone as signalling interested in donating.

On the snippet flow shown above. A user only makes it to the donation form after actively engaing with a decision to make a donation (ie selecting a donation amount and clicking a Donate Now button). The audience arriving on the donate form from the snippet are pre-filtered because we’ve pushed the donation ask further up in the flow. So these conversion rates look great in isolation.

In many other flows, the user might click a small donate link, or a donate banner without much context and then land directly on the donation form. These users are not converting as well (by comparison with the pre-filtered snippet traffic, and also by benchmarks from our team’s experience working on other fundraising campaigns).

So that’s the context.

The test we ran

If you look closely at the two donation flows shown above, one of the most important differences (at least in theory!) is easy to overlook when you’re close to Mozilla, and the campaign. In the second example, there is no fundraising ‘ask’. There is no ‘reason to give’.

So our hyptohesis was that giving these users more information about who we are and why we need your donations would increase our conversion rate.

So for users arriving from mozilla.org, or clicking our Firefox directory-tile (which was a simple donate graphic) we A/B tested these two form variations.


And the result was…

Screenshot 2015-12-17 11.20.54

…no impact!

So is our hypothosis wrong? Perhaps.

But, there are too many variables in play to give up on the current line of testing yet. So we’re going to test this page further.

Next steps

The first variable we are going to play with on this new ‘additional info’ form is the language of the fundraising ask. Words are powerful (see this for example) and the language used on this first iteration had not been tested previously. Making a donation is largely an emotional decision and those particular words might not motivate people to give.

So we’ve started a follow up test to use the most effective language from our snippet testing in 2014.

As always, we’ll share what we learn here.

2 responses

  1. Anders wrote on :

    People that come to (or known about) mozilla.org are people that are most likely already donating their time (code, bugreports, translations, etc.) to the project. So why would they also spend money? Are there a donate button on getfirefox.com (or somewhere in the “download flow” including the installer) where people might be more inclined/guilty to donate?

    1. Andrea Wood wrote on :

      Hello Anders, thanks for your thoughts and question. To date Mozilla has raised $87,000 from the mozilla.org website since mid-November this year. That is an average gift of $20. Firefox desktop browser users donate an average of $9. People visiting mozilla.org are likely more invested in the project, and many of them probably are volunteers or friends of mozilla in some way. There is no way to know for certain. I also know a high number of people that visit mozilla.org are searching for the Download Firefox page. In other words, not all volunteers are non-donors, and not all visitors to mozilla.org are volunteers.