![\"A](\"http:\/\/mozfundraising.wpengine.com\/wp-content\/uploads\/2014\/01\/winning_snippet-600x356.png\")
A “standard” snippet<\/p><\/div>\n
![\"2014](\"http:\/\/mozfundraising.wpengine.com\/wp-content\/uploads\/2014\/12\/Screen-Shot-2014-11-28-at-12.36.53.png\")
2014 EOY Snippet with Donation amounts and button<\/p><\/div>\n
However, even with this new snippet design performing so well, we wanted to test adding payment provider options in for four reasons:<\/h4>\n\n- All Snippet content suffers \u2018 fatigue\u2019 so changing the design keeps it fresh<\/li>\n
- Making PayPal a clearer option on our donation form encouraged what looks like a new type of donor to support the campaign<\/a> – we thought this could happen on the Snippet too<\/li>\n
- The Snippet is our biggest audience by far; small % changes here have significant impact on our overall fundraising campaign<\/li>\n
- For donors choosing PayPal (~%75 of them) it would cut another step out of their donation flow<\/li>\n<\/ul>\n
This was our champion\/control Snippet at that point in the campaign:<\/strong><\/p>\n![\"Screen](\"http:\/\/mozfundraising.wpengine.com\/wp-content\/uploads\/2015\/01\/Screen-Shot-2015-01-26-at-13.47.01.png\")
<\/p>\n
And this is the test variation we ran (the challenger):<\/strong><\/p>\n![\"Screen](\"http:\/\/mozfundraising.wpengine.com\/wp-content\/uploads\/2015\/01\/Screen-Shot-2015-01-26-at-13.47.12.png\")
<\/p>\n
It\u2019s important to note that quite a few things changed in this variation. We also had less data to compare because a significant percentage of the users responding to the snippet were sent directly to the PayPal donation flow without touching our web pages, so we don\u2019t even have impression data. We are comparing these snippets on a strictly end-to-end basis. How many impressions of each snippet vs how many donations were made.<\/p>\n
Results<\/h2>\n
![\"Screen](\"http:\/\/mozfundraising.wpengine.com\/wp-content\/uploads\/2015\/01\/Screen-Shot-2015-01-26-at-13.49.00-600x431.png\")
<\/a>Blue<\/span><\/strong> = The control \/ champion
\nRed<\/span><\/strong> = The new test snippet (challenger)<\/p>\nThe test variant fatigued quickly even though it was only seeing a small percentage of the snippet traffic in relation to the \u2018Select Amount Snippet\u2019 which had been live for a while. From experience testing lots of snippets if we scale up the percentage of the audience to see a variant, performance will fatigue even faster, meaning the \u2018at scale\u2019 results would be worse than these testing results.<\/p>\n
In short, this new design would have a significant negative impact on fundraising income if we were to use it instead of the version we have been using before. So we didn\u2019t scale this up.<\/p>\n
But, we don\u2019t have a clear answer on why<\/em> this was the case (and we never will unless we isolate and test some of these ideas further).<\/p>\nThese are some of the things that could have caused the reduction in donations:<\/h4>\n\n- The icon (fox in a box)\n
\n- Adding an icon might be problematic<\/li>\n
- Adding this particular icon might be problematic (prior snippet testing shows icon selecting is important for driving click-through-rate, and finding the right icon requires testing)<\/li>\n<\/ol>\n<\/li>\n
- The text highlight (or lack of)\n
\n- We may need to highlight the same text in the new snippet design as we did in the control<\/li>\n<\/ol>\n<\/li>\n
- The relative complexity\n
\n- Decision complexity: This form may require too many decisions from the donor too quickly<\/li>\n
- Visual complexity: Too many elements to the design might cause some people to switch off<\/li>\n<\/ol>\n<\/li>\n
- Sending a user straight to PayPal skips some of the Mozilla brand experience \/ loyalty \/ reassurance part of the donation process which may reduce conversion\n
\n- This one is harder to test as we don\u2019t get analytics data from the PayPal funnel<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
In conclusion:<\/h2>\nShould we put payment provider options directly on the snippet?<\/h4>\n
Not in this format.<\/strong> There may be an alternative design that does work really well, but this is not it. We should keep testing in the future, but keep this data point in the back of our minds as we prioritize future work.<\/p>\n","protected":false},"excerpt":{"rendered":"While our End of Year (EOY) fundraising campaign is finished, we still have a few updates to share with you. This post documents one of the A\/B tests we ran during the campaign. Should we put payment provider options directly … Continue reading<\/a><\/p>\n","protected":false},"author":144,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[327515],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/652"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/comments?post=652"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/652\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/media?parent=652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/categories?post=652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/tags?post=652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
This was our champion\/control Snippet at that point in the campaign:<\/strong><\/p>\n And this is the test variation we ran (the challenger):<\/strong><\/p>\n It\u2019s important to note that quite a few things changed in this variation. We also had less data to compare because a significant percentage of the users responding to the snippet were sent directly to the PayPal donation flow without touching our web pages, so we don\u2019t even have impression data. We are comparing these snippets on a strictly end-to-end basis. How many impressions of each snippet vs how many donations were made.<\/p>\n The test variant fatigued quickly even though it was only seeing a small percentage of the snippet traffic in relation to the \u2018Select Amount Snippet\u2019 which had been live for a while. From experience testing lots of snippets if we scale up the percentage of the audience to see a variant, performance will fatigue even faster, meaning the \u2018at scale\u2019 results would be worse than these testing results.<\/p>\n In short, this new design would have a significant negative impact on fundraising income if we were to use it instead of the version we have been using before. So we didn\u2019t scale this up.<\/p>\n But, we don\u2019t have a clear answer on why<\/em> this was the case (and we never will unless we isolate and test some of these ideas further).<\/p>\n Not in this format.<\/strong> There may be an alternative design that does work really well, but this is not it. We should keep testing in the future, but keep this data point in the back of our minds as we prioritize future work.<\/p>\n","protected":false},"excerpt":{"rendered":" While our End of Year (EOY) fundraising campaign is finished, we still have a few updates to share with you. This post documents one of the A\/B tests we ran during the campaign. Should we put payment provider options directly … Continue reading<\/a><\/p>\n","protected":false},"author":144,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[327515],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/652"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/comments?post=652"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/652\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/media?parent=652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/categories?post=652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/tags?post=652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/p>\n<\/p>\nResults<\/h2>\n
<\/a>Blue<\/span><\/strong> = The control \/ champion
\nRed<\/span><\/strong> = The new test snippet (challenger)<\/p>\nThese are some of the things that could have caused the reduction in donations:<\/h4>\n
\n
\n
\n
\n
\n
In conclusion:<\/h2>\n
Should we put payment provider options directly on the snippet?<\/h4>\n