Our end of year (EOY) fundraising campaign is getting started today, so I wanted to write a note about our A/B testing plan and the preparation work that has gone into this so far.
Although right now our donation form looks pretty much the same as it did at the end of 2014, we’ve rebuilt it from the ground up with a much more robust development and deployment workflow giving our engineers (and the rest of the team) much more confidence in deploying test variations while our campaign is live. This is going to make the A/B testing work even more fun in 2015. Watch this space as we share the test results during December.
Keeping up with the pace of testing
One of the excellent challenges we have during our campaign is that we see so much traffic in a short period of time that we can run and conclude A/B very quickly. In 2014 we were concluding tests quicker than we could design and develop new test ideas, so some of our tests ended up being on smaller details which were impactful, but risked going down the route of optimizing for the local maximum.
Another challenge of 2014 was that updating our forms involved multiple copy and paste tasks into the live production environment. Which is not good for the stress levels of any engineer working on a form that sees millions of sessions during this campaign. This year we’re in much better shape and with many layers of testing and staged deployment we can be confident testing more significant changes to the form.
Generating ideas for 2015
Our A/B testing plan got started when the team working on fundraising met in June. Andrea supplied a collection of creative toys and games to get our brains in gear and we generated enough ideas to see us through to 2017 at least!
We gathered up these ideas, and along with photos of the sketches put them all into Github tickets as distinct items to discuss and prioritise over time. A number of these tickets evolved into wire-framing, design and development tasks that we’ve been working on in the months between that meet-up and the launch of the campaign.
Prioritisation
More recently, we spent time evaluating our list of possible A/B test ideas and decided which we were best able to implement this year, and how we should order these.
In A/B testing, it’s often hard to predict which tests are going to have the biggest impact, so this process is more art than science but basically we’re weighing up the impact we think each idea could have on the campaign versus the effort to implement it. This is a screenshot of our current test plan – you can click it to see it in more detail.
Bluesky
Many of the ideas we generated were beyond the capacity of our team right now, but we don’t want to lose them, and we want a place to keep those discussions alive until we are ready to work on them. These tickets live a section of the fundraising repo called Bluesky.
In 2016, we’ll be building on that thinking rather than starting from scratch.
So, here goes 2015…
And like last year, we’ll be publishing and sharing the results of the A/B tests we’re running throughout the campaign. Keep an eye on this fundraising blog for more updates from the team.