Mozilla Online Advertising (Part 2) – Experiment Findings and Marginal Cost

Ken Kovash

19

Here’s a quick quiz to start our discussion:

If you acquire three customers through online advertising and your total cost is $3 and your reporting system shows your cost per acquisition as $1, what was your true cost to acquire each customer?

Hint: the correct answer is not $1 to acquire customer A, $1 for customer B, and $1 for customer C. You’ll find our answer at the end of this post.

In our previous post, we discussed an upcoming experiment related to Mozilla’s online advertising efforts. We saw with funnelcake that approximately 38% of Firefox downloaders originate from search engines, so understanding users’ behavior within this channel – along with the full costs and benefits of our paid search advertising – is critical from a marketing perspective.

The question we set out to answer is this: if we turn off our pay-per-click ads, will a user who would have normally been expected to click on our paid ad simply click on our regular (i.e., organic) search listing?

Our problem solving methodology somewhat mirrored the flicking on/off of a light switch, as we conducted the experiment in such a way that we were able to largely isolate the effect of having our search ads on/off on a change in the number of Firefox downloads. Here’s the methodology we used:

  • Utilized a Monday through Thursday time period within a particular week. We’ve previously noted that internet traffic appears fairly consistent across these days.
  • On alternate days, we turned off our paid ads. For example, one week our ads were active on Tuesday and Thursday and turned off on Monday and Wednesday. In the other week, we reversed the on/off days. We alternated like this to reduce the chance of any biasness or exogenous factors affecting the experiment.
  • The audience involved includes only users downloading the en-US version of Fx
  • The results from Week 1 and Week 2 are presented below. The pie charts answer the question: Where are downloaders coming from?

Here’s what we found in Week 1:

Here’s what we found in Week 2:

How should we interpret these pie charts?

Approach A – How much of that green pie slice are we still able to capture with zero ads?

  • You’ll see in Week 1 that 19.3% of downloaders came through organic search and 7.4% arrived via paid search. When we turned off our paid ads, our organic search did not pick up that entire group of 7.4%, but it did pick up most of that group (24.6-19.3, or 5.3%).
  • Taking 5.3/7.4 tells us that we capture 72% of would be paid ad downloaders through our regular (i.e., organic) search listing when we turn off our paid search ads.

Approach B – Total effect on aggregate numbers

  • What was the effect on total number of downloads (just the raw number)?
  • In Week 1 we saw 142K downloads through organic search when our paid ads were turned off. And when our paid ads were active? We saw 114K downloads via organic search and 44K downloads through the paid search ads.
  • So, we captured a total 158K downloads vs. 142K by having our paid search ads active, or an 11% increase when considering just total numbers.

What does this mean for Mozilla marketing and our future efforts related to online advertising? While the findings are both surprising and telling, it’s important to remember that the steps outlined above were only an experiment. We’re not going to dramatically transform our business practices overnight without fully considering all ramifications (i.e., we’re not going to instantly turn off all Firefox advertising tomorrow). That said, these findings do tell us that we need to seriously rethink our existing online advertising strategies and tactics.

David and I look forward to returning to this subject again in the future, and we encourage your thoughts and feedback.

Lastly, returning to our original quiz, Mozilla’s cost of acquiring three new users would look like this (please note: the dollar figures are hypothetical):

customer A = $0, customer B = $0, and customer C = $3

How can we explain this? Marketers have been taught to think of costs associated with acquisition marketing as average costs (e.g, cost per acquisition). Borrowing a term from economics, we would benefit by transforming our thinking to that of marginal costs. In other words, marketers should be asking themselves, “is it worth $3 to acquire customer C?”, not “is it worth $1 to acquire each customer?”.

19 responses

Post a comment

  1. Seth wrote on :

    This could also mean you’re advertising under the wrong keywords– keywords that people only search for when they already want Firefox, maybe? This may just mean you should start advertising under words like “spyware” for when people try to figure out how to get rid of it.

  2. Ben Karel wrote on ::

    Ken,

    I’m a little unsure of your analysis on the relative percentages.

    First, two explicit assumptions: turning off paid search should have no effect on “other” traffic, and should not decrease the originally-organic searchers.

    Okay, so for week A, we start with 7.4% of traffic coming from paid ads. If we turn off ads and assume that NONE of the people who would have clicked those ads will become organic referrals, we would expect the percentage of organic referrals to increase to 20.8%, and other traffic to be 79.2%. 19.3/(100-7.4) = .2084

    Of course, the actual number was more than that: 24.6 – 20.8 = 3.8%, and 3.8/7.4 = 51.3%, instead of 72%.

    If you look at the raw numbers, the increase appears to be almost exactly in between: (142-114)/44 = 63% of the paid searchers were converted to organic searchers.

  3. Ville wrote on :

    I love your pie charts. :) What do use to make ‘em?

    Btw. thanks for not using horrible “3d” pies that just distort the presentation.

  4. kkovash wrote on :

    Ben,

    thanks for the comments! while our analyses took on slightly different approaches (based on differing assumptions), I think our conclusions roughly match: approx. 2/3 of “paid searchers were converted to organic searchers.” this is how the $0, $0, $3 analogy was derived.

    Ken

  5. kkovash wrote on :

    Ville,

    thanks for the note. I used Excel 2007 for the charts.

    Ken

  6. David Naylor wrote on ::

    Yes, I’m with Ben here. The maths in the post were a bit iffy.

  7. Stephen wrote on :

    It seems to me that the total number of downloads when the ads were on vs. when they were off is the most significant data you have, not these charts. You only list the difference (11%) for the first week, but what about the second?

    Also, your conclusion seems warranted, but it is also obvious. I’m not sure what your marketing strategy is, but you probably aren’t advertising to make firefox more accessible to those who already know about it, you’re advertising to people who might be interested in trying it for the first time. So, it’s pretty clear that the cost of the ads shouldn’t be diluted by the total number of downloads, but by the increase your study has shown.

  8. timfry wrote on :

    Kevin,

    Were the ads only placed on google’s search page? What about google ads on blogs and such? Or in domain only searches? If ads are on the other two – you may have the answer. No, ads for Firefox on the search term Firefox on the google page don’t help but you do gain people on google ads on blogs.

    I’m assuming you limited the ads to the google search page but I didn’t see that posted anywhere.

  9. a wrote on :

    Fifth grade math does not an analysis make. While the approach is good, the mathematical/statistical analysis is pathetic. Hint: What is the probability that such a difference in distribution is random? How should the sample sizes be to be confident about these number?

    This is coffee table mathematics that good only for impressing people who do not have a background in sciences and how to collect and analyze data and draw conclusions. But its still much better than what most people do.

  10. Pingback from » Firefox Gets the Trial and Error Economy Economist Blog: Discussing the economy from A to Z on ::

    […] put out the Firefox browser (which is great) understand the Trial and Error economy. They recently wrote up a description of an experiment they did in marketing their download web site. (Hat tip to Steve Levitt at […]

  11. Pingback from 1800blogger - Blog Directory of Industry Leading Blogs on ::

    […] put out the Firefox browser (which is great) understand the Trial and Error economy. They recently wrote up a description of an experiment they did in marketing their download web site. (Hat tip to Steve Levitt at […]

  12. Pingback from What did our advertising experiment really tell us? < Blog of Metrics on ::

    […] Blog of Metrics When in doubt, sample it out… < Mozilla Online Advertising (Part 2) – Experiment Findings and Marginal Cost […]

  13. Pingback from What did our advertising experiment really tell us? · Get Latest Mozilla Firefox Browsers on ::

    […] did our advertising experiment really tell us? I did an admittedly poor job of framing my last post in its broader context. We set out to discover the true effect of paid search advertising on a […]

  14. Pingback from Omie James » Blog Archive » Firefox Gets the Trial and Error Economy on ::

    […] the Firefox browser (which is great) understand the Trial ahd Error economy.  They recently wrote up a description of an experiment they did in marketing their download web site. (Hat tip to Steve Levitt at […]

  15. Pingback from Another Mozilla Advertising Experiment - Bidding on Keywords < Blog of Metrics on ::

    […] previously looked at an experiment with our online marketing programs here, here, and here. The results revolved around the interaction of regular (organic) search with paid search […]

  16. gayathriherath wrote on ::

    Has anyone out there heard about WideCircles.com. It seems like a way better service then wasting money on PPC. Apparently they are using refering websites ( forums, blogs, wiki, etc. ) and have a viral word of mouth distributed approach to it. My friend told me he got around 100 visits from single post which cost him $0.40c. I am going to give them a try today . In case you are intrested here is it. http://widecircles.com?s=imt1

  17. crack wrote on :

    If someone searches and finds a page that then refers them to your downloads does that show up as a referral or organic search?

  18. Form ADV wrote on ::

    Apparently they are using refering websites ( forums, blogs, wiki, etc. ) and have a viral word of mouth distributed approach to it. My friend told me he got around 100 visits

  19. RIA Compliance wrote on ::

    I’m not sure what your marketing strategy is, but you probably aren’t advertising to make firefox more accessible to those who already know about it, you’re advertising to people who might be interested in trying it for the first time. So, it’s pretty clear that the cost of the ads shouldn’t be diluted by the total number of downloads,

Post Your Comment