Firefox & Page Load Speed – Part I

With our last experiment, we found that a simplified download page drove conversions up 2.3%. We hypothesized that much of this improvement was due to improved page load speed. Today, we’ll test this theory — answering not just whether speed matters, but also how much it matters.

Specifically, we want to know: for each second improvement in page load speed, by how much does our conversion rate improve?

Let’s begin by answering a simpler question: was there a difference in page load speed between visitors who downloaded Firefox and visitors who didn’t? We could compare average page load times, but that metric only tells part of the story and can be heavily skewed by outliers. Instead, let’s look at the distribution of page load speed of these two groups (1 = visitors who downloaded, 0 = visitors who did not download):

The difference is surprisingly large. Half of the downloaders loaded our landing page in under 2.10 seconds. For non-downloaders, that figure was 3.67 seconds — 75% slower. Furthermore, a quarter of non-downloaders waited over 7 seconds for our page to load!

Now that we’ve confirmed a difference exists, let’s estimate by how much page load speed improves conversions. To do so, I ran a simple logistic regression, controlling for the experimental variations.

We found that a 1 second increase in page load time decreases conversions by 2.7%! Assuming our model is correctly specified, we have massive room from improvement and can easily increase our conversion rate by 5 – 10%.

We must be careful, however, not to reach too far with our conclusions. There are two factors which complicate our analysis.

First, page load speed does not precisely measure what we care about. What’s important isn’t whether the page fully loads, but rather how quickly our main calls to action appear. Nonetheless, page load time serves as a strong proxy for visible content.

Second, our model suffers from omitted variable bias. We may imagine, for example, that visitors with faster Internet connections are naturally more inclined to download Firefox. In this case, it’s not page load speed that explains the higher conversion rate, but rather an unknown variable that’s correlated with both page load speed and download conversions.

Correlation is not the same thing as causation, but as Tufte said, it’s a darn good hint! And we can verify this hint by running a second experiment.

With the help of Ryan Doherty and Frederic Wenzel, we have already built an optimized version of the IE download page. By randomly assigning users to either this faster variation or the slower control, we will learn whether speed itself drove a higher conversion rate. Look for a discussion of our results later this week.

8 responses

  1. Steve Souders wrote on :

    Awesome stuff. I can’t wait to read the next post.

  2. Justin Dolske wrote on :

    Interesting. How is the page loadtime data generated? Is it just the delta between the request for page.html and image_at_bottom.png, or something like that?

  3. Robert Eisenbraun wrote on :

    Yes, it’s fun to watch you investigating your data! I’ll check for your update next week.

  4. Adam Fletcher wrote on :

    I agree, great post. Any chance we could get an RSS feed of the metrics blog? Maybe I’m dumb, but I can’t find an RSS link.

  5. Gerald Hartenhoff wrote on :

    I was very reluctant to download and use firefox. I was used to internet explorer. I joined aweber and to get a sign-up box for my website I had to use firefox to get it to work. So happy that it worked out that way otherwise I don’t think that I would have ever used firefox. It is quite a bit faster.

  6. Stoyan wrote on :

    This is great stuff, thanks for sharing!

    @Adam – the RSS feed URL:

  7. Brent wrote on :

    echoing Justin – Any chance you can describe your method for determining page load times for users?

  8. Alexsandro wrote on :

    Good observations. I’m waiting for part 2.