Blog of Metrics

Summary: (Total Crashes)/(Active Daily Users) is a low resolution metric for crash trending. We can improve Firefox’s stability for more users if we understand the distribution of crashes.

Over the last few months, improving Firefox’s stability has become a top priority. And our early results are encouraging! Both our survey data and crash reporter data show a downward trend in crashes. We should be careful, however, not to read too much into this data.

Due to privacy concerns, we do not store user ids when collecting crash data. Accordingly, our crash data suffers from a number of limitations. For example, we can only determine the number of crashes per daily session, not per session length. As a result, the number of crashes per user will appear to rise if users browse more often per day. Changes to the crash reporter UI will similarly bias our data. A 5% increase in the reporter response rate will lead to a 5% increase in reported crashes.

Perhaps most importantly, the lack of a user id limits our ability to draw inferences about the distribution of our data. Why does this matter? If Firefox crashes are skewed, we may reduce overall crashes by 10%, but have 80% of users experience more crashes.

A quick look at the “Week in the Life of a Browser” study suggests that crashes are indeed highly skewed. By examining start-up events without corresponding shut-down events, Jono calculated the number of unexplained session interruptions per user. While there are other causes of session interruptions, such as a computer losing its power, we can reasonably assume that session interruptions serve as a (perhaps highly overstated) proxy for crashes.

Session interruptions per user does not take on the bell-curve shape of normally distributed data. Rather, it follows a power law distribution. 49% of users did not experience a single session interruption, while 70% experienced one or fewer. The mean number of session interruptions, however, was 1.4. If our crash data follows a similar distribution, the average crash per user metric tells us little about the experience of a typical Firefox user.

Anecdotal evidence supports this hypothesis. While we all know people who swear by Firefox’s stability, we also know people who complain of frequent failures. I, for one, haven’t experienced any crashes since upgrading to the 3.6 beta a few weeks ago.

With this in mind, I suggest we use Test Pilot to run a longitudinal study of true Firefox crashes. Because Test Pilot is opt-in and allows users to review their data before submitting it, we’re able to consider data at a more granular level. As with previous TP experiments, we will go to great lengths to respect the privacy of participants.

In addition to crash events and session length, I would like to collect data we can correlate with crashes. Firefox version, operating system, and Add-ons installed immediately come to mind.

Have suggestions for additional data that we should, or shouldn’t, collect? Please leave them in the comments.

Tuesday’s earthquake in Haiti devastated hundreds of thousands of lives and the aftershocks are felt physically and emotionally throughout the world.

The Mozilla community includes thousands of people in Haiti, and in our metrics, we can see the hole this disaster has left in our community.  The chart below shows Firefox usage in Haiti this week (it’s based on a once daily “ping” that we see from active users) and it’s broken down by hour (local time in Haiti).  This picture is not pretty and it suggests that much of the communications infrastructure within Haiti has been adversely affected.

Everyone in Haiti could use our thoughts and help, and we’d like to put out a personal plea to the Mozilla family to consider these members of our community and donate if you can.

(Thanks to Daniel Einspanjer for contributing to this post.)

Summary: In a survey of 966 visitors to the Update page, 47.8% open new empty tabs from the new tab button, 30.4% from the file menu, and 21.8% from a keyboard shortcut.

In addition to running A/B and multivariate tests, we have begun supplementing our quantitative analysis with more qualitative feedback. Historically, many of our most impactful insights have come from survey data.

With this in mind, we recently launched a usability survey on the Firefox Update page page. Our goal? Understand how users open new empty tabs.

The Update page is an ideal place to run this survey because nearly every Firefox user views the page once. Additionally, because this page receives such high traffic, we are able to expose the survey to ~1% of users and get statistically significant results within a week.

Of the 6000 visitors who saw the survey, 966 responded–resulting in response rate of 16.1%. The data breaks down as follows:

Our early results are encouraging, but we can make a number of improvements. First, we may reduce our response bias by randomizing the order in which the responses are presented. Second, we will have more representative data if we run the survey immediately after a dot release. This will be a factor if the population of users who update Firefox immediately and the population of users who update later are heterogeneous (a likely assumption). Third, if we collect a larger sample, we can segment our analysis by operating system. We can run this analysis now (using SiteSpect’s excellent web optimization tool), but our results won’t be statistically significant. And finally, we can elicit a higher response rate by testing a few different survey styles.

Update page surveys provide one additional benefit. By comparing our survey results with our Test Pilot data, we can roughly approximate the bias in the Test Pilot population. Both data sets are biased, but our survey data should be more representative because it has a lower barrier to entry and a much larger sample.

Next up? We want to determine what’s driving the unusual traffic on the First Run page. Ken already has a survey in the works.

Have a survey question that would help us build a better Firefox experience? Please leave it in comments!