When a Firefox User Reports a Broken Web Site

mozilla

6 responses

One of the ways Firefox users can easily and directly contribute to making the browser a better product is through our Help->Report Broken Web Site feature:

Picture 1

BrokenSiteBox

This allows users to let us know if they experience problems with Firefox on specific websites. Let’s take a high-level look at this “Broken Site Reporter” data over the past few years:

TotalReportsOverTime

These numbers are significant.  Each day, over a thousand Firefox users are showing us pain points they have experienced while surfing the web. This sort of feedback can be wildly impactful for internet users everywhere.

A few things to note:

  • The reporter started with Firefox 2.0.
  • The reporter was down for about 2.5 months (6/25/08 – 9/9/08)

We can see a BIG spike for the 3.0 launch in June ’08 and a less pronounced jump for the 3.5 release in June ’09. The reports are trending upwards as our daily usage grew over this period, but it would be more telling to look at a direct comparison between daily users and daily broken site reports:

AvgAUSandReports

This view of the data conveys some good news. Just looking at the chart, it seems our daily usage and daily broken site reports are diverging – meaning we are getting fewer reports per user on average. Here are two more views revealing this information:

AvgReportsAndAUS

The chart above shows two Ordinary Least Squares (OLS) regression lines tacked onto our original graph. The linear regression lines are clearly diverging – more evidence that we are getting fewer reports per user.

AvgReportsPer100kUsers

Since July 1st, 2007 our daily reports per 100,000 users figure has decreased from about 2.5 to 2 (a fall of 20%).

I think it is safe to conclude that daily usage is pulling away from daily broken site reports for two main reasons. The first is that the web has improved – websites users navigate to are more likely to function correctly in any browser. The second is that Firefox is a better product today than it was in July 2007. Either way we look at it, the major win is that Firefox users are seeing fewer and fewer broken websites.

In parts II and III of our Reporter analysis (coming soon!), we’ll be looking at the types of problems reported by users, where these problems occur, how problems with websites on Firefox have evolved over time, and what recommendations could be made for improved user experiences moving forward.

6 responses

  1. Toe wrote on :

    Are you really sure that conclusion is safe? One other possible explanation is that Firefox is simply attracting a greater proportion of ‘mainstream’/less geeky users who are less likely to file such reports.

    Granted, it’s probably a mix of all three, but I think it’s still important to consider what type of user files reports and how many of them there are.

  2. frankyfrank wrote on :

    Are there more stats, like most hosts, and a split into the different categories?

  3. ehergenrader wrote on :

    Toe:

    This is definitely a valid hypothesis. The only real counter-argument I can see is the VERY simple interface of the tool. It really requires no technical expertise to use, and I have confidence all users COULD use the tool. That said, it’s true the more technical users are probably more likely to know it is there.

    There really is no plausible way (that I can think of) to quantify this kind of relationship (any ideas?). This is one reason why I did not attempt any explicit statistical modeling, but I think from an economical point of view you’re exactly right – the decrease is most likely a mix of all three.

    Thanks for the post!

  4. ehergenrader wrote on :

    frankyfrank:

    Yes, and we’ll be looking at a lot of this in future posts 🙂 . If you have a specific idea about analysis you think could be interesting/important, definitely let us know and we can try to incorporate it in out Broke Site Reporter series of posts.

  5. nicole wrote on :

    Hey.

    I think that you can’t imply correlation = causation in the way that you are above, sadly. http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation

    I happen to agree that Firefox *has* gotten better, and frankly, I suspect it’s very possible your conclusions are close to truth. And yet you can’t assert them as truth, darn it!

    Also, have you read Clay Shirky’s work on how users and their online habits follow a power law distribution? Assuming Shirky’s law is right, the savvier experts are reporting more sites, and the less-savvy users report less. This would also support the assumption that your logic above is flawed. I’d recommend Shirky’s Here Comes Everybody as a place to read about it.

  6. David R wrote on :

    nicole: Yeah, you’re right, it’s possible that people filing fewer reports caused Firefox to get better, instead of the other way around!

    Wait, no, that makes absolutely no sense. So annoying when people trot out the “correlation != causation” in absolutely any discussion of statistics.