Top Crashers By Url and MTBF

Austin King

5 responses

 

Sofa's Crash Reporter Icon via Alex Faaborg

Working with ss and chofman, we’ve created 2 new types of reports: a Top Crashers by Url and a Mean Time Before Failure (MTBF).

 

Given the current state of performance of the non-report parts of Socorro’s webapp, most of the thought and time have gone into the backend piece of these reports. You can read about the ReportDatabaseDesign on the project’s wiki.

Top Crashers by URL

On which websites do our browser builds crash the most? Which curses do our users hurl at us when this happens?

This report uses the optional url feild of a crash report to answer this question. It has two modes byurl and bydomain. You can read more about the details on TopCrashersByUrl. Crashes which have a comment, include the comment and a link to the actual crash report. Don’t worry, personal details have been removed, we don’t tie a specific user to a specific url.

We will be putting links into Socorro to these new reports, with the work neilio is doing, but for now here are various links.

We’ve enabled top crashers by URL for Firefox 3.0.5, 3.1b2, 3.1b3pre, and 3.0.6pre. Each of these link to “by domain” breakdowns, so 3.0.6pre has a link to this by domains view.

MTBF

Is this new release more crashy than previous releases?

Squeaking in before New Year’s Eve’s MFBT comes the MTBF report. It is a graph of the average number of seconds a release runs before crashing. Details are at MeanTimeBeforeFailure on the wiki.

We’re running MTBF reports for 14 releases:

Firefox major, milestone, and development releases.

Thunderbird milestone, and development releases. (No Milestone releases in Socorro yet)

Coming Soon: SeaMonkey

These reports are for a release in general as well as stats for Mac and Win, allowing for drilling down into OS. Several frontend enhancements to this report are coming.

, Product and versions in these reports include:

  • Firefox 3.0.4
  • Firefox 3.0.5  
  • Firefox 3.1a2
  • Firefox 3.1b1
  • Firefox 3.1b2 
  • Firefox 3.0.4pre
  • Firefox 3.0.5pre
  • Firefox 3.0.6pre
  • Firefox 3.1b3pre
  • Firefox 3.1b2pre
  • Thunderbird 3.0a3
  • Thunderbird 3.0b1
  • Thunderbird 3.0b1pre
  • Thunderbird 3.0b2pre

I’ve gotten a good dose of feedback on tweaks to make and bugs to fix, but hopefully you’ll find these new reports useful. Tomcat has already mentioned augmenting his list of urls to populate his test automation for 3.1 (using spider to test most popular urls) with the urls in these reports.

5 responses

  1. Justin Dolske wrote on :

    It would also be interesting to have a “top crashers by domain/url” that’s weighted by the site’s popularity. In a perfect (Big Brother) world this would be “crashes per visit”, but using something like comScore rankings might be a good proxy.

    This would allow looking for sites that are inherently crashy for us. In other words, the top 3 domains on the list (facebook, google, youtube) are there in large part just because they have so much traffic, not because Firefox is terribly crashy there. Well, probably. There’s a lot of JS on Gmail, so who knows? Maybe we are abnormally crashy, it’s hard to say either way with the current data.

    What’s with the spikes on some of the MTBF graphs? Surely we’re not putting out random builds that are amazingly stable? 🙂 Are these data glitches that can be filtered out, so the graphs don’t look like flat lines with a few spikes?

    A MTBF list that’s binned MTBF counts (10 users crashed after 30 minutes, 20 users after 60 minutes, etc) would also be interesting to see. I wouldn’t expect crashes to vary that much by day-of-release — it’s the same code.

    It would also be interesting to see if the crash reporter could include the number of page loads for the session (or something similar), to look at stable-but-idle vs stable-and-busy.

    Anyway, overall great stuff!

  2. DigDug wrote on :

    Forgive the rant, but someone at Mozilla needs to start putting real labels on their graph axises. Log or semi log plots are nice too. Either that, or scale the axis to get rid of the noise points which mean… well… nothing.

  3. bernd wrote on :

    http://crash-stats.mozilla.com/mtbf/of/Firefox/major

    when drilling down on OS Firefox 3.0.5 Mac shows 15E6 seconds of MTBF on the third day, however there are only 259 200 seconds elapsed in this time. This is probably just a script error like dividing by 0.

  4. aking wrote on :

    The frontend of MTBF got the least love in this release as most of the time was on the backend. Justin, we will be smoothing values over 10 day average, as well as refining start and stop dates for the window.

  5. Ian M wrote on :

    +1 to Justin Dolske’s suggestion.

    You could use stats like Alexa Rank, Google PageRank (page-based not site-based) or (US-only) Compete.com as well.

    Compete.com data you can download a dump file (with license restrictions but I imagine they’d give it free for the free publicity they’d get). There are backdoor ways of getting the other two stats, and probably other useful stats out there too.