How Mozilla finds crash bugs

Jesse Ruderman

6

This Tuesday (2009-07-21), I’m organizing a crash bug triage day where anyone interested can help us classify the swamp of open crash bugs. Join us in #bugday on irc.mozilla.org if you’d like to help.

Crashes and security

Some Firefox crash bugs are severe security bugs. A crash bug is likely to be exploitable if it can be triggered by a web page and the bug is a memory safety bug such as calling a virtual method on a dangling pointer.

Although only a fraction of our crashes are exploitable, two thirds of our most severe security bugs are crashes. We’re striving to improve how we find and fix crash bugs, since the better we can find and fix the bugs, the more stable and secure Firefox will be.

Bug reports

When a user reports a bug, a loose team of volunteers tries to reproduce the bug, improve the bug report, and inform the correct developers about the bug.

The history of bug 503286 demonstrates the variety of skills involved in fixing crash bugs. Soon after zbyte filed the bug, Ria Klaassen reproduced the bug on her machine, captured a stack trace, and determined when the bug had been introduced. Volunteers Lucas and Natch reduced the web page to a simple testcase that demonstrated the bug. Using this information, JavaScript engine developers Andreas Gal and Blake Kaplan had little trouble creating a patch. All of this happened within 14 hours of the bug being reported.

Our current bug triage process is not perfect, and sometimes leaves valid bugs unprocessed. During this Tuesday’s crash bug triage day, we will experiment with a new triage workflow that should be both more efficient and more effective. If you’d like to help clear the crash bug backlog, please join us in #bugday this Tuesday (2009-07-21).

Reporting bugs requires substantial effort from users, and requires both English-language and technical skills, so we prefer not to rely exclusively on user-reported bugs. Luckily, crashes are easier to detect automatically than most other types of bugs, so we can find many of them using other methods.

Crash reports

Whenever Firefox crashes, a dialog appears that allows users to submit information about the crash to Mozilla. Although the average Firefox user only sends us 1.5 crash reports per year, this information is valuable in aggregate. (Note added December 2009: the total number of crashes is likely higher, since users may choose not to submit crash reports, and for 90% of users the checkbox for submitting is unchecked by default.)

Currently, our main use for these crash reports is to identify the most common crashes. If a crash is common but has not been reported in Bugzilla, we can look at the comments that come with some crash reports to get an idea of what triggers the crash.

Fixing common crashes is enough to make Firefox stable for most users, but even a rare crash could be a security hole, so we need to do more. To address this issue, Mozilla is creating a “crash reproduction farm” of computers that will automatically load URLs that come with crash reports. This will let us identify and fix most crashes that result from simply loading a web page.

Fuzz testing

Fuzz testing is the art of creating random input for software in order to find bugs. We do extensive fuzz testing of our most complex components, such as the JavaScript engine. Fuzz testing finds two thirds of the exploitable crashes we fix, including many that are too obscure for normal web sites to trigger.

Fixing crash bugs

For developers who are familiar with the relevant code, crashes are often easier to debug and fix than they are to find. To identify the developers familiar with the code, you can click the source code links in crash reports and see who last touched each line of code.

Jesse Ruderman
Security bug hunter

6 responses

  1. Gc wrote on :

    “Whenever Firefox crashes” may be stating it a little too broadly. I’ve had maybe a dozen crashes (spontaneous exits?) over the past year, and every time there’s a dialog that says something like there was no dump to report. I’m starting to wonder if maybe the crash reporter doesn’t capture dumps for the builds or platform I’ve been using. (FF3 release builds and shiretoko nightlies on w2k.) The problems have not been reproducible so far (they don’t recur after restarting and reloading the tabs) so I haven’t bothered to investigate further. I haven’t found any information on what class of crashes are captured and what classes are not.

  2. Jesse Ruderman wrote on ::

    I’m aware of 3 cases where Breakpad fails to put together a crash report:

    * Bug 422308 – On Windows, with Flash 9.

    * Bug 503645 – On Linux, with Flash 10.

    * Bug 382124 – On Windows 2000, unless you go out of your way to download dbghelp.dll.

    You’re hitting the last one. I filed bug 505817 about making the dialog more helpful on Windows 2000.

  3. 番石榴 wrote on :

    Why blog.mozilla.org is in China?
    http://i31.tinypic.com/2wem5ae.jpg
    噢,我知道怎么回事了。

  4. Stanley A. Forwand wrote on ::

    I don’t understand any of this. However, I do know that since I upgraded to Firefox 3.5.2, it has crashed every time I tried to open it. I’ve sent in the “report” about 15 times over the past week but no answer. If I didn’t have Internet Explorer to fall back on, I’d be without email (gasp!)

  5. Daniel Veditz wrote on :

    @Stanley A. Forwand:

    Crashing after an upgrade means something installed on your system is incompatible with the new version. Since we try very hard to keep the standard public interfaces consistent from version to version that tends to suggest the incompatible thing is trying to hook into Firefox in a non-standard way, and that often means malicious software.

    If you upgraded from Firefox 3.0.x to 3.5.2 it might just be an add-on that hasn’t had a chance to update yet, in which case starting in “safe-mode” should prevent the crash and let you update the add-on. If it was the upgrade from 3.5.1 to 3.5.2 then I’m afraid it may be something unwanted. You can find information about “safe-mode” and help diagnosing the problem at http://support.mozilla.com

  6. Daniel Veditz wrote on :

    @番石榴:

    Our original content is served from California, but we do have data centers in Europe and China that host copies of our sites to give better performance to people in those regions.