1. Test Nightly

We want bugs identified as early as possible. The nightly builds correspond directly to the mozilla-central HG repository, and always contain the newest features being prepared for release. They offer the best opportunity to test changes as early as possible. Our bounty program covers Nightly as well, so if you are specifically looking for security problems, Nightly is the place to be. Of course we would be very happy to get reports for all kinds of defects if you find them, not only security related, to be able to provide the best user experience. If you have test/steps to reproduce and you don’t want to invest further work into minimizing, no problem: Every such report is better than none and if our developers can somewhat reproduce the issue, then it’s very likely to get fixed.

2. Special Builds

Regular release builds are not very good for fuzzing. They lack some important features that debug builds have. For instance, debug builds have a variety of memory invalidation routines enabled, e.g. poisoning free’d memory after a garbage collect. This poisoning causes the browser to crash with clearly recognizable memory patterns, e.g. on use-after-free. The patterns include 0xdada, 0xdbdb, 0xa5a5 and of course 0xdeadbeef. Another good thing about debug builds are assertions. While all assertion failures indicate bugs, some types of assertions are especially likely to indicate the presence of security holes:

• Assertions containing the words “wrapper” and/or “compartment”
• Assertions with indications of “array out of bounds”
• The assertion “Some pres arena objects were not freed”
• Assertions in the JavaScript engine (js/src/)

You can find nightly debug builds here. Then scroll down and find the latest build that ends in “mozilla-central-debug”.
Another build type that can be helpful for fuzzing is the Address Sanitizer build (ASan). There’s a more detailed blog post about ASan, but in short it allows to detect a variety of memory safety violations that wouldn’t necessarily crash in normal builds, while still being very fast for testing. We currently provide unofficial nightly ASan builds for Linux (64 bit).

3. Using Debug/Internal Functions with an Addon

There are certain functions available in privileged context only that are very powerful for automated testing. Some examples are calling the garbage collector, enabling zealous garbage collection (will run the garbage collector very frequently), invoking the cycle collector or quitting Firefox (useful for test case reduction).
Fortunately, there is already an addon, written by Jesse Ruderman, which is publically available. The addon adds a new global object “fuzzPriv” available in content, which provides several functions. Of course, you should not install this addon in your regular browser, as it might expose APIs that are not safe for web content.

4. Communication/Logging

Especially when testing browsers, communication between the component running inside the browser, and the outside harness is important. When the fuzzer is running inside the browser (e.g. a fuzzer written in JS) and there is only an outside harness monitoring it, then communication from the fuzzer to the harness is usually helpful to log any actions the fuzzer takes, so they can be reproduced more easily.
If the fuzzer is outside the browser, and the part in the browser is only executing tests, then we usually need a way to get the test inside the browser. Of course this can be done over HTTP but it’s usually quite slow.
For these situations, there are two helpful techniques that one should be aware of:

• The window.dump method (Firefox only): This method is a debug method that can be enabled by setting the pref “browser.dom.window.dump.enabled” to true (when doing this in about:config, you might have to add the pref yourself). Once this pref is enabled, you can call the window.dump() method with a message as parameter and it will be sent to stderr where your external harness can observe it.
• Websockets: Websockets (see Wikipedia and MDN) provide a bidirectional communication channel based on a TCP connection with a special protocol inside. Using Websockets, your component inside the browser can connect to the component outside (either that component provides a websocket compatible server, e.g. using nodejs, or the component uses standard TCP and an intermediate proxy program like em-websocket-proxy is translating the connection). Once connected, the client can send and receive data easily, all the API is Javascript.

NOTE: The ADBFuzz mobile fuzzing framework uses Websockets to connect from the mobile device back to the host. It might be worth taking a look at the code in helloworld.js and websocklog.py (with em-websocket-proxy in the middle) if you plan to implement such a feature.

5. Multiple Instances and Proper Automation Settings

Using multiple profiles, it is possible to run many Firefox instances in parallel on the same host. You can specify the profile name on the command line using -no-remote -P name or -no-remote -profile dir.
Especially when running/deploying multiple instances, using the right set of preferences for each of them is important to prevent certain interactive behavior (e.g. safe-mode prompts when restarting). Take a look at the prefs.js file delivered with ADBFuzz. It contains some valuable options which can be directly added to the prefs.js file in your fuzzing profile.

6. Use Minidumps or Linux Core Dumps

Running Firefox under a debugger for fuzzing is not very efficient. Instead, you can use the minidumps that the Firefox crash reporter produces: If you set your environment variables to match

MOZ_CRASHREPORTER_NO_REPORT=1
MOZ_CRASHREPORTER_SHUTDOWN=1

then the crash reporter will just store a minidump in a subdirectory of the profile, without prompting for submitting it. Instead it will just exit afterwards. Using the minidump_stackwalk tool, you can obtain a stack trace from the dump for triage then. One advantage of this approach is that it works on all supported platforms. Another helpful fact is that the minidumps get stored in the respective profile directories, so running multiple instances does not cause problems.
Of course, you can also use regular Linux core dumps using the following environment variables:

MOZ_CRASHREPORTER_DISABLE=1
XRE_NO_WINDOWS_CRASH_DIALOG=1
XPCOM_DEBUG_BREAK=stack-and-abort

You still have to enable core dumps in the system (e.g. with ulimit -c unlimited). If you run multiple instances of Firefox, you can use the PID to map the core dump to one of the instances by writing “1″ to /proc/sys/kernel/core_uses_pid.

7. Reduce test cases automatically

When a fuzzer finds a problem, the test case is often very large and might even span multiple files. Reducing this manually is tedious and a waste of time if the same process can be automated. For crashes and assertions, automation turns out to be quite easy. One possibility is to use Lithium, a tool written by Jesse Ruderman which already comes with support for Firefox. Another possibility is to use delta or similar scripts.
The principle of operation for these tools is always the same: Remove some input, start the program with the new input, determine if the failure still occurs and if so, start again with removing input. If the test no longer fails, they revert the last step. The only difference is how they actually decide which input to remove. There is no “best” strategy here, as it highly depends on the input. Source code for example can have a lot of inter-line dependencies and behaves entirely different than just independent lines of input. I will soon publish two Perl scripts that specifically work well with (well-formatted) source code (while they can also be used on regular input), by focusing on removing blocks and line pairs.

Conclusion

Fuzzing browsers is a complex endeavor and I hope that one or more of the tips above will make your testing efforts on Firefox even more awesome. Of course if you have any additional tips or suggestions yourself, please let me know.

Fortunately, we discover most regressions before we ship, thanks in large part to security researchers whose patience gives us time to review and test each patch well. But sometimes security releases are delayed slightly because we notice a regression as we are getting ready to release. Worse, we sometimes discover regressions after shipping a release.

This post explores patterns among regressions and suggests changes that could help us catch them earlier.

Motivation

In six cases since Firefox 2, we decided regressions in minor updates were serious enough to rush a followup regression-fix update. For comparison, we shipped 38 other minor updates, mostly to fix security holes, since releasing Firefox 2 in October 2006.

It’s hard to see trends and patterns in a list of six bugs, so I looked at a larger set of bug reports: all bugs reports marked as regressions caused by security bug fixes. I focused on the 176 such bugs filed between December 2007 and January 2010.

Most of these regressions were fixed before release and many of them were minor. But understanding why they weren’t caught immediately can help prevent major incidents in the future.

Overall regression frequency

Regressions seem to be declining in frequency. This calls for cake!

I believe regressions are declining in frequency mostly because of improvements in automated testing. The number of automated tests increased fourfold since Firefox 3.0: every trunk checkin now results in 220,000 tests running on each platform. Regressions caught by automated tests tend to be fixed immediately, without even being filed in Bugzilla.

Further improvements to automating testing could reduce the number of regressions even more. I noticed four patterns in the bugs that suggest areas where automated testing could be improved: (1) web site breakage; (2) firefox extensions; (3) printing; and (4) document navigation.

Web site breakage

About 48 of the 176 bug reports involved specific web sites. The sites ranged from private intranet sites to the most popular sites on the web. When simply loading a site is enough to reproduce the bug, it may be easy to detect it through automated testing.

Crashes, hangs, and leaks are easy to identify in automated testing. Carsten Book and Bob Clary have set up automated loading of top sites to find these bugs.

Visual problems are trickier to identify in automated testing, because no algorithm can reliably determine when a web site “looks wrong”. But determining whether a site’s appearance has changed can be done automatically. Since security fixes do not intentionally change the appearance of web sites, a tool to detect web site appearance changes could catch these regressions.

In addition to improving automated testing, we should remove support for some regression-prone, Mozilla-specific web features: signed scripts, enablePrivilege, XUL, and XBL. (Eight of the regressions affected sites using these features). These features have proven to be not especially useful to web developers, hard to keep stable, and hard to make secure. XUL and XBL, in particular, have together been responsible for over a hundred vulnerabilities.

Firefox extensions

About 21 of the regressions involved Firefox extensions. One way we could detect these bugs is to run automated tests with extensions installed. We should run Firefox’s entire test suite with popular extensions, and we should run at least basic tests with every extension.

Some regressions affect a feature of an extension rather than a feature built into Firefox. To catch these, we would need to let extension developers provide tests for their extension’s features.

We can also let Firefox developers search the source code of Firefox extensions, so when developers change APIs, they can tell which extensions the changes are likely to affect. We’ve already started to do this.

Printing

Five of the regressions involved printing. Printing bugs frequently make it to release users in part because most nightly users do not print web pages frequently.

I suggested adding printing to topsite testing and running existing reftests in print mode.

Document navigation

Five of the regressions involve document navigation. I suspect document navigation is prone to security bugs, and prone to regressions when those security bugs are fixed, for three reasons:

First, it is a place where many web features interact. Loads can be triggered by users (back, reload, link clicks, bookmarks clicks, submitting a form) or scripts (setting location, location.replace, history.go). Loads trigger many events (unload, beforeunload, pagehide), and scripts running from those events can trigger additional navigation. Additional layers of complexity arise from restoration of form contents, frames, and in-page navigation between hashes.

Second, expectations are complicated. To prevent accidental or deliberate “traps”, the “Back” button has to skip over sequences of purely-script-triggered navigations, which are often difficult to distinguish from user-triggered navigations. At the same time, AJAX web applications have good reasons to want the “Back” button to only change the hash part of the URL rather than leave the page.

Third, document navigation semantics have evolved along with the web, rather than being designed in a holistic way. Browser developers tried to fix security problems and web compatibility problems as they were discovered. This resulted in messy expectations and even messier implementations.

I suggested to our resident exhaustive-testing expert that we try to create a specification and a corresponding set of tests for document navigation and related features. In addition, I wonder whether rewriting document navigation code could simplify it and make it less fragile.

Nightly and beta testing

Firefox nightly build testers found more regressions than beta and release users combined. This is a testament to the power of the Mozilla community, which includes over ten thousand Firefox nightly testers and an incredible bug triage team.

But it’s also worrying: it suggests that in cases when a security patch can only have a few days of nightly testing, rather than several weeks, a major regression could easily slip through unnoticed. A patch might get only a few days of nightly testing if a developer feels that landing a fix effectively discloses a vulnerability, or when we’re rushing to fix a security hole that has already been disclosed.

Automated tests will never be able to catch every regression, so we want to help nightly and beta testers identify bugs as effectively as they can, whether a patch undergoes testing for weeks or days. David Boswell has been improving the nightly start page to highlight the best ways to look for bugs. Developers can now post temporary notes on this page to highlight features that need special testing.

We’re also making it easier to be a nightly tester. Automated tests now keep the worst regressions out of nightlies, making nightlies less prone to breaking. Automatic nightly updates now work for all localizations, and we’re planning to make nightly updates smoother.

To expand beta testing, we’re thinking of adding a checkbox to update preferences to let users choose to become beta testers. Beta testers are not always as active at reporting bugs as nightly users, but their numbers allow clear trends to appear in crash statistics.

Raw data

Maybe you can spot patterns that I didn’t, or suggest other methods of automated testing that could catch these kinds of bugs earlier?

I focused on the symptoms and testcases of the regressions. Someone reading the patches might notice different patterns. Would more bugs have been caught by writing new custom static analyses, paying attention to new compiler warnings, or getting some testers to report assertion failures from running debug builds? Could some architectural change have prevented a class of regressions (or the security bugs whose fixes caused them)?

These links show the 176 regressions caused by security bug fixes between December 2007 and January 2010. The spreadsheet contains my guesses as to how we found each bug and how we could have found it earlier.

List of 176 bugs | Spreadsheet on Google Docs | Spreadsheet as CSV

Thanks to Murali Nandigama for helping with Bugzilla queries. Dan Veditz, Melissa Shapiro, and Drew Ruderman provided valuable feedback on drafts of this post.

Jesse Ruderman
Security bug hunter