In the past half year I learned quite a lot about the different fuzzing approaches that security researchers and contributors use on Firefox. Although information on the subject should be public, a lot of it seems hard to find for people that are new in that area. Here are some tips for making your fuzzing on Firefox more successful, based on the problems that I encountered myself and mistakes that others made that I learned about.
1. Test Nightly
We want bugs identified as early as possible. The nightly builds correspond directly to the mozilla-central HG repository, and always contain the newest features being prepared for release. They offer the best opportunity to test changes as early as possible. Our bounty program covers Nightly as well, so if you are specifically looking for security problems, Nightly is the place to be. Of course we would be very happy to get reports for all kinds of defects if you find them, not only security related, to be able to provide the best user experience. If you have test/steps to reproduce and you don’t want to invest further work into minimizing, no problem: Every such report is better than none and if our developers can somewhat reproduce the issue, then it’s very likely to get fixed.
2. Special Builds
Regular release builds are not very good for fuzzing. They lack some important features that debug builds have. For instance, debug builds have a variety of memory invalidation routines enabled, e.g. poisoning free’d memory after a garbage collect. This poisoning causes the browser to crash with clearly recognizable memory patterns, e.g. on use-after-free. The patterns include 0xdada, 0xdbdb, 0xa5a5 and of course 0xdeadbeef. Another good thing about debug builds are assertions. While all assertion failures indicate bugs, some types of assertions are especially likely to indicate the presence of security holes:
- Assertions containing the words “wrapper” and/or “compartment”
- Assertions with indications of “array out of bounds”
- The assertion “Some pres arena objects were not freed”
- Assertions in the JavaScript engine (js/src/)
You can find nightly debug builds here. Then scroll down and find the latest build that ends in “mozilla-central-debug”.
Another build type that can be helpful for fuzzing is the Address Sanitizer build (ASan). There’s a more detailed blog post about ASan, but in short it allows to detect a variety of memory safety violations that wouldn’t necessarily crash in normal builds, while still being very fast for testing. We currently provide unofficial nightly ASan builds for Linux (64 bit).
3. Using Debug/Internal Functions with an Addon
There are certain functions available in privileged context only that are very powerful for automated testing. Some examples are calling the garbage collector, enabling zealous garbage collection (will run the garbage collector very frequently), invoking the cycle collector or quitting Firefox (useful for test case reduction).
Fortunately, there is already an addon, written by Jesse Ruderman, which is publically available. The addon adds a new global object “fuzzPriv” available in content, which provides several functions. Of course, you should not install this addon in your regular browser, as it might expose APIs that are not safe for web content.
4. Communication/Logging
Especially when testing browsers, communication between the component running inside the browser, and the outside harness is important. When the fuzzer is running inside the browser (e.g. a fuzzer written in JS) and there is only an outside harness monitoring it, then communication from the fuzzer to the harness is usually helpful to log any actions the fuzzer takes, so they can be reproduced more easily.
If the fuzzer is outside the browser, and the part in the browser is only executing tests, then we usually need a way to get the test inside the browser. Of course this can be done over HTTP but it’s usually quite slow.
For these situations, there are two helpful techniques that one should be aware of:
- The window.dump method (Firefox only): This method is a debug method that can be enabled by setting the pref “browser.dom.window.dump.enabled” to true (when doing this in about:config, you might have to add the pref yourself). Once this pref is enabled, you can call the window.dump() method with a message as parameter and it will be sent to stderr where your external harness can observe it.
- Websockets: Websockets (see Wikipedia and MDN) provide a bidirectional communication channel based on a TCP connection with a special protocol inside. Using Websockets, your component inside the browser can connect to the component outside (either that component provides a websocket compatible server, e.g. using nodejs, or the component uses standard TCP and an intermediate proxy program like em-websocket-proxy is translating the connection). Once connected, the client can send and receive data easily, all the API is Javascript.
NOTE: The ADBFuzz mobile fuzzing framework uses Websockets to connect from the mobile device back to the host. It might be worth taking a look at the code in helloworld.js and websocklog.py (with em-websocket-proxy in the middle) if you plan to implement such a feature.
5. Multiple Instances and Proper Automation Settings
Using multiple profiles, it is possible to run many Firefox instances in parallel on the same host. You can specify the profile name on the command line using -no-remote -P name or -no-remote -profile dir.
Especially when running/deploying multiple instances, using the right set of preferences for each of them is important to prevent certain interactive behavior (e.g. safe-mode prompts when restarting). Take a look at the prefs.js file delivered with ADBFuzz. It contains some valuable options which can be directly added to the prefs.js file in your fuzzing profile.
6. Use Minidumps or Linux Core Dumps
Running Firefox under a debugger for fuzzing is not very efficient. Instead, you can use the minidumps that the Firefox crash reporter produces: If you set your environment variables to match
MOZ_CRASHREPORTER_NO_REPORT=1
MOZ_CRASHREPORTER_SHUTDOWN=1
then the crash reporter will just store a minidump in a subdirectory of the profile, without prompting for submitting it. Instead it will just exit afterwards. Using the minidump_stackwalk tool, you can obtain a stack trace from the dump for triage then. One advantage of this approach is that it works on all supported platforms. Another helpful fact is that the minidumps get stored in the respective profile directories, so running multiple instances does not cause problems.
Of course, you can also use regular Linux core dumps using the following environment variables:
MOZ_CRASHREPORTER_DISABLE=1
XRE_NO_WINDOWS_CRASH_DIALOG=1
XPCOM_DEBUG_BREAK=stack-and-abort
You still have to enable core dumps in the system (e.g. with ulimit -c unlimited). If you run multiple instances of Firefox, you can use the PID to map the core dump to one of the instances by writing “1” to /proc/sys/kernel/core_uses_pid.
7. Reduce test cases automatically
When a fuzzer finds a problem, the test case is often very large and might even span multiple files. Reducing this manually is tedious and a waste of time if the same process can be automated. For crashes and assertions, automation turns out to be quite easy. One possibility is to use Lithium, a tool written by Jesse Ruderman which already comes with support for Firefox. Another possibility is to use delta or similar scripts.
The principle of operation for these tools is always the same: Remove some input, start the program with the new input, determine if the failure still occurs and if so, start again with removing input. If the test no longer fails, they revert the last step. The only difference is how they actually decide which input to remove. There is no “best” strategy here, as it highly depends on the input. Source code for example can have a lot of inter-line dependencies and behaves entirely different than just independent lines of input. I will soon publish two Perl scripts that specifically work well with (well-formatted) source code (while they can also be used on regular input), by focusing on removing blocks and line pairs.
Conclusion
Fuzzing browsers is a complex endeavor and I hope that one or more of the tips above will make your testing efforts on Firefox even more awesome. Of course if you have any additional tips or suggestions yourself, please let me know.