Combining Vulnerabilities and Coverage

Ideally all code should be well tested, but we all know that competing priorities in software development don’t always make this possible. From a security perspective, it makes sense to focus on uncovered code that recently had one or more security problems to take advantage of this coverage inequity. One hypothesis is that such code suffers from structural problems, as such bugs with similar root causes might still be present.
As an example of this; we started by creating a list of files that were patched due to security problems and then manually checked their testing coverage. The most interesting files were picked and bugs were filed to increase their test coverage.

We created some publicly available scripts that are able to collect the data automatically. Of course since current security bug reports are not available to the public, you will only be able to use this on older time frames where bugs have been unhidden.

Check Coverage of your Fuzzer

If you are using your own tools to test Firefox, then it often makes sense to try a coverage build to see what your tool is actually hitting and how often. This not only gives you a binary result of code that is tested or not; it can also tell you if certain code is reached, but very rarely compared to the rest of the covered code. Since fuzzers can be complex software they can also contain bugs, and while not hitting code at all might still be somewhat observable without special tools, hitting code just very rarely is hard to see.
To try this out, just compile Firefox as described in the MDN article, run your tool for a while and take a look at the lcov results.

1. Test Nightly

We want bugs identified as early as possible. The nightly builds correspond directly to the mozilla-central HG repository, and always contain the newest features being prepared for release. They offer the best opportunity to test changes as early as possible. Our bounty program covers Nightly as well, so if you are specifically looking for security problems, Nightly is the place to be. Of course we would be very happy to get reports for all kinds of defects if you find them, not only security related, to be able to provide the best user experience. If you have test/steps to reproduce and you don’t want to invest further work into minimizing, no problem: Every such report is better than none and if our developers can somewhat reproduce the issue, then it’s very likely to get fixed.

2. Special Builds

Regular release builds are not very good for fuzzing. They lack some important features that debug builds have. For instance, debug builds have a variety of memory invalidation routines enabled, e.g. poisoning free’d memory after a garbage collect. This poisoning causes the browser to crash with clearly recognizable memory patterns, e.g. on use-after-free. The patterns include 0xdada, 0xdbdb, 0xa5a5 and of course 0xdeadbeef. Another good thing about debug builds are assertions. While all assertion failures indicate bugs, some types of assertions are especially likely to indicate the presence of security holes:

• Assertions containing the words “wrapper” and/or “compartment”
• Assertions with indications of “array out of bounds”
• The assertion “Some pres arena objects were not freed”
• Assertions in the JavaScript engine (js/src/)

You can find nightly debug builds here. Then scroll down and find the latest build that ends in “mozilla-central-debug”.
Another build type that can be helpful for fuzzing is the Address Sanitizer build (ASan). There’s a more detailed blog post about ASan, but in short it allows to detect a variety of memory safety violations that wouldn’t necessarily crash in normal builds, while still being very fast for testing. We currently provide unofficial nightly ASan builds for Linux (64 bit).

3. Using Debug/Internal Functions with an Addon

There are certain functions available in privileged context only that are very powerful for automated testing. Some examples are calling the garbage collector, enabling zealous garbage collection (will run the garbage collector very frequently), invoking the cycle collector or quitting Firefox (useful for test case reduction).
Fortunately, there is already an addon, written by Jesse Ruderman, which is publically available. The addon adds a new global object “fuzzPriv” available in content, which provides several functions. Of course, you should not install this addon in your regular browser, as it might expose APIs that are not safe for web content.

4. Communication/Logging

Especially when testing browsers, communication between the component running inside the browser, and the outside harness is important. When the fuzzer is running inside the browser (e.g. a fuzzer written in JS) and there is only an outside harness monitoring it, then communication from the fuzzer to the harness is usually helpful to log any actions the fuzzer takes, so they can be reproduced more easily.
If the fuzzer is outside the browser, and the part in the browser is only executing tests, then we usually need a way to get the test inside the browser. Of course this can be done over HTTP but it’s usually quite slow.
For these situations, there are two helpful techniques that one should be aware of:

• The window.dump method (Firefox only): This method is a debug method that can be enabled by setting the pref “browser.dom.window.dump.enabled” to true (when doing this in about:config, you might have to add the pref yourself). Once this pref is enabled, you can call the window.dump() method with a message as parameter and it will be sent to stderr where your external harness can observe it.
• Websockets: Websockets (see Wikipedia and MDN) provide a bidirectional communication channel based on a TCP connection with a special protocol inside. Using Websockets, your component inside the browser can connect to the component outside (either that component provides a websocket compatible server, e.g. using nodejs, or the component uses standard TCP and an intermediate proxy program like em-websocket-proxy is translating the connection). Once connected, the client can send and receive data easily, all the API is Javascript.

NOTE: The ADBFuzz mobile fuzzing framework uses Websockets to connect from the mobile device back to the host. It might be worth taking a look at the code in helloworld.js and websocklog.py (with em-websocket-proxy in the middle) if you plan to implement such a feature.

5. Multiple Instances and Proper Automation Settings

Using multiple profiles, it is possible to run many Firefox instances in parallel on the same host. You can specify the profile name on the command line using -no-remote -P name or -no-remote -profile dir.
Especially when running/deploying multiple instances, using the right set of preferences for each of them is important to prevent certain interactive behavior (e.g. safe-mode prompts when restarting). Take a look at the prefs.js file delivered with ADBFuzz. It contains some valuable options which can be directly added to the prefs.js file in your fuzzing profile.

6. Use Minidumps or Linux Core Dumps

Running Firefox under a debugger for fuzzing is not very efficient. Instead, you can use the minidumps that the Firefox crash reporter produces: If you set your environment variables to match

MOZ_CRASHREPORTER_NO_REPORT=1
MOZ_CRASHREPORTER_SHUTDOWN=1

then the crash reporter will just store a minidump in a subdirectory of the profile, without prompting for submitting it. Instead it will just exit afterwards. Using the minidump_stackwalk tool, you can obtain a stack trace from the dump for triage then. One advantage of this approach is that it works on all supported platforms. Another helpful fact is that the minidumps get stored in the respective profile directories, so running multiple instances does not cause problems.
Of course, you can also use regular Linux core dumps using the following environment variables:

MOZ_CRASHREPORTER_DISABLE=1
XRE_NO_WINDOWS_CRASH_DIALOG=1
XPCOM_DEBUG_BREAK=stack-and-abort

You still have to enable core dumps in the system (e.g. with ulimit -c unlimited). If you run multiple instances of Firefox, you can use the PID to map the core dump to one of the instances by writing “1″ to /proc/sys/kernel/core_uses_pid.

7. Reduce test cases automatically

When a fuzzer finds a problem, the test case is often very large and might even span multiple files. Reducing this manually is tedious and a waste of time if the same process can be automated. For crashes and assertions, automation turns out to be quite easy. One possibility is to use Lithium, a tool written by Jesse Ruderman which already comes with support for Firefox. Another possibility is to use delta or similar scripts.
The principle of operation for these tools is always the same: Remove some input, start the program with the new input, determine if the failure still occurs and if so, start again with removing input. If the test no longer fails, they revert the last step. The only difference is how they actually decide which input to remove. There is no “best” strategy here, as it highly depends on the input. Source code for example can have a lot of inter-line dependencies and behaves entirely different than just independent lines of input. I will soon publish two Perl scripts that specifically work well with (well-formatted) source code (while they can also be used on regular input), by focusing on removing blocks and line pairs.

Conclusion

Fuzzing browsers is a complex endeavor and I hope that one or more of the tips above will make your testing efforts on Firefox even more awesome. Of course if you have any additional tips or suggestions yourself, please let me know.

What’s wrong with the blocked version of Java?

With the most recent Java update, Oracle fixed quite a few security vulnerabilities (see advisory page), including a vulnerability listed as CVE-2012-0507. Up to this point, this isn’t really unusual, as most of these updates fix one or more security problems.

However, not even 6 weeks after the release of the Java update, the Microsoft Malware Protection Center received malware samples that exploit the specified vulnerability in a very reliable way and use it to install a well known trojan, called the ZeuS bot. There was now evidence that the vulnerability was not just theoretical, and had become a practical avenue to infect machines with malware. You can read the full blog post from Microsoft here (this is a technical post, so it’s likely only interesting to you if you have a technical background).

“I’m not getting attacked anyway”

Shortly after the Microsoft post, Brian Krebs published a blog post on the topic, stating that the exploit is now being integrated into well known exploit kits, e.g. the Blackhole exploit kit. These kits can be purchased on the black market and are usually integrated invisibly into hacked websites or indirectly served on websites through hacked content providers (imagine the number of vulnerable users you can reach when you break into an advertisement server and include your malware into banners served to other sites). With this step, the threat became more serious: Unless you just don’t browse the Internet at all, the risk of getting infected is very real. Of course there might be a minority of users that only ever browses a corporate intranet which will mitigate the risk, but this isn’t really the common use case for a web browser. If a user absolutely needs the older, vulnerable version of Java, they can still bypass the warning.

“I have a Mac, I’m safe”

While this might have been true at some point in the past, the threat landscape for Mac/OS X has changed quite a bit in the last few years. As the popularity of the Mac platform has grown so has its attactiveness as a target for attackers. Only a few days ago, F-Secure announced that the Flashback trojan, a well known piece of Mac malware, is now exploiting the very same vulnerability we’ve been discussing so far. You can read the post here to get all the technical details about it. According to recent reports, the number of infected Mac/OS X machines recently grew rapidly over half a million and is still growing. As such the threat to Mac users is evident and imminent, thus prompting our response on all platforms. Note that Apple has recently released a Java update as well that addresses this vulnerability.

Summary

In reviewing the actions and threats for this Java vulnerability, or, for that matter, any security issue, we take the balance of security vs. usability very seriously. This situation is an instance where we felt the balance had tipped towards taking a security action that has a minor adverse affect to the user community as a whole. The desired goal is to protect our users, to encourage them to upgrade to more recent, secure versions of plug-ins and to maintain our history of thoughtful security action.

Acknowledgements

Thanks to the various people involved in getting the proper blocks in place so quickly, that was an amazing job. Thanks also to Curtis Koenig, Dan Veditz and Ian Melven for reviewing/editing this post

Firefox 3.6.12 and 3.5.15 security updates now available
Thunderbird 3.1.6 and 3.0.10 security updates now available

Issue:
Mozilla is aware of a critical vulnerability affecting Firefox 3.5 and Firefox 3.6 users. We have received reports from several security research firms that exploit code leveraging this vulnerability has been detected in the wild.

Impact to users:
Users who visited an infected site could have been affected by the malware through the vulnerability. The trojan was initially reported as live on the Nobel Peace Prize site, and that specific site is now being blocked by Firefox’s built-in malware protection. However, the exploit code could still be live on other websites.

Status:
We have diagnosed the issue and are currently developing a fix, which will be pushed out to Firefox users as soon as the fix has been properly tested.

In the meantime, users can protect themselves by doing either of the following:

• Disabling JavaScript in Firefox
• Using the NoScript Add-on

Credit:
Morten Kråkvik of Telenor SOC

—
Brandon Sterne
Man-in-the-middle

For additional information please see Mozilla Foundation’s Security Advisory MFSA-10-08 as well as the Firefox 3.6.2 Release Notes. We urge users to promptly update to this release by selecting “Check for Updates…” from the “Help” menu, or by visiting https://www.mozilla.com/ for a free download.

Original blog entry follows below.

Two add-ons in the experimental section of addons.mozilla.org were found to be containing malware.  These were not originally detected with the anti-malware scanning tools that we have been using.  We have since increased the number of scanning tools, and will be taking additional steps to minimize the risk of further incidents.  Full details of the issue and recommended mitigation steps are here on the AMO blog:

http://blog.mozilla.org/addons/2010/02/04/please-read-security-issue-on-amo/

I’ve previously posted about the .NET Framework Assistant add-on that was delivered via Windows Update earlier this year. It’s recently surfaced that it has a serious security vulnerability, and Microsoft is recommending that all users disable the add-on.

Because of the difficulties some users have had entirely removing the add-on, and because of the severity of the risk it represents if not disabled, we contacted Microsoft today to indicate that we were looking to disable the extension and plugin for all users via our blocklisting mechanism. Microsoft agreed with the plan, and we put the blocklist entry live immediately. (Some users are already seeing it disabled, less than an hour after we added it!)

Update (Sunday Oct 18, 6:30pm PDT): Microsoft has now confirmed that the Framework Assistant add-on is not a vector for this attack, and we have removed the entry from the blocklist. We are also working on a mechanism to allow Firefox users to re-enable the WPF plugin ahead of its eventual removal from the blocklist. For more information, see Mike Shaver’s latest blog post.

The URL in the address bar can be spoofed when a new window or tab is opened by a malicious web page.

Impact to users

If a user visits a page hosting this malicious code, a new window or tab can be opened with a faked URL.  There is no way of determining if the URL is authentic.  This could result in the user disclosing confidential information to the malicious site, known as a phishing attack.

Status

This vulnerability is known to affect all current versions of Firefox.  Mozilla is actively working on fixing this vulnerability.  Users can mitigate this vulnerability by only sharing confidential information with websites that were opened from a bookmark, a trusted source, or by manually opening a new tab or window and entering a URL.

Credit

This issue was originally reported by Juan Pablo Lopez Yacubian.