We recently started measuring C/C++ code coverage on mozilla-central again and documented the various efforts around it in a new MDN article. This article describes why coverage measurements are useful, how to create the necessary builds and also provides a link to the most recent coverage measurements of Mozilla’s automated test suites. This data can also be used to improve the security of Gecko based products, Firefox included. Below we list some approaches to that you may want to consider when testing Mozilla software with automated tools.
Combining Vulnerabilities and Coverage
Ideally all code should be well tested, but we all know that competing priorities in software development don’t always make this possible. From a security perspective, it makes sense to focus on uncovered code that recently had one or more security problems to take advantage of this coverage inequity. One hypothesis is that such code suffers from structural problems, as such bugs with similar root causes might still be present.
As an example of this; we started by creating a list of files that were patched due to security problems and then manually checked their testing coverage. The most interesting files were picked and bugs were filed to increase their test coverage.
We created some publicly available scripts that are able to collect the data automatically. Of course since current security bug reports are not available to the public, you will only be able to use this on older time frames where bugs have been unhidden.
Check Coverage of your Fuzzer
If you are using your own tools to test Firefox, then it often makes sense to try a coverage build to see what your tool is actually hitting and how often. This not only gives you a binary result of code that is tested or not; it can also tell you if certain code is reached, but very rarely compared to the rest of the covered code. Since fuzzers can be complex software they can also contain bugs, and while not hitting code at all might still be somewhat observable without special tools, hitting code just very rarely is hard to see.
To try this out, just compile Firefox as described in the MDN article, run your tool for a while and take a look at the lcov results.