Jeff Jones, a director of security strategy at Microsoft published a report today about counting bugs. I blogged a few months ago about why I think counting bugs is less than useful:
Since all software has bugs, it’s more important to consider how long it takes to get a fix out when a security issue is discovered than it is to count bugs. Number of vulnerabilities identified is a function of how many bugs are present, but is probably more influenced by things like who is looking, and how good they are at finding security issues. That makes it a misleading metric.
When you compare how long it takes Microsoft to fix Internet Explorer vulnerabilities versus how long it takes Mozilla to fix vulnerabilities in Firefox it becomes clear why he chose to count vulnerabilities in this report instead. Earlier this year Brian Krebs of the Washington Post came up with this when comparing IE to Firefox:
For a total 284 days in 2006 (or more than nine months out of the year), exploit code for known, unpatched critical flaws in pre-IE7 versions of the browser was publicly available on the Internet…
In contrast, Internet Explorer’s closest competitor in terms of market share — Mozilla’s Firefox browser — experienced a single period lasting just nine days last year in which exploit code for a serious security hole was posted online before Mozilla shipped a patch to remedy the problem.
Mike Schroepfer goes into this in more detail in his post today.
One of the goals of the bug counting report is to demonstrate that Microsoft fixed fewer bugs for IE than Mozilla did for Firefox. Unfortunately for Microsoft (and for anyone trying to use this report as analysis of useful metrics) he does not count all the security issues. If he were able to count them all, Microsoft could get credit for all the bugs they fixed. He counts only the public issues, because that is all Microsoft will tell us about. Microsoft is worried that if it ever says it has fixed X security issues, the world will focus on that it had X vulnerabilities in the first place, not that they are now fixed and no longer a risk for users. So the set of issues that are available for public comparison is limited to the set of vulnerabilities that are reported externally AND fixed in security updates.
This is a small subset of all the vulnerabilities, because the vulnerabilities that are found through the QA process and the vulnerabilities that are found by the security folks they engage as contractors to perform penetration testing are fixed in service packs and major updates. For Microsoft this makes sense because these fixes get the benefit of a full test pass which is much more robust for a service pack or major release than it is for a security update.
Unfortunately for Microsoft’s users this means they have to wait sometimes a year or more to get the benefit of this work. That’s a lot of time for an attacker to identify the same issue and exploit it to hurt users. Sometimes it just takes time to put in a complicated fix. Anyone that has shipped a major piece of software can relate to that. But this is not the case for every internally found security issue. Extending this process to include fixes that are ready and just sitting on the tree waiting for the preferred vehicle to ship increases risk for users. But it sure keeps those bug count numbers down.
If we as an industry would just acknowledge that counting bugs is useless then vendors could feel safe talking about what they are doing to protect users. At Mozilla we fix our bugs openly. When you count Mozilla security bugs you are seeing not just those that are reported externally, but also the ones that would be considered internal if we acted like most other software vendors. Since all software has security vulnerabilities, we consider a vulnerability identified and fixed a win. It speaks to the strength of our community based security efforts to actively identify and quickly fix security issues. We don’t let fixes languish on the tree waiting for a major release while users are vulnerable. We ship fixes regularly because securing our users is more important than protecting our PR team from having to respond to articles about counting bugs instead of looking at the metrics that actually indicate whether a vendor is doing reasonable things to keep users secure.
We’re not building fixes for our PR team, we’re building them for our users. Go ahead and count.