There has been some interest in the last few days about a recent report from a company called Bit9 about application vulnerabilities. While we’re always happy to see stories that focus on educating our users about security, there are some problems with Bit9’s methodology that hinder its ability to draw any meaningful conclusions.
Bit9 says it drew up this list by identifying popular applications that have had a critical vulnerability reported in 2008. This is an ineffective test, as it rewards software companies that conceal their security vulnerabilities. Mozilla focuses a great deal of energy on building world class code, and we stand by our reputation on security; we don’t play games with it.
Mozilla security process involves regularly identifying, fixing, testing, and releasing security updates to keep our users safe, and we do that in a public way so that others can scrutinize our processes and help make them better. To suggest that this openness is a weakness because it means that we have “reported vulnerabilities” is to miss the reality: that software has bugs. A product’s responsiveness to those bugs and its ability to contain them quickly and effectively is a much more meaningful metric than counting them.
Bit9 seems to understand this in its focus on application support for updates, but again it fails to account for the real world experience. Firefox does not deliver WSUS updates, but our built-in update mechanism requires no user intervention, and we consistently see 90% adoption within six days of a new update being released.
The Firefox vulnerabilities Bit9 discusses are long-since fixed, with the majority of these fixes coming within days of it being announced. That is the real measure of application security: are known vulnerabilities fixed promptly, tested carefully, and deployed thoroughly? When people have asked that question, Firefox and Mozilla have consistently come out ahead.
Bug counting is unfortunately common because it’s easy, but it should not be a substitute for real security measurement. This is why we’ve continued to work on things like the Mozilla security metrics project, to help people make informed decisions about the security of their software. We invite people who are interested to be a part of that process.
Johnathan Nightingale
Human Shield