Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia “2008 Report” (http://secunia.com/gfx/Secunia2008Report.pdf). It tries to break down vulnerabilities reported by browser, and specifically states:
31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security
Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.
From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.
So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.
Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.
Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.
The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.
Director of Security Engineering