Beware the Security Metric

Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia “2008 Report” ( It tries to break down vulnerabilities reported by browser, and specifically states:

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.

The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.

Lucas Adamski
Director of Security Engineering

29 comments on “Beware the Security Metric”

  1. Daniel Veditz wrote on

    @Idan: We do publish it, and above I got a roughly half-and-half count going through our public advisories. Any independent observer can go through the bugs linked from those advisories (or alternately, from bugzilla queries) and make their own count if they don’t believe me. The advisories have names attached, the bugs have names attached, it’s all public and published.

  2. Wilson Perdomo wrote on

    Other software vendors do not report all the bugs they have with their software unlike Firefox. I believe is unfair to compare Firefox with dishonest Micrsoft, Apple, and Opera.

  3. Mister Smith wrote on

    ClickJacking …. Firefox has an addon, NoScript, which was the only one able to stop the attempt. The rest allowed the clickjackers to exploit all the friends I know using Internet Explorer. Boon of business for me to fix these machines, but they all swear by Firefox now. These number do not fool me. Firefox finds a problem, lets me know what it is and has it fixed by the time i read about it. THANK you Mozilla …

    Mozilla is #1

  4. Patrick wrote on

    While I am no expert on coding anything. I do however repair/remove a ton of virus and other crap my customers get. My repeat customers are the ones who don’t listen to me about Firefox. They refuse to understand that you can’t just wait till your software vendor release’s their normal updates to fix security holes. Firefox, while not perfect has helped me to gain repeat customers, not by being a bad product but by doing so well, that when something else goes wrong i.e. hard drive fails, those customers return since they were happy with my advice.(Granted it’s not just me recommend Firefox, I recommend other open source programs, it also has to do with the quick service times, and speaking to them in language that they can understand)

    I tell all my customers not only do you need to use a good browser but you also have to use some common sense on the web. Sorry even those who stick with IE if they use their brains would have far fewer problems.

    I never expect Firefox or any other product to be perfect, however I am pleased with the response times that Firefox has, not to mention that it’s open source, which allows me to use a ton of add ons 🙂

    Any type of report is bound to be wrong one way or another. Just look at the way company’s are doing their polling. The post the numbers but often leave out how the question was worded.

    If I asked a 100 people if they liked scrambled eggs served COLD, and then reported that 99% of the people I polled do not like eggs, Would that be a lie?

  5. sly wrote on

    mozilla and firefox are awesome. Good response to the security report. I feel safe again. Seriously, keep up your great and hard work.

  6. Bill Pacos wrote on


    Nobody said that secunia only hates firefox, but secunia sure doesn’t try to even pretend to be impartial in their latest “report”.

    I’m very curious how you feel justified stating that a group that releases a “report” isn’t responsible for its accuracy. Are you saying that it’s ok to do sloppy work as long as it’s easy? I just did a study and found that 97% of the people that use the handle “stillwaiter” have below average intelligence – it’s a good thing I’m not responsible for the accuracy of those results, right? It’s just my job to report them!

    I might almost agree with you that it’s not secunia’s job to educate people about the meaning of the numbers, but any self-respecting data collector should at least have the integrity to highlight numbers that were collected in different ways when “comparing” them. As has been pointed out in the comments above, even just having an asterisk next to the number and a quick explanation in the chart would have at least been a step in the right direction. Explaining large variations in your data is not showing bias, it’s explaining large variations in your data. Since this data was collected in a different manner, it’s entirely biased not to disclose this. It’s a good thing real scientists can’t be this sloppy! Maybe next year Secunia should double its budget to TWO dollars to generate a useful report.

    Regarding the definition of “neutrality”, perhaps this might help since you can’t seem to find it on wikipedia:
    The first definition is “not taking part or giving assistance in a dispute or war between others”. Secunia releasing a report based on information it knows to be incorrect with misleading graphs fits well here. This does not fall into the category of “neutral” since it IS giving assistance to all non-firefox browsers in its report. Perhaps your English could use a bit of polishing too.

    Ari T is right. You can tell your employer they are wrong.

  7. question2 wrote on

    This was never answered:

    Braggin about Firefox’s short time to fix…

    Does that include flaws discovered by Mozilla, and therefore reported as having a TTF of 1 day?

    So if Mozilla does indeed disclose more internally discovered flaws than other vendors, then what is their TTF at Secunia? 1 day?

    If so, this might make Mozilla’s TTF claim bogus because the stats are deflated by artificially short TTF bugs.

  8. Daniel Veditz wrote on


    The reports that have measured “time to fix” have generally only counted externally-reported flaws. Vulnerabilities that were not announced until fixed were not counted, both internally-discovered (and possibly not reported at all depending on vendor) and reported by 3rd-parties following “reponsible disclosure”.

    To make this distinction clear such reports often use the term “window of exposure” or “days of risk”. And they’re cumulative rather than averaged: a whole mess of TTF 1 day bugs doesn’t make you look better, each one adds to the “window of exposure”.

  9. Tom Anderson wrote on

    Securia’s research proves that IE is definately safer than Firefox (joke). But it does point out that 93% of everyone patches IE. Only 84% of people keep Firefox updated, which shows that more people are having trouble keeping Firefox updated. The reason that 115 patched vulnerabilities in Firefox would be cause for alarm is that people aren’t patching it. Even if MSIE patched 200 undisclosed vulnerabilities, a much higher percentage of IE users (at least those users who ran the Secunia Software Inspector) are safe because their browser has been updated.

    And nobody is using those insecure plugins such as ActiveX, right? By putting the two graphs next to each other (browser vulnerabilities vs. plugin vulnerabilities) IMO Secunia did a good job of reporting.

    A key thing would be to remind people that Firefox is NOT secure, so it IS important to keep it updated. Also it’s important to make updating an easy process. And I note that Vista security prevents Firefox from checking for updates automatically, but WHY?

    This is kind of absurd. We should be able to CHECK for updates even if we can’t install those updates because we’re not running elevated privileges. This Secunia article should be a swift kick: Mozilla should get on the ball.

More comments: 1 2