Beware the Security Metric

Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia “2008 Report” ( It tries to break down vulnerabilities reported by browser, and specifically states:

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.

The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.

Lucas Adamski
Director of Security Engineering

29 responses

  1. Sean Kerner wrote on :

    Remember of course that Secunia also showed Firefox to be faster to patch.

    There is nothing wrong with finding bugs – so long as they’re fixed fast right?

    [Lucas]: True, but the title of the very article you provide is propagating that same problem. Since we don’t know the total number of issues for other browsers (since only the tip of the iceberg is visible), you actually cannot determine who has the most bugs. All this does is encourage other vendors to continue hiding as much information as possible about the security of their products so they can be mistakenly perceived as being more secure, when in fact they are simply more opaque.

  2. Ian wrote on :

    Firefox Devs have been criticised in the past for making all their security issues public. This may not back up the accuracy of secunia’s research, but it makes your argument, that other vendors are encouraged to be secretive, a moot point, since they may very well intend to be secretive for the sake of user security.

    [Lucas]: Mozilla only makes security issues public once they have been fixed and the update has had time to propagate. Keeping details of fixed issues private doesn’t help user security; the bad guys already know how to reverse-engineer those fixes. That is why security researchers responsibly disclose their findings once a fix is public, so other developers can learn from those mistakes.

  3. leonore wrote on :

    I am using Firefox again only because Safary doesn’t work as well on Windows XP, last time I got a virus navigating in Firefox, I couldn’t fix the problem so I had to reinstall Windows, the virus was called Vundo!grb
    It was a really bad thing I couldn’t even connect to internet without having constant pop outs linked to explorer and firefox

  4. Ken Saunders wrote on :

    Yikes and yikes again!
    Judging by the headlines around the Internet, it’s almost as if people have been anxiously waiting for some dirt on Mozilla. I imagine that they have a collection of negative Mozilla related posts pre-written just ready to go. Since when did Mozilla become the bad guys anyway?

    Is this an attempt to bury the news of yet another great month for Firefox whereas Firefox’s market share is reported to be at 24% and 63% (a new low) for Internet Explorer?

    And of all things to report on incorrectly, Mozilla’s security.
    What a low blow.

    When promoting Firefox, Thunderbird, and Mozilla itself, I tell people that not only do I implicitly entrust my passwords, personal and other data to Mozilla and its products, so do their employees, the developers, and a few hundred other million people worldwide so take comfort in that.

    Is it that Secunia is not unbiased and neutral? Are they monetarily and politically motivated by a dude who happens to be one of the richest men on the planet? Or is that they just don’t care about the quality of what they produce and what their name goes on?

    I just don’t get it and it all pi**es me off because rebutting this and trying to stave off any further damages will not be easy when the momentum is already on the side of the idiots.

  5. stillwaiter wrote on :

    Secunia is NOT biased, and it IS neutral, just that its rules are outdated in the upcoming age of FOSS.

    You can’t really blame them, since they can only count the issues that’s available to them, so naturally they count more issues from the browsers that’s more open towards its own issues.

    I think we need to have a better security metrics system. Secunia was good, but nowadays it’s becoming less and less relevant. Just like VB100, which was a good metric for antivirus softwares, but is becoming less and less relevant nowadays.

  6. Ari T. wrote on :

    Of course Secunia is biased and not neutral. When they published the report, they knew that the numbers don’t give an accurate picture. They could have easily explained this, but they choose not to. Yes, you can blame them.

  7. Tgr wrote on :

    So what is the number of Firefox vulnerabilities reported by independent external partners?

  8. Daniel Veditz wrote on :

    Did a quick count of my own. For 2008 there were 69 Mozilla advisories and 87 individual CVE’s (only found 85 linked, but I assume the two Thunderbird advisories without CVE’s actually have them somewhere).

    Of those CVEs I found 44 from Mozilla developers and 42 from independent reporters (I missed one somewhere but this exercise was too tedious to go and recount).

    Ranked by severity there were
    external: 13 Critical, 8 High, 11 Moderate, 10 Low
    Mozilla: 31 Critical, 4 High, 5 Moderate, 4 Low

  9. stillwaiter wrote on :

    @Ari T.

    “Of course Secunia is biased and not neutral. When they published the report, they knew that the numbers don’t give an accurate picture. They could have easily explained this, but they choose not to. Yes, you can blame them.”

    Now you are being ridiculous. So Secunia must be hate Firefox and ONLY Firefox to be specifically biased against Firefox, but not any other browsers out there? They publish their report every year, following their own set rules according to their own (outdated) system, whether the numbers give an accurate picture or not, that’s not their responsibility.

    No you can’t blame them for doing their job, which is to collect the numbers and gave them to the public, it’s not their job to educate the public about the numbers, and they are not biased and they are neutral in doing their job of collecting and publishing the numbers. Thus of course Secunia is NOT biased and it IS neutral. IF they specifically explain things about Firefox like you have suggested, THEN they’d be biased and not neutral. So nope you can’t blame them, you can only try to educate the public yourself about those numbers, else it’s just meaningless whining.

    Ari T. you need to learn English better and know what “bias” and “neutrality” really means first

  10. Phil Agcaoili wrote on :


    I see your point, but the reality of reporting security incidents and vulnerabilities went in the the wrong direction a few years ago.

    By fully disclosing your security vulnerabilities in today’s environment, you expose yourselves to this inequity in the industry.

    The good, for those of us that know your policies we understand the obvious discrepancies in the Secunia report.

    Perhaps you should write them (stillwater perhaps) and ask them to add an Astrix to your data. Add something to the affect that public disclosure by Mozilla is based on Full Disclosure while other browser vulns rely only on publicly available disclosures.

    Maybe you should also add a flag to your vuln reports to level the playing field?

    Good luck,
    Phil Agcaoili

  11. Ken Saunders wrote on :

    “Mozilla Patches Fastest. NOT!”
    Jeff Jones, security strategy director at Microsoft (he actually gets paid).

    Brian Krebs, Washington Post’s technology reporter.
    “I hope readers will look past the sheer numbers of security holes that each browser maker fixed this past year, to the metric that in my opinion matters most: How long did it take each browser maker to address security flaws once those vendors knew about them?”

    I really enjoy reading Brian’s posts. Especially when he takes on Jeff Jones’s Mickey Mouse calculations and interpretations.

    Bimonthly (if that) security meeting at Microsoft.
    Jeff Jones:
    So we’re all in agreement on this right?
    Use Firefox until we fix these holes?
    MS security team member:
    Way ahead of you. I’m already using 3.2a1pre. I figured that it’ll be a while (if ever) before we can start using IE again.
    Jeff Jones:
    Ok, cool. Send in the strippers!!

    The above is a totally fictional account of a (snicker) Microsoft security meeting from a guy (me) hopped up on caffeine..
    It does not represent any publicly known facts, and it does not represent the views or opinions of the Mozilla Foundation or any of its employees (but they can LOL at it).

  12. Openminded wrote on :

    @Ken Saunders
    Jeff Jones says on page 2 of his article, “Secunia report specifically limited scope to vulnerabilities disclosed during 2008.”

    He lists six high/critical severity vulnerabilities in Firefox 2 that were publicly disclosed before 2008 but never fixed. 352 days of risk in 2008 before Firefox 2 went end of life, for each of those vulnerabilities.

    I’m not sure you read that far in Jeff’s article, but it’s a interesting statistic against Brian Krebs’ note that you quoted:

    “How long did it take each browser maker to address security flaws once those vendors knew about them?”

    Also, I like the fact Jeff quotes a variety of sources including this one. Strangely, Mozilla does not provide a link back to Jeff, which would faciliate a more open discussion IMHO.

    By the way, I find it easier to follow Jeff’s blog, he tends to go in more detail there and there’s less advertising noise.

  13. question wrote on :

    Braggin about Firefox’s short time to fix…

    Does that include flaws discovered by Mozilla, and therefore reported as having a TTF of 1 day?

  14. Natanael_L wrote on :

    @Ken Saunders: Now I’m offended! πŸ˜‰
    (I’m a geeky Mozilla fan Laughing Out Low, does that count? Hehehe…)

  15. AZZAM KROUMA wrote on :






  16. Jork wrote on :

    So, Opera has more security than Mozilla ?

  17. SadSac wrote on :


    That’s Google’s AdSense or an alternative, and has nothing to do with FireFox.

  18. Daniel Veditz wrote on :


    None of the “not fixed by end-of-life, high rated by NVD” issues Jones brought up were actual vulnerabilities. Some were fixed, but no advisory because they were unexploitable crashes.

    “I was able to identify this one as CVE-2008-4324…. Mozilla has
    not released a security advisory that mentions either the Secunia
    advisory or the vulnerability identifier.”

    No advisory because this was an unexploitable null deref (DoS) bug.
    NVD lists it as “MEDIUM” (which I think is too high) at

    “CVE-2007-1736, disclosed 3/28/2007, no MFSA after 631 days
    (352 in 2008) at product end-of-life”

    I don’t consider it a vulnerability, let alone “high” as rated at NVD
    (by whom?). The anti-malware feature in FF3 fixed this more as a
    side-effect than because we thought it was a vulnerability. (note: it _is_ important for malware; it isn’t for phishing since phishers have to get you to their site thinking it’s somewhere else, that is from a misleading link probably in mail. The list contains the links actually found in the wild)

    “CVE-2007-2162, disclosed 4/18/07, no MFSA after 610 days
    (352 in 2008) at product end-of-life”

    Can’t find much on this one. The references in the CVE are to a
    full-disclosure thread titled (wait for it) “Internet Explorer Crash”. It’s a DoS attack — /(.)*/.exec(“reallylongstring”) — that a couple of people said crashed but most reported just hung up their machine for a while. NIST rates it 7.8 HIGH (out of 10) anyway

    “CVE-2007-2671, disclosed 5/1/2007, no MFSA after 597 days
    (352 in 2008) at product end-of-life”

    A hang/DoS in the anti-phishing code

    “CVE-2007-3072, disclosed 6/4/2007, no MFSA after 563 days
    (352 in 2008) at product end-of-life”

    Using resource: for directory traversal, can load random .js files (the “OMG I can read Firefox default prefs” bug). We rated it sg:low, fixed in NIST rated it 7.1 HIGH.

    “CVE-2007-3073, disclosed 6/4/2007, no MFSA after 563 days
    (352 in 2008) at product end-of-life appears to be silently
    fixed in FF3.0 on 9/30/08–maybe in FF2, can’t tell”

    Same problem, but %2F on Mac/Linux instead of %5C on windows. Rated even higher by NIST: 7.8 — wha? Fixed in (MFSA 2008-44) When we wrote the advisory we assigned it CVE-2008-4067

    “CVE-2007-5896, disclosed 11/2/2007, no MFSA after 412 days
    (352 in 2008) at product end-of-life”

    Not fixed, not rated as a security bug by us, DoS at best. NIST rates it 7.1 HIGH.

    “Anybody remember this headline? Code execution vulnerability
    found in Firefox 3.0 | Zero Day | …
    CVE-2008-2786 was assigned to this vulnerability.”

    No, it wasn’t. The TippingPoint bug was CVE-2008-2785 (MFSA 2008-34) and fixed very quickly. CVE-2008-2786 was assigned to a full-disclosure mail by “hexpode” that consisted only of hashes. This was filed by hexpode as and was duped to a resolved-INVALID crash in Download Accelerator Plus.

  19. red wrote on :

    i know it

    firefox is the most secure browser

    115 vulnerabilities?!! are you kidding me?

    keep good working mozilla πŸ™‚

  20. Idan wrote on :

    IF so, why not publishing how many issues were found by your team and how many were found externally? Until you publish it, I simply don’t believe you. Me – switching back to IE8. Good and safe.

More comments:1 2