Beware the Security Metric

Lucas Adamski

29

Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia “2008 Report” (http://secunia.com/gfx/Secunia2008Report.pdf). It tries to break down vulnerabilities reported by browser, and specifically states:

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security
Bulletins.

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.

The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.

Lucas Adamski
Director of Security Engineering

29 responses

  1. Sean Kerner wrote on ::

    Remember of course that Secunia also showed Firefox to be faster to patch.

    There is nothing wrong with finding bugs – so long as they’re fixed fast right?

    http://blog.internetnews.com/skerner/2009/03/firefox-has-the-most-bugs-and.html

    [Lucas]: True, but the title of the very article you provide is propagating that same problem. Since we don’t know the total number of issues for other browsers (since only the tip of the iceberg is visible), you actually cannot determine who has the most bugs. All this does is encourage other vendors to continue hiding as much information as possible about the security of their products so they can be mistakenly perceived as being more secure, when in fact they are simply more opaque.

  2. Ian wrote on :

    Firefox Devs have been criticised in the past for making all their security issues public. This may not back up the accuracy of secunia’s research, but it makes your argument, that other vendors are encouraged to be secretive, a moot point, since they may very well intend to be secretive for the sake of user security.

    [Lucas]: Mozilla only makes security issues public once they have been fixed and the update has had time to propagate. Keeping details of fixed issues private doesn’t help user security; the bad guys already know how to reverse-engineer those fixes. That is why security researchers responsibly disclose their findings once a fix is public, so other developers can learn from those mistakes.

  3. leonore wrote on :

    I am using Firefox again only because Safary doesn’t work as well on Windows XP, last time I got a virus navigating in Firefox, I couldn’t fix the problem so I had to reinstall Windows, the virus was called Vundo!grb
    It was a really bad thing I couldn’t even connect to internet without having constant pop outs linked to explorer and firefox

  4. Ken Saunders wrote on ::

    Yikes and yikes again!
    Judging by the headlines around the Internet, it’s almost as if people have been anxiously waiting for some dirt on Mozilla. I imagine that they have a collection of negative Mozilla related posts pre-written just ready to go. Since when did Mozilla become the bad guys anyway?

    Is this an attempt to bury the news of yet another great month for Firefox whereas Firefox’s market share is reported to be at 24% and 63% (a new low) for Internet Explorer?

    And of all things to report on incorrectly, Mozilla’s security.
    What a low blow.

    When promoting Firefox, Thunderbird, and Mozilla itself, I tell people that not only do I implicitly entrust my passwords, personal and other data to Mozilla and its products, so do their employees, the developers, and a few hundred other million people worldwide so take comfort in that.

    Is it that Secunia is not unbiased and neutral? Are they monetarily and politically motivated by a dude who happens to be one of the richest men on the planet? Or is that they just don’t care about the quality of what they produce and what their name goes on?

    I just don’t get it and it all pi**es me off because rebutting this and trying to stave off any further damages will not be easy when the momentum is already on the side of the idiots.

  5. stillwaiter wrote on :

    Secunia is NOT biased, and it IS neutral, just that its rules are outdated in the upcoming age of FOSS.

    You can’t really blame them, since they can only count the issues that’s available to them, so naturally they count more issues from the browsers that’s more open towards its own issues.

    I think we need to have a better security metrics system. Secunia was good, but nowadays it’s becoming less and less relevant. Just like VB100, which was a good metric for antivirus softwares, but is becoming less and less relevant nowadays.

  6. Ari T. wrote on :

    Of course Secunia is biased and not neutral. When they published the report, they knew that the numbers don’t give an accurate picture. They could have easily explained this, but they choose not to. Yes, you can blame them.

  7. Tgr wrote on :

    So what is the number of Firefox vulnerabilities reported by independent external partners?

  8. Daniel Veditz wrote on :

    Did a quick count of my own. For 2008 there were 69 Mozilla advisories and 87 individual CVE’s (only found 85 linked, but I assume the two Thunderbird advisories without CVE’s actually have them somewhere).

    Of those CVEs I found 44 from Mozilla developers and 42 from independent reporters (I missed one somewhere but this exercise was too tedious to go and recount).

    Ranked by severity there were
    external: 13 Critical, 8 High, 11 Moderate, 10 Low
    Mozilla: 31 Critical, 4 High, 5 Moderate, 4 Low

  9. stillwaiter wrote on :

    @Ari T.

    “Of course Secunia is biased and not neutral. When they published the report, they knew that the numbers don’t give an accurate picture. They could have easily explained this, but they choose not to. Yes, you can blame them.”

    Now you are being ridiculous. So Secunia must be hate Firefox and ONLY Firefox to be specifically biased against Firefox, but not any other browsers out there? They publish their report every year, following their own set rules according to their own (outdated) system, whether the numbers give an accurate picture or not, that’s not their responsibility.

    No you can’t blame them for doing their job, which is to collect the numbers and gave them to the public, it’s not their job to educate the public about the numbers, and they are not biased and they are neutral in doing their job of collecting and publishing the numbers. Thus of course Secunia is NOT biased and it IS neutral. IF they specifically explain things about Firefox like you have suggested, THEN they’d be biased and not neutral. So nope you can’t blame them, you can only try to educate the public yourself about those numbers, else it’s just meaningless whining.

    Ari T. you need to learn English better and know what “bias” and “neutrality” really means first

    http://en.wikipedia.org/wiki/Bias
    http://en.wikipedia.org/wiki/Objectivity_(journalism)

  10. Phil Agcaoili wrote on :

    Lucas,

    I see your point, but the reality of reporting security incidents and vulnerabilities went in the the wrong direction a few years ago.

    By fully disclosing your security vulnerabilities in today’s environment, you expose yourselves to this inequity in the industry.

    The good, for those of us that know your policies we understand the obvious discrepancies in the Secunia report.

    Perhaps you should write them (stillwater perhaps) and ask them to add an Astrix to your data. Add something to the affect that public disclosure by Mozilla is based on Full Disclosure while other browser vulns rely only on publicly available disclosures.

    Maybe you should also add a flag to your vuln reports to level the playing field?

    Good luck,
    Phil Agcaoili

  11. Ken Saunders wrote on ::

    “Mozilla Patches Fastest. NOT!”
    Jeff Jones, security strategy director at Microsoft (he actually gets paid).

    http://xrl.in/1qna
    or
    http://www.computerworld.com/action/article.do?command=viewArticleBasic&taxonomyName=Security+Hardware+and+Software&articleId=9129270&taxonomyId=145&pageNumber=1
    –
    Brian Krebs, Washington Post’s technology reporter.
    “I hope readers will look past the sheer numbers of security holes that each browser maker fixed this past year, to the metric that in my opinion matters most: How long did it take each browser maker to address security flaws once those vendors knew about them?”
    http://voices.washingtonpost.com/securityfix/2009/03/fanning_the_flames_of_the_brow.html
    –
    I really enjoy reading Brian’s posts. Especially when he takes on Jeff Jones’s Mickey Mouse calculations and interpretations.
    http://voices.washingtonpost.com/securityfix/2009/01/blogfight_the_truth_about_ie_v.html
    –
    Bimonthly (if that) security meeting at Microsoft.
    Jeff Jones:
    So we’re all in agreement on this right?
    Use Firefox until we fix these holes?
    MS security team member:
    Way ahead of you. I’m already using 3.2a1pre. I figured that it’ll be a while (if ever) before we can start using IE again.
    Jeff Jones:
    Ok, cool. Send in the strippers!!
    –
    - DISCLAIMER -
    The above is a totally fictional account of a (snicker) Microsoft security meeting from a guy (me) hopped up on caffeine..
    It does not represent any publicly known facts, and it does not represent the views or opinions of the Mozilla Foundation or any of its employees (but they can LOL at it).

  12. Openminded wrote on ::

    @Ken Saunders
    Jeff Jones says on page 2 of his article, “Secunia report specifically limited scope to vulnerabilities disclosed during 2008.”

    He lists six high/critical severity vulnerabilities in Firefox 2 that were publicly disclosed before 2008 but never fixed. 352 days of risk in 2008 before Firefox 2 went end of life, for each of those vulnerabilities.

    I’m not sure you read that far in Jeff’s article, but it’s a interesting statistic against Brian Krebs’ note that you quoted:

    “How long did it take each browser maker to address security flaws once those vendors knew about them?”

    Also, I like the fact Jeff quotes a variety of sources including this one. Strangely, Mozilla does not provide a link back to Jeff, which would faciliate a more open discussion IMHO.

    By the way, I find it easier to follow Jeff’s blog, he tends to go in more detail there and there’s less advertising noise.

    http://blogs.technet.com/security/archive/2009/03/09/supplemental-data-for-calculating-mozilla-patching-speed.aspx

  13. question wrote on :

    Braggin about Firefox’s short time to fix…

    Does that include flaws discovered by Mozilla, and therefore reported as having a TTF of 1 day?

  14. Natanael_L wrote on ::

    @Ken Saunders: Now I’m offended! ;-)
    (I’m a geeky Mozilla fan Laughing Out Low, does that count? Hehehe…)

  15. AZZAM KROUMA wrote on :

    DESPITE USING fIREFOX 3, I NOTICED THAT IS NOT PERFECTLY PROTECTED FROM ATTACKERS. ACTUALLY, AFTER ANY SEARCH BY FIREFOX3, MANY ATTAKERS SEND TO ME THEIR ADDS THAT MATCHES MY REQUEST. THAT IS MEAN THAT YOUR SECURITY ISN’T OF HIGH LEVEL OF PROTECTION.

    YOUR ARTICLE ABOUT SECURE IS NOT AS IT IS.

    THEREFORE, YOU ARE KINDLY REQUESTED TO TAKE INTO YOUR CONSIDERATION THAT SOME UNSECURED OPENINGS ARE STILL EXIST IN YOUR PROGRAM.

    ANYWAY, THANK YOU FOR YOUR KEEN ENDEAVORS, WHICH ARE PUT UP IN REACHING TO THE OPTIMAL RESULTS.

    Dr. AZZAM KROUMA

  16. Jork wrote on :

    So, Opera has more security than Mozilla ?

  17. SadSac wrote on :

    @Dr. AZZAM KROUMA

    That’s Google’s AdSense or an alternative, and has nothing to do with FireFox.

  18. Daniel Veditz wrote on :

    @Openminded:

    None of the “not fixed by end-of-life, high rated by NVD” issues Jones brought up were actual vulnerabilities. Some were fixed, but no advisory because they were unexploitable crashes.

    “I was able to identify this one as CVE-2008-4324…. Mozilla has
    not released a security advisory that mentions either the Secunia
    advisory or the vulnerability identifier.”

    No advisory because this was an unexploitable null deref (DoS) bug.
    https://bugzilla.mozilla.org/show_bug.cgi?id=457543
    NVD lists it as “MEDIUM” (which I think is too high) at
    http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2008-4324

    “CVE-2007-1736, disclosed 3/28/2007, no MFSA after 631 days
    (352 in 2008) at product end-of-life”

    http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2007-1736

    I don’t consider it a vulnerability, let alone “high” as rated at NVD
    (by whom?). The anti-malware feature in FF3 fixed this more as a
    side-effect than because we thought it was a vulnerability.
    https://bugzilla.mozilla.org/show_bug.cgi?id=427364 (note: it _is_ important for malware; it isn’t for phishing since phishers have to get you to their site thinking it’s somewhere else, that is from a misleading link probably in mail. The list contains the links actually found in the wild)

    “CVE-2007-2162, disclosed 4/18/07, no MFSA after 610 days
    (352 in 2008) at product end-of-life”

    Can’t find much on this one. The references in the CVE are to a
    full-disclosure thread titled (wait for it) “Internet Explorer Crash”. It’s a DoS attack — /(.)*/.exec(“reallylongstring”) — that a couple of people said crashed but most reported just hung up their machine for a while. NIST rates it 7.8 HIGH (out of 10) anyway
    http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2007-2162

    “CVE-2007-2671, disclosed 5/1/2007, no MFSA after 597 days
    (352 in 2008) at product end-of-life”

    http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2007-2671

    A hang/DoS in the anti-phishing code
    https://bugzilla.mozilla.org/show_bug.cgi?id=379390

    “CVE-2007-3072, disclosed 6/4/2007, no MFSA after 563 days
    (352 in 2008) at product end-of-life”

    Using resource: for directory traversal, can load random .js files (the “OMG I can read Firefox default prefs” bug). We rated it sg:low, fixed in 2.0.0.12. NIST rated it 7.1 HIGH.
    https://bugzilla.mozilla.org/show_bug.cgi?id=367428

    “CVE-2007-3073, disclosed 6/4/2007, no MFSA after 563 days
    (352 in 2008) at product end-of-life appears to be silently
    fixed in FF3.0 on 9/30/08–maybe in FF2, can’t tell”

    Same problem, but %2F on Mac/Linux instead of %5C on windows. Rated even higher by NIST: 7.8 — wha? Fixed in 2.0.0.17 (MFSA 2008-44) When we wrote the advisory we assigned it CVE-2008-4067
    https://bugzilla.mozilla.org/show_bug.cgi?id=380994

    “CVE-2007-5896, disclosed 11/2/2007, no MFSA after 412 days
    (352 in 2008) at product end-of-life”

    Not fixed, not rated as a security bug by us, DoS at best. NIST rates it 7.1 HIGH. https://bugzilla.mozilla.org/show_bug.cgi?id=403746

    “Anybody remember this headline? Code execution vulnerability
    found in Firefox 3.0 | Zero Day | ZDNet.com …
    CVE-2008-2786 was assigned to this vulnerability.”

    No, it wasn’t. The TippingPoint bug was CVE-2008-2785 (MFSA 2008-34) and fixed very quickly. CVE-2008-2786 was assigned to a full-disclosure mail by “hexpode” that consisted only of hashes. This was filed by hexpode as https://bugzilla.mozilla.org/show_bug.cgi?id=439800 and was duped to a resolved-INVALID crash in Download Accelerator Plus.

  19. red wrote on ::

    i know it

    firefox is the most secure browser

    115 vulnerabilities?!! are you kidding me?

    keep good working mozilla :)

  20. Idan wrote on :

    IF so, why not publishing how many issues were found by your team and how many were found externally? Until you publish it, I simply don’t believe you. Me – switching back to IE8. Good and safe.

  21. Daniel Veditz wrote on :

    @Idan: We do publish it, and above I got a roughly half-and-half count going through our public advisories. Any independent observer can go through the bugs linked from those advisories (or alternately, from bugzilla queries) and make their own count if they don’t believe me. The advisories have names attached, the bugs have names attached, it’s all public and published.

  22. Wilson Perdomo wrote on ::

    Other software vendors do not report all the bugs they have with their software unlike Firefox. I believe is unfair to compare Firefox with dishonest Micrsoft, Apple, and Opera.

  23. Mister Smith wrote on :

    ClickJacking …. Firefox has an addon, NoScript, which was the only one able to stop the attempt. The rest allowed the clickjackers to exploit all the friends I know using Internet Explorer. Boon of business for me to fix these machines, but they all swear by Firefox now. These number do not fool me. Firefox finds a problem, lets me know what it is and has it fixed by the time i read about it. THANK you Mozilla …

    Mozilla is #1

  24. Patrick wrote on ::

    While I am no expert on coding anything. I do however repair/remove a ton of virus and other crap my customers get. My repeat customers are the ones who don’t listen to me about Firefox. They refuse to understand that you can’t just wait till your software vendor release’s their normal updates to fix security holes. Firefox, while not perfect has helped me to gain repeat customers, not by being a bad product but by doing so well, that when something else goes wrong i.e. hard drive fails, those customers return since they were happy with my advice.(Granted it’s not just me recommend Firefox, I recommend other open source programs, it also has to do with the quick service times, and speaking to them in language that they can understand)

    I tell all my customers not only do you need to use a good browser but you also have to use some common sense on the web. Sorry even those who stick with IE if they use their brains would have far fewer problems.

    I never expect Firefox or any other product to be perfect, however I am pleased with the response times that Firefox has, not to mention that it’s open source, which allows me to use a ton of add ons :)

    Any type of report is bound to be wrong one way or another. Just look at the way company’s are doing their polling. The post the numbers but often leave out how the question was worded.

    If I asked a 100 people if they liked scrambled eggs served COLD, and then reported that 99% of the people I polled do not like eggs, Would that be a lie?

  25. sly wrote on :

    mozilla and firefox are awesome. Good response to the security report. I feel safe again. Seriously, keep up your great and hard work.

  26. Bill Pacos wrote on :

    @stillwaiter

    Nobody said that secunia only hates firefox, but secunia sure doesn’t try to even pretend to be impartial in their latest “report”.

    I’m very curious how you feel justified stating that a group that releases a “report” isn’t responsible for its accuracy. Are you saying that it’s ok to do sloppy work as long as it’s easy? I just did a study and found that 97% of the people that use the handle “stillwaiter” have below average intelligence – it’s a good thing I’m not responsible for the accuracy of those results, right? It’s just my job to report them!

    I might almost agree with you that it’s not secunia’s job to educate people about the meaning of the numbers, but any self-respecting data collector should at least have the integrity to highlight numbers that were collected in different ways when “comparing” them. As has been pointed out in the comments above, even just having an asterisk next to the number and a quick explanation in the chart would have at least been a step in the right direction. Explaining large variations in your data is not showing bias, it’s explaining large variations in your data. Since this data was collected in a different manner, it’s entirely biased not to disclose this. It’s a good thing real scientists can’t be this sloppy! Maybe next year Secunia should double its budget to TWO dollars to generate a useful report.

    Regarding the definition of “neutrality”, perhaps this might help since you can’t seem to find it on wikipedia:
    http://dictionary.reference.com/browse/neutral
    The first definition is “not taking part or giving assistance in a dispute or war between others”. Secunia releasing a report based on information it knows to be incorrect with misleading graphs fits well here. This does not fall into the category of “neutral” since it IS giving assistance to all non-firefox browsers in its report. Perhaps your English could use a bit of polishing too.

    Ari T is right. You can tell your employer they are wrong.

  27. question2 wrote on :

    This was never answered:

    Braggin about Firefox’s short time to fix…

    Does that include flaws discovered by Mozilla, and therefore reported as having a TTF of 1 day?

    So if Mozilla does indeed disclose more internally discovered flaws than other vendors, then what is their TTF at Secunia? 1 day?

    If so, this might make Mozilla’s TTF claim bogus because the stats are deflated by artificially short TTF bugs.

  28. Daniel Veditz wrote on :

    @question2

    The reports that have measured “time to fix” have generally only counted externally-reported flaws. Vulnerabilities that were not announced until fixed were not counted, both internally-discovered (and possibly not reported at all depending on vendor) and reported by 3rd-parties following “reponsible disclosure”.

    To make this distinction clear such reports often use the term “window of exposure” or “days of risk”. And they’re cumulative rather than averaged: a whole mess of TTF 1 day bugs doesn’t make you look better, each one adds to the “window of exposure”.

  29. Tom Anderson wrote on ::

    Securia’s research proves that IE is definately safer than Firefox (joke). But it does point out that 93% of everyone patches IE. Only 84% of people keep Firefox updated, which shows that more people are having trouble keeping Firefox updated. The reason that 115 patched vulnerabilities in Firefox would be cause for alarm is that people aren’t patching it. Even if MSIE patched 200 undisclosed vulnerabilities, a much higher percentage of IE users (at least those users who ran the Secunia Software Inspector) are safe because their browser has been updated.

    And nobody is using those insecure plugins such as ActiveX, right? By putting the two graphs next to each other (browser vulnerabilities vs. plugin vulnerabilities) IMO Secunia did a good job of reporting.

    A key thing would be to remind people that Firefox is NOT secure, so it IS important to keep it updated. Also it’s important to make updating an easy process. And I note that Vista security prevents Firefox from checking for updates automatically, but WHY?

    http://support.mozilla.com/en-US/kb/Check+for+Updates+is+disabled

    This is kind of absurd. We should be able to CHECK for updates even if we can’t install those updates because we’re not running elevated privileges. This Secunia article should be a swift kick: Mozilla should get on the ball.