Mozilla Security Blog
Categories: Firefox Musings Press Security

Critical Vulnerability in Microsoft Metrics

Window Snyder

14 responses

Jeff Jones, a director of security strategy at Microsoft published a report today about counting bugs. I blogged a few months ago about why I think counting bugs is less than useful:

Since all software has bugs, it’s more important to consider how long it takes to get a fix out when a security issue is discovered than it is to count bugs. Number of vulnerabilities identified is a function of how many bugs are present, but is probably more influenced by things like who is looking, and how good they are at finding security issues. That makes it a misleading metric.

When you compare how long it takes Microsoft to fix Internet Explorer vulnerabilities versus how long it takes Mozilla to fix vulnerabilities in Firefox it becomes clear why he chose to count vulnerabilities in this report instead. Earlier this year Brian Krebs of the Washington Post came up with this when comparing IE to Firefox:

For a total 284 days in 2006 (or more than nine months out of the year), exploit code for known, unpatched critical flaws in pre-IE7 versions of the browser was publicly available on the Internet…

In contrast, Internet Explorer’s closest competitor in terms of market share — Mozilla’s Firefox browser — experienced a single period lasting just nine days last year in which exploit code for a serious security hole was posted online before Mozilla shipped a patch to remedy the problem.

Mike Schroepfer goes into this in more detail in his post today.

One of the goals of the bug counting report is to demonstrate that Microsoft fixed fewer bugs for IE than Mozilla did for Firefox. Unfortunately for Microsoft (and for anyone trying to use this report as analysis of useful metrics) he does not count all the security issues. If he were able to count them all, Microsoft could get credit for all the bugs they fixed. He counts only the public issues, because that is all Microsoft will tell us about. Microsoft is worried that if it ever says it has fixed X security issues, the world will focus on that it had X vulnerabilities in the first place, not that they are now fixed and no longer a risk for users. So the set of issues that are available for public comparison is limited to the set of vulnerabilities that are reported externally AND fixed in security updates.

This is a small subset of all the vulnerabilities, because the vulnerabilities that are found through the QA process and the vulnerabilities that are found by the security folks they engage as contractors to perform penetration testing are fixed in service packs and major updates. For Microsoft this makes sense because these fixes get the benefit of a full test pass which is much more robust for a service pack or major release than it is for a security update.

Unfortunately for Microsoft’s users this means they have to wait sometimes a year or more to get the benefit of this work. That’s a lot of time for an attacker to identify the same issue and exploit it to hurt users. Sometimes it just takes time to put in a complicated fix. Anyone that has shipped a major piece of software can relate to that. But this is not the case for every internally found security issue. Extending this process to include fixes that are ready and just sitting on the tree waiting for the preferred vehicle to ship increases risk for users. But it sure keeps those bug count numbers down.

If we as an industry would just acknowledge that counting bugs is useless then vendors could feel safe talking about what they are doing to protect users. At Mozilla we fix our bugs openly. When you count Mozilla security bugs you are seeing not just those that are reported externally, but also the ones that would be considered internal if we acted like most other software vendors. Since all software has security vulnerabilities, we consider a vulnerability identified and fixed a win. It speaks to the strength of our community based security efforts to actively identify and quickly fix security issues. We don’t let fixes languish on the tree waiting for a major release while users are vulnerable. We ship fixes regularly because securing our users is more important than protecting our PR team from having to respond to articles about counting bugs instead of looking at the metrics that actually indicate whether a vendor is doing reasonable things to keep users secure.

We’re not building fixes for our PR team, we’re building them for our users. Go ahead and count.

14 comments on “Critical Vulnerability in Microsoft Metrics”

  1. pd wrote on

    Surely one of the ways in which the security of a browser should be judged is in how many bugs are actually ‘built in’ to the code? Fixing them is one thing, preventing them from appearing in the first place is another.

    That said, all this childish bickering between developers is quite cute really. Broadly speaking it just reinforces that we have a browser market where developers are very focused on security. Was not always the way.

    Also, isn’t it at least nice to hear someone at Microsoft talking about browsers again? That’s something they’ve stopped doing since IE7.

  2. Paul wrote on

    “Since all software has security vulnerabilities, we consider a vulnerability identified and fixed a win. It speaks to the strength of our community based security efforts to actively identify and quickly fix security issues. ”

    That remembers me of the 3rd dubest idea:
    http://www.ranum.com/security/computer_security/
    editorials/dumb/

    By the way, there are also other people that disagree with you:
    http://secunia.com/product/914/
    http://secunia.com/product/300/

  3. Gérard Talbot wrote on

    Hello Ms Snyder,

    I’ve asked this to Mike Shaver and I’m asking you the same request. Please read the Firefox FAQ

    3) Is Firefox more secure than Internet Explorer?

    http://www.mozilla.org/support/firefox/faq.html#mozvsie

    and then comment on the given answer and submit proposals, suggestions you feel are suitable for this issue. Like what would be a fair, honest, meaningful and relevant answer to this important question (more secure, better security, etc..).

    Browsehappy.com is also about this precise issue: so, it’s not a trivial question. Same thing about anti-phishing counter-measures performance, anti-phishing protection reliability and capabilities.

    If I recall correctly, you used to work for Microsoft in the area of security… right? If so, then you are most likely the best person possible to provide a wide and deep look into these questions and matters.

    Thank you for your time,

    Regards, Gérard Talbot

  4. Chris Sherlock wrote on

    I understand what you are saying, but what about the issues to do with product lifecycle? He basically says that Microsoft supports Internet Explorer versions for longer than Mozilla supports old Firefox versions. he claims that this causes big problems for users.

    What is your take on this?

  5. Phil wrote on

    Of course a vendor will pick statistics that make them look good. That is of course why you prefer to focus on time to fix 😉 But neither way gives a complete picture, so neither way should be dismissed out of hand. I happen to think Jeff does a fair job of transparency given his obviously biased position, and I’ve never seen him dismiss an alternate opinion without good solid data to back him up.

    And whilst I don’t believe that ‘hiding’ security fixes in a new release is a good thing, I do understand the need to roll these things up whereever possible when you have to deal with an enterprise market as well as a home user one. As a home user I will happily enable autoupdate on my machine, so the frequency of patches has little impact on me. But as an enterprise administrator, the last thing I want is a new security patch to test and deploy every couple of days with no particular schedule. This is why Microsoft’s patch Tuesday exists, and rolling patches into a service pack or new release is just an extension of this – balancing risk against convenience for me, the end customer.

    I would also be careful at trying to subtly promote the many eyes argument:
    ‘Number of vulnerabilities identified is a function of how many bugs are present, but is probably more influenced by things like who is looking, and how good they are at finding security issues.’
    I would argue (with no hard data to back me up of course) that the combination of hackers and security researchers reviewing Microsoft code probably beats the number and quality of eyes reviewing Firefox code, in which case more discovered bugs in IE would be the expected outcome surely?

  6. Igor Bukanov wrote on

    With a critical bugs discovered internally Mozilla also does not push the update immediately but rather includes it into the next scheduled update. Sometime it means that the patch is not released for over 2 months. This is not very different from MS policy that results in “a lot of time for an attacker to identify the same issue and exploit it to hurt users”.

  7. Peter Corcoran wrote on

    Here Here! I completely agree.

  8. John wrote on

    The other major reason for pushing fixes to service packs is reverse engineering. If Microsoft issues a spot change for each vulnerability, it’s fairly trivial to diff the binary before and after, discover what the vulnerability was, and exploit all un-updated machines. With the larger code churn of a service pack, Microsoft can fix un-exploited vulnerabilities without having to worry as much about putting the rest of their users (including past OSes sometimes) at risk.

  9. Nicky wrote on

    PR.. a form of survival when one fails to deliver. Oh well.. we love firefox just for that.

  10. GIles Jones wrote on

    I don’t see why Microsoft is so worried? it’s not like anyone has made huge inroads into their desktop monopoly.

    A browser is just a gateway to the web. Until governments demand the removal of IE there is no danger of the computer illiterate masses downloading and installing Firefox.

    Open source projects are always going to be more transparent regarding bugs. Commercial organisations are likely to get more criticism or be embarrassed by public disclosure of internal defects.

    The whole argument about who writes better code, commercial organisations or the open source community is lost if commercial organisations reveal that their software is rushed and buggy.

  11. Luci Stanescu wrote on

    “Fixing them is one thing, preventing them from appearing in the first place is another.”
    and
    “That remembers me of the 3rd dubest idea:
    http://www.ranum.com/security/computer_security/editorials/dumb/”

    Adopting a secure model is one thing. Creating something perfect is another. Either the author of those “dumb ideas” was exagerating or what I just read there is plain dumb. There is no such thing as “unhackable”. Developers are human beings. While it may be hard to introduce great security flaws in something like cp, creating a perfect >10.000 line software is far from possible. As he puts it, I understand that we should spend about 200 hours writing a software and distribute it as soon as it compiles (and it must compile in the first shot, as we adopted a good model, right?). Why, for God’s sake, would anyone want to test software? Oh, one more thing: don’t you dare compare a network to cp. That would be just plain stupid.

    pd and Paul:
    I’m waiting for a paper describing the flawed model of Firefox and the perfect model of IE (oh, and I would really love to find out how you managed to do that).

    “I would argue (with no hard data to back me up of course) that the combination of hackers and security researchers reviewing Microsoft code probably beats the number and quality of eyes reviewing Firefox code, in which case more discovered bugs in IE would be the expected outcome surely?”

    Of course, it’s so easy to find vulnerabilities in a binary, while, obviously, having the source code around can only make your life harder on this one. I wonder how many security experts Microsoft has reviewing IE…

    “With the larger code churn of a service pack, Microsoft can fix un-exploited vulnerabilities without having to worry as much about putting the rest of their users (including past OSes sometimes) at risk.”

    How exactly can you use “un-exploited” there? What makes you think that some other people won’t find the vulnerability independently? If you ask me, they are taking a huge risk if this is true…

  12. Phil wrote on

    Luci,
    >>Of course, it’s so easy to find vulnerabilities in a binary, while, obviously, having the source code around can only make your life harder on this one.

    Actually I think you’ll find plenty of serious researchers find it easier to see bugs in assembler rather than in C/C++, so raw binary or source code makes little difference.

  13. abcd wrote on

    Please ! Keep it simple and don’t play their game : They are not counting vulnerabilities. It’s just a word game.

    Let me repeat : They are counting PATCHES and not VULNERABILITIES. It’s THIS simple. Please read dictionary or wikip definitions if you don’t believe me.

    Please read their “reports” by putting the right words, and you will see there is no meaning at ALL.

    And, oh… other words they mess with : “MS security chief” => “MS public-face-advertising-chief”
    “report” => “advertising paper”.

  14. DonB wrote on

    “We’re not building fixes for our PR team, we’re building them for our users. Go ahead and count…”

    Hahaha….