{"id":61651,"date":"2009-03-07T00:00:00","date_gmt":"2009-03-07T00:00:00","guid":{"rendered":"https:\/\/blog.mozilla.org\/foxtail\/2009\/03\/07\/beware-the-security-metric\/"},"modified":"2021-02-08T20:27:57","modified_gmt":"2021-02-08T20:27:57","slug":"beware-the-security-metric","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/","title":{"rendered":"Beware the Security Metric"},"content":{"rendered":"

Editor’s note: Lucas Adamski<\/a>, director of security engineering for Mozilla, has posted a response to Secunia’s recently released 2008 security report (PDF link)<\/a>. We’ve reposted the full post from the Mozilla Security Blog<\/a> here.<\/em><\/p>\n

\n

Beware the Security Metric<\/h3>\n

Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia \u201c2008 Report\u201d (http:\/\/secunia.com\/gfx\/Secunia2008Report.pdf<\/a>). It tries to break down vulnerabilities reported by browser, and specifically states:<\/p>\n

\n

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
\npublicly disclosed prior to vendor patch as well as those included in Microsoft Security
\nBulletins.<\/p>\n

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.<\/p>\n<\/blockquote>\n

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.<\/p>\n

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It\u2019s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.<\/p>\n

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.<\/p>\n

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.<\/p>\n

The Secunia report is deeply disappointing on a number of levels. Frankly, it\u2019s disappointing that security researchers aren\u2019t taking the \u201cresearch\u201d part of their jobs as seriously as they once did. It\u2019s also disappointing that Secunia would publish something like this as one really expect better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.<\/p>\n

Lucas Adamski
\nDirector of Security Engineering<\/p>\n<\/blockquote>\n

Comment on this post at the Mozilla Security Blog<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Editor’s note: Lucas Adamski, director of security engineering for Mozilla, has posted a response to Secunia’s recently released 2008 security report (PDF link). We’ve reposted the full post from the Mozilla Security Blog here. Beware the Security Metric Security metrics are very difficult to do well, and easy to do poorly. For example, take a […]<\/p>\n","protected":false},"author":144,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,5],"tags":[],"coauthors":[],"acf":[],"yoast_head":"\nBeware the Security Metric<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/\",\"url\":\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/\",\"name\":\"Beware the Security Metric\",\"isPartOf\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/#website\"},\"datePublished\":\"2009-03-07T00:00:00+00:00\",\"dateModified\":\"2021-02-08T20:27:57+00:00\",\"author\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/33edd7d4d73723140487082573041c83\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.mozilla.org\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Beware the Security Metric\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#website\",\"url\":\"https:\/\/blog.mozilla.org\/en\/\",\"name\":\"The Mozilla Blog\",\"description\":\"News and Updates about Mozilla\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.mozilla.org\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/33edd7d4d73723140487082573041c83\",\"name\":\"Mozilla\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/image\/f32381c01597770b1131dff44b9d6de1\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f84bd67e8e3ab3bcc9676910aecf5700?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f84bd67e8e3ab3bcc9676910aecf5700?s=96&d=mm&r=g\",\"caption\":\"Mozilla\"},\"url\":\"https:\/\/blog.mozilla.org\/en\/author\/mozilla\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Beware the Security Metric","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/","url":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/","name":"Beware the Security Metric","isPartOf":{"@id":"https:\/\/blog.mozilla.org\/en\/#website"},"datePublished":"2009-03-07T00:00:00+00:00","dateModified":"2021-02-08T20:27:57+00:00","author":{"@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/33edd7d4d73723140487082573041c83"},"breadcrumb":{"@id":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.mozilla.org\/en\/mozilla\/beware-the-security-metric\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.mozilla.org\/en\/"},{"@type":"ListItem","position":2,"name":"Beware the Security Metric"}]},{"@type":"WebSite","@id":"https:\/\/blog.mozilla.org\/en\/#website","url":"https:\/\/blog.mozilla.org\/en\/","name":"The Mozilla Blog","description":"News and Updates about Mozilla","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.mozilla.org\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/33edd7d4d73723140487082573041c83","name":"Mozilla","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/image\/f32381c01597770b1131dff44b9d6de1","url":"https:\/\/secure.gravatar.com\/avatar\/f84bd67e8e3ab3bcc9676910aecf5700?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f84bd67e8e3ab3bcc9676910aecf5700?s=96&d=mm&r=g","caption":"Mozilla"},"url":"https:\/\/blog.mozilla.org\/en\/author\/mozilla\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts\/61651"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/comments?post=61651"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts\/61651\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/media?parent=61651"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/categories?post=61651"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/tags?post=61651"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/coauthors?post=61651"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}