{"id":62540,"date":"2019-05-09T00:00:00","date_gmt":"2019-05-09T00:00:00","guid":{"rendered":"http:\/\/blog.mozilla.org\/foxtail\/2019\/05\/09\/what-we-do-when-things-go-wrong\/"},"modified":"2021-01-22T22:49:33","modified_gmt":"2021-01-22T22:49:33","slug":"what-we-do-when-things-go-wrong","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/","title":{"rendered":"What we do when things go wrong"},"content":{"rendered":"

We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry.<\/p>\n

An error on our part prevented new add-ons from being installed, and stopped existing add-ons from working. Now that we\u2019ve been able to restore this functionality for the majority of Firefox users, we want to explain a bit about what happened and tell you what comes next.<\/p>\n

Add-ons are an important feature of Firefox. They enable you to customize your browser and add valuable functionality to your online experience. We know how important this is, which is why we\u2019ve spent a great deal of time over the past few years coming up with ways to make add-ons safer and more secure. However, because add-ons are so powerful, we\u2019ve also worked hard to build and deploy systems to protect you from malicious add-ons. The problem here was an implementation error in one such system, with the failure mode being that add-ons were disabled. Although we believe that the basic design of our add-ons system is sound, we will be working to refine these systems so similar problems do not occur in the future.<\/p>\n

In order to address this issue as quickly as possible, we used our “Studies<\/a>” system to deploy the initial fix, which requires users to be opted in to Telemetry<\/a>. \u00a0Some users who had opted out of Telemetry opted back in, in order to get the initial fix as soon as possible. As we announced in the Firefox Add-ons blog<\/a> at 2019-05-08T23:28:00Z there is now no longer a need to have Studies on to receive updates anymore; please check that your settings match your personal preferences before we re-enable Studies, which will happen sometime after 2019-05-13T16:00:00Z. In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z.<\/p>\n

Our CTO, Eric Rescorla, shares more about what happened technically in this post<\/a>.<\/p>\n

We would like to extend our thanks to the people who worked hard to address this issue, including the hundred or so community members and employees localizing content and answering questions on https:\/\/support.mozilla.org\/<\/a>, Twitter, and Reddit.<\/p>\n

There\u2019s a lot more detail we will be sharing as part of a longer post-mortem which we will make public — including details on how we went about fixing this problem and why we chose this approach. You deserve a full accounting, but we didn\u2019t want to wait until that process was complete to tell you what we knew so far. We let you down and what happened might have shaken your confidence in us a bit, but we hope that you\u2019ll give us a chance to earn it back.<\/p>\n","protected":false},"excerpt":{"rendered":"

We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry. An error on our part prevented new add-ons from being installed, and stopped existing add-ons from working. Now that we\u2019ve been able to restore this functionality for the majority of Firefox users, we want to explain a bit about what […]<\/p>\n","protected":false},"author":1681,"featured_media":11295,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[72],"coauthors":[],"acf":[],"yoast_head":"\nWhat we do when things go wrong<\/title>\n<meta name=\"description\" content=\"We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry. There\u2019s more info in this post, but one thing to call out: for anyone who had to enable telemetry to get the initial fix, we\u2019re deleting all of the data collected since May 4th to respect our users\u2019 intentions as much as possible.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/\",\"url\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/\",\"name\":\"What we do when things go wrong\",\"isPartOf\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png\",\"datePublished\":\"2019-05-09T00:00:00+00:00\",\"dateModified\":\"2021-01-22T22:49:33+00:00\",\"author\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/a5a4dfa899cd35b0e7e77440c17e449f\"},\"description\":\"We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry. There\u2019s more info in this post, but one thing to call out: for anyone who had to enable telemetry to get the initial fix, we\u2019re deleting all of the data collected since May 4th to respect our users\u2019 intentions as much as possible.\",\"breadcrumb\":{\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage\",\"url\":\"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png\",\"contentUrl\":\"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png\",\"width\":1400,\"height\":770},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.mozilla.org\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What we do when things go wrong\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#website\",\"url\":\"https:\/\/blog.mozilla.org\/en\/\",\"name\":\"The Mozilla Blog\",\"description\":\"News and Updates about Mozilla\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.mozilla.org\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/a5a4dfa899cd35b0e7e77440c17e449f\",\"name\":\"Joe Hildebrand\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/image\/ca117a39de498debf51ed67f2580a753\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1f23df1fc8c03161522fe0b547a56bef?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1f23df1fc8c03161522fe0b547a56bef?s=96&d=mm&r=g\",\"caption\":\"Joe Hildebrand\"},\"url\":\"https:\/\/blog.mozilla.org\/en\/author\/jhildebrandmozilla-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What we do when things go wrong","description":"We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry. There\u2019s more info in this post, but one thing to call out: for anyone who had to enable telemetry to get the initial fix, we\u2019re deleting all of the data collected since May 4th to respect our users\u2019 intentions as much as possible.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/","url":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/","name":"What we do when things go wrong","isPartOf":{"@id":"https:\/\/blog.mozilla.org\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage"},"image":{"@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png","datePublished":"2019-05-09T00:00:00+00:00","dateModified":"2021-01-22T22:49:33+00:00","author":{"@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/a5a4dfa899cd35b0e7e77440c17e449f"},"description":"We strive to make Firefox a great experience. Last weekend we failed, and we\u2019re sorry. There\u2019s more info in this post, but one thing to call out: for anyone who had to enable telemetry to get the initial fix, we\u2019re deleting all of the data collected since May 4th to respect our users\u2019 intentions as much as possible.","breadcrumb":{"@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#primaryimage","url":"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png","contentUrl":"https:\/\/blog.mozilla.org\/wp-content\/blogs.dir\/278\/files\/2018\/03\/Logo2.png","width":1400,"height":770},{"@type":"BreadcrumbList","@id":"https:\/\/blog.mozilla.org\/en\/firefox\/what-we-do-when-things-go-wrong\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.mozilla.org\/en\/"},{"@type":"ListItem","position":2,"name":"What we do when things go wrong"}]},{"@type":"WebSite","@id":"https:\/\/blog.mozilla.org\/en\/#website","url":"https:\/\/blog.mozilla.org\/en\/","name":"The Mozilla Blog","description":"News and Updates about Mozilla","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.mozilla.org\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/a5a4dfa899cd35b0e7e77440c17e449f","name":"Joe Hildebrand","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/en\/#\/schema\/person\/image\/ca117a39de498debf51ed67f2580a753","url":"https:\/\/secure.gravatar.com\/avatar\/1f23df1fc8c03161522fe0b547a56bef?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1f23df1fc8c03161522fe0b547a56bef?s=96&d=mm&r=g","caption":"Joe Hildebrand"},"url":"https:\/\/blog.mozilla.org\/en\/author\/jhildebrandmozilla-com\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts\/62540"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/users\/1681"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/comments?post=62540"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/posts\/62540\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/media\/11295"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/media?parent=62540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/categories?post=62540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/tags?post=62540"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/en\/wp-json\/wp\/v2\/coauthors?post=62540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}