{"id":27,"date":"2007-03-23T12:43:08","date_gmt":"2007-03-23T20:43:08","guid":{"rendered":"http:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/"},"modified":"2017-03-08T14:50:19","modified_gmt":"2017-03-08T22:50:19","slug":"amo-site-updated-rolled-back","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/","title":{"rendered":"AMO Site Updated, Rolled Back"},"content":{"rendered":"<p>AMO was updated on Thursday, March 22nd around 8pm. Overnight, we watched the web infrastructure to ensure that AMO could withstand peak load times, but this morning near peak time cluster load levels became too high and we were forced to rollback yet again to prevent affecting other critical applications.<\/p>\n<p>Good news:<\/p>\n<ul>\n<li>Our database bottleneck is now nonexistent<\/li>\n<li>The app servers are fine during off-peak times<\/li>\n<li>During this short period, we received as much if not more feedback than during previously announced beta window (2 weeks)<\/li>\n<\/ul>\n<p>Bad news:<\/p>\n<ul>\n<li> App server load is unacceptable<\/li>\n<li>Traffic on web nodes has more than doubled as a result of absorbing releases.mozilla.org traffic<\/li>\n<\/ul>\n<p>Strategy:<\/p>\n<ul>\n<li>Profile application to look for new pain points (already done, no obvious culprits)<\/li>\n<li>Move public add-ons .xpi traffic back over to releases.mozilla.org<\/li>\n<li>Only sandbox files will be served from webheads for policy reasons<\/li>\n<li>Remove locale strings from all image URLs to improve cache rate for images<\/li>\n<li>Reassess and redeploy at earliest reasonable time<\/li>\n<\/ul>\n<p>Because life isn&#8217;t complete without colorful graphs, here is a graph showing database CPU (load reflected this too) going down dramatically:<img decoding=\"async\" loading=\"lazy\" src=\"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png\" alt=\"remora db cpu usage\" align=\"middle\" height=\"241\" width=\"597\" \/><br \/>\nBut the bad news is that our app nodes were angry:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"http:\/\/people.mozilla.org\/~morgamic\/remora-load.png\" alt=\"mrapp07 load graph\" align=\"middle\" height=\"241\" width=\"597\" \/><\/p>\n<p>This is largely due to a dramatic increase in overall traffic that was moved onto the cluster from releases.mozilla.org:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"http:\/\/people.mozilla.org\/~morgamic\/remora-traffic.png\" alt=\"remora traffic\" align=\"middle\" height=\"213\" width=\"597\" \/><\/p>\n<p>This traffic graph is what leads us to believe that offloading file transfers back onto releases.mozilla.org will give apache the breathing room it needs. We will post an update as soon as we can verify that.<\/p>\n<p>Thanks again everyone for supporting us as we work through these issues.  As for the negative comments, they serve as good motivation, too.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>AMO was updated on Thursday, March 22nd around 8pm. Overnight, we watched the web infrastructure to ensure that AMO could withstand peak load times, but this morning near peak time cluster load levels became too high and we were forced &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/\">Continue reading<\/a><\/p>\n","protected":false},"author":24,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"coauthors":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>AMO Site Updated, Rolled Back - Mozilla Web Development<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mike Morgan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/\",\"name\":\"AMO Site Updated, Rolled Back - Mozilla Web Development\",\"isPartOf\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png\",\"datePublished\":\"2007-03-23T20:43:08+00:00\",\"dateModified\":\"2017-03-08T22:50:19+00:00\",\"author\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage\",\"url\":\"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png\",\"contentUrl\":\"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.mozilla.org\/webdev\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AMO Site Updated, Rolled Back\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/\",\"name\":\"Mozilla Web Development\",\"description\":\"For make benefit of glorious tubes\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f\",\"name\":\"Mike Morgan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1d6151e93e27d281a7fc06534ee8eae0\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g\",\"caption\":\"Mike Morgan\"},\"description\":\"Mike Morgan is the former Director of Web Development for the Mozilla Corporation.\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/author\/morgamicmozillacom\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AMO Site Updated, Rolled Back - Mozilla Web Development","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/","twitter_misc":{"Written by":"Mike Morgan","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/","url":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/","name":"AMO Site Updated, Rolled Back - Mozilla Web Development","isPartOf":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage"},"image":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage"},"thumbnailUrl":"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png","datePublished":"2007-03-23T20:43:08+00:00","dateModified":"2017-03-08T22:50:19+00:00","author":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f"},"breadcrumb":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#primaryimage","url":"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png","contentUrl":"http:\/\/people.mozilla.org\/~morgamic\/remora-db-cpu.png"},{"@type":"BreadcrumbList","@id":"https:\/\/blog.mozilla.org\/webdev\/2007\/03\/23\/amo-site-updated-rolled-back\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.mozilla.org\/webdev\/"},{"@type":"ListItem","position":2,"name":"AMO Site Updated, Rolled Back"}]},{"@type":"WebSite","@id":"https:\/\/blog.mozilla.org\/webdev\/#website","url":"https:\/\/blog.mozilla.org\/webdev\/","name":"Mozilla Web Development","description":"For make benefit of glorious tubes","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f","name":"Mike Morgan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1d6151e93e27d281a7fc06534ee8eae0","url":"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g","caption":"Mike Morgan"},"description":"Mike Morgan is the former Director of Web Development for the Mozilla Corporation.","url":"https:\/\/blog.mozilla.org\/webdev\/author\/morgamicmozillacom\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/27"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/tags?post=27"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/coauthors?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}