{"id":179,"date":"2009-02-02T10:29:15","date_gmt":"2009-02-02T18:29:15","guid":{"rendered":"http:\/\/blog.mozilla.org\/webdev\/?p=179"},"modified":"2017-03-08T14:50:09","modified_gmt":"2017-03-08T22:50:09","slug":"socorro-partitioning-rolled-back","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/","title":{"rendered":"Socorro Partitioning Rolled Back"},"content":{"rendered":"<p>This Thursday and Friday we attempted to push <a href=\"http:\/\/blog.mozilla.org\/webdev\/2009\/01\/20\/socorro-database-partitioning-is-coming\/\">updates to re-partition our crash report database<\/a> and optimize the reporting tool to take advantage of it.\u00a0 This was the deployment of <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=432450\">bug 432450<\/a> and a fix for <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=444749\">bug 444749<\/a>, among others.<\/p>\n<p>Our first attempt suffered from a network timeout, which required an eleven hour restore and re-run.\u00a0 The re-run, done Friday, was done using a socket connection but would have required an additional 1-3 days of downtime, which was well outside our originally announced window.\u00a0 Consequently, the database was rolled back to its contents as of 6:55PM PDT, January 29.\u00a0 Reports have since resumed processing.<\/p>\n<p>We plan on doing the following:<\/p>\n<ul>\n<li>Set up a complete replica of production to test this process end-to-end.\u00a0 Our dry runs were done on a staging database that was roughly 1\/5 the size.  We anticipated a scaling of O(n), but in practice on the production server, we got performance more inline with O(n^2).   So we did not expect the full extent of timeouts or how much downtime would be needed.\u00a0 This will be avoided in future updates and we are setting up a stage database from a recent dump (once we gather the hardware for it).<\/li>\n<li>Push a now+ partitioning script.\u00a0 The work done in <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=432450\">bug 432450<\/a>, on top of a complex migration script for old data, has logic for handling new partitions automatically which benefits new reports.\u00a0 Since we don&#8217;t want to keep adding to our old database schema, we will push these updates so that new reports are properly partitioned.\u00a0 Pros &#8211; in a week or two, things will be speedy and we aren&#8217;t going to struggle with timeouts.\u00a0 Cons &#8211; we aren&#8217;t migrating the last 4 weeks.  We will not see a performance increase when querying data older than the date of the repartitioning.<\/li>\n<\/ul>\n<p>We would like to push the partitioning script (without migration of old data) on Thursday.\u00a0 We will announce when it will be as soon as we know.<\/p>\n<p>Long term, we are already in the process of seeking additional resources to help examine our database configuration and systems architecture.\u00a0 We will have more updates on that process in the future.<\/p>\n<p>Our team wants this work deployed as much as everyone else.\u00a0 Thanks to everyone for their patience as we work through these issues.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This Thursday and Friday we attempted to push updates to re-partition our crash report database and optimize the reporting tool to take advantage of it.\u00a0 This was the deployment of bug 432450 and a fix for bug 444749, among others. &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/\">Continue reading<\/a><\/p>\n","protected":false},"author":24,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[96],"tags":[],"coauthors":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Socorro Partitioning Rolled Back - Mozilla Web Development<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mike Morgan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/\",\"name\":\"Socorro Partitioning Rolled Back - Mozilla Web Development\",\"isPartOf\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\"},\"datePublished\":\"2009-02-02T18:29:15+00:00\",\"dateModified\":\"2017-03-08T22:50:09+00:00\",\"author\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.mozilla.org\/webdev\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Socorro Partitioning Rolled Back\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/\",\"name\":\"Mozilla Web Development\",\"description\":\"For make benefit of glorious tubes\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f\",\"name\":\"Mike Morgan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1d6151e93e27d281a7fc06534ee8eae0\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g\",\"caption\":\"Mike Morgan\"},\"description\":\"Mike Morgan is the former Director of Web Development for the Mozilla Corporation.\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/author\/morgamicmozillacom\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Socorro Partitioning Rolled Back - Mozilla Web Development","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/","twitter_misc":{"Written by":"Mike Morgan","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/","url":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/","name":"Socorro Partitioning Rolled Back - Mozilla Web Development","isPartOf":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#website"},"datePublished":"2009-02-02T18:29:15+00:00","dateModified":"2017-03-08T22:50:09+00:00","author":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f"},"breadcrumb":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.mozilla.org\/webdev\/2009\/02\/02\/socorro-partitioning-rolled-back\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.mozilla.org\/webdev\/"},{"@type":"ListItem","position":2,"name":"Socorro Partitioning Rolled Back"}]},{"@type":"WebSite","@id":"https:\/\/blog.mozilla.org\/webdev\/#website","url":"https:\/\/blog.mozilla.org\/webdev\/","name":"Mozilla Web Development","description":"For make benefit of glorious tubes","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/373fd84922ef95d8d5171d1a72770d4f","name":"Mike Morgan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1d6151e93e27d281a7fc06534ee8eae0","url":"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ff5890a3252cd74f006ee33360e71c76?s=96&d=mm&r=g","caption":"Mike Morgan"},"description":"Mike Morgan is the former Director of Web Development for the Mozilla Corporation.","url":"https:\/\/blog.mozilla.org\/webdev\/author\/morgamicmozillacom\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/179"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/comments?post=179"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/179\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/media?parent=179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/categories?post=179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/tags?post=179"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/coauthors?post=179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}