Socorro Status Blog http://blog.mozilla.org/socorro Announcements about crash-stats.mozilla.com Wed, 05 Dec 2012 18:18:06 +0000 en-US hourly 1 http://wordpress.org/?v=3.5.1 Planned Maintenance 5-8pm PT – PostgreSQL 9.2 upgrade http://blog.mozilla.org/socorro/2012/12/05/planned-maintenance-5-8pm-pt-postgresql-9-2-upgrade/ http://blog.mozilla.org/socorro/2012/12/05/planned-maintenance-5-8pm-pt-postgresql-9-2-upgrade/#comments Wed, 05 Dec 2012 18:17:46 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=173 On Dec 5, 2012, from 5pm PT to 8pm PT, crash-stats.mozilla.com will be undergoing maintenance and will be temporarily unavailable.

This evening’s outage will involve routine maintenance which includes upgrading our systems to use PostgreSQL version 9.2. We will also be deploying version 29 of Socorro, which primarily supports the database upgrade.

We expect all systems to be recovered by 8pm PT.

]]> http://blog.mozilla.org/socorro/2012/12/05/planned-maintenance-5-8pm-pt-postgresql-9-2-upgrade/feed/ 0 Socorro upgrade: short downtime at 5.30pm PDT today http://blog.mozilla.org/socorro/2012/09/25/socorro-upgrade-short-downtime-at-5-30pm-pdt-today/ http://blog.mozilla.org/socorro/2012/09/25/socorro-upgrade-short-downtime-at-5-30pm-pdt-today/#comments Tue, 25 Sep 2012 22:03:27 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=169 We will be doing a minor PostgreSQL update from version 9.0.9 to 9.0.10 tonight at 5.30pm PT. The Socorro webapp will be unavailable for 1-2 minutes at that time.

This is being tracked in

]]> http://blog.mozilla.org/socorro/2012/09/25/socorro-upgrade-short-downtime-at-5-30pm-pdt-today/feed/ 0 Socorro maintenance window tonight, Sept 12, @ 5pm PDT http://blog.mozilla.org/socorro/2012/09/12/socorro-maintenance-window-tonight-sept-12-5pm-pdt/ http://blog.mozilla.org/socorro/2012/09/12/socorro-maintenance-window-tonight-sept-12-5pm-pdt/#comments Wed, 12 Sep 2012 20:57:39 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=165 Tonight we are pushing an update to Socorro (http://crash-stats.mozilla.com) to support rapid betas. This involves complex database changes to support all the by-build-date views that are in the new release.

We intend to do this without significant user-facing downtime, by breaking replication, failing over to secondary, pushing the changes to primary, verifying, and failing back. This is the first time we’ve tried a procedure like this for a code release. In the past we’ve done pushes like this one with downtime, although we have used this procedure for database upgrades. This is an attempt at making things disruption free for you, our users.

If this doesn’t work for any reason, in the worst case the webapp will be offline for up to an hour, starting at around 5pm PDT.

We will continue collecting crashes throughout, but processing will be paused during the maintenance window.

You can track progress in this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=790707

The bugs that will be shipping in this release are documented here:
https://bugzilla.mozilla.org/buglist.cgi?target_milestone=18&product=Socorro

]]> http://blog.mozilla.org/socorro/2012/09/12/socorro-maintenance-window-tonight-sept-12-5pm-pdt/feed/ 0 HBase outage – more leap second fallout http://blog.mozilla.org/socorro/2012/07/11/hbase-outage-more-leap-second-fallout/ http://blog.mozilla.org/socorro/2012/07/11/hbase-outage-more-leap-second-fallout/#comments Wed, 11 Jul 2012 13:39:30 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=160 We are currently experiencing an HBase outage due to an NTP offset. This was caused by accidentally re-running the leap second correction script (via puppet).

The NTP problem has been fixed – tracked in this bug (private infra only, sorry) https://bugzilla.mozilla.org/show_bug.cgi?id=772792.

We are in the process of restoring HBase services and will update when it’s done.

In the meantime you will be unable to load individual crash reports, and new crashes will not be processed, although they are being safely stored for later processing.

Update: we’re back. Took two attempts to restore HBase as it was running low on file descriptors. We will solve that problem medium term with a purge according to our data retention policy. New purge process is in testing on staging at present.

]]> http://blog.mozilla.org/socorro/2012/07/11/hbase-outage-more-leap-second-fallout/feed/ 0 HBase is down due to Java Leap Second bug http://blog.mozilla.org/socorro/2012/06/30/hbase-is-down-due-to-java-leap-second-bug/ http://blog.mozilla.org/socorro/2012/06/30/hbase-is-down-due-to-java-leap-second-bug/#comments Sun, 01 Jul 2012 01:36:41 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=156 A problem with Java’s handling of the Leap Second today caused our HBase clusters to crash. Currently we are accepting but not processing crashes. You will also be unable to view individual crash reports until the problem is resolved.

Tracking in

https://bugzilla.mozilla.org/show_bug.cgi?id=769972

I’ll update when the problem is fixed.

Update: As of 11.25pm EDT, Socorro is functioning normally.

]]> http://blog.mozilla.org/socorro/2012/06/30/hbase-is-down-due-to-java-leap-second-bug/feed/ 0 Socorro outage Sunday 4/15 http://blog.mozilla.org/socorro/2012/04/15/socorro-outage-sunday-415/ http://blog.mozilla.org/socorro/2012/04/15/socorro-outage-sunday-415/#comments Mon, 16 Apr 2012 03:39:29 +0000 Laura Thomson http://blog.mozilla.org/socorro/?p=152 The Socorro admin node was down for most of the day. This means that various pieces of batch processing did not happen. While the machine is now back online, we are still cleaning up.

All data was collected, and processing is up to date. You may see some abnormalities in aggregate data and graphs until we complete backfilling and cleanup.

Further detail can be found in bug 745539.

]]> http://blog.mozilla.org/socorro/2012/04/15/socorro-outage-sunday-415/feed/ 0 Pushing 2.4.2 database changes tonight http://blog.mozilla.org/socorro/2012/02/14/pushing-2-4-2-database-changes-tonight/ http://blog.mozilla.org/socorro/2012/02/14/pushing-2-4-2-database-changes-tonight/#comments Wed, 15 Feb 2012 01:35:43 +0000 jberkus http://blog.mozilla.org/socorro/?p=148 Socorro version 2.4.2 database changes are being pushed tonight, Feb 14, 6pm PST to 9pm PST.  This may cause some sluggishness, but will not require a downtime.

Code changes will be pushed on Feb. 15th.

]]> http://blog.mozilla.org/socorro/2012/02/14/pushing-2-4-2-database-changes-tonight/feed/ 1 Downtime planned for Socorro 2.4 Upgrade http://blog.mozilla.org/socorro/2012/01/10/downtime-planned-for-socorro-2-4-upgrade/ http://blog.mozilla.org/socorro/2012/01/10/downtime-planned-for-socorro-2-4-upgrade/#comments Tue, 10 Jan 2012 23:00:05 +0000 jberkus http://blog.mozilla.org/socorro/?p=143 Socorro/crash-stats will be down from 16:00 UTC to approximately 18:00 UTC ( 8am to 10am PST ) this Friday, January 13th.  This is to accomplish the 2.4 upgrade, whose primary feature is changing the whole socorro database to consistently use UTC times instead of PST/PDT times.  Followed by this downtime period will be a somewhat longer ( 2 to 5 hour ) period of degraded performance as we backfill a bunch of data in the background.

]]> http://blog.mozilla.org/socorro/2012/01/10/downtime-planned-for-socorro-2-4-upgrade/feed/ 5 individual crash reports temporarily inaccessible http://blog.mozilla.org/socorro/2011/11/28/individual-crash-reports-temporarily-inaccessible/ http://blog.mozilla.org/socorro/2011/11/28/individual-crash-reports-temporarily-inaccessible/#comments Mon, 28 Nov 2011 19:40:53 +0000 rhelmer http://blog.mozilla.org/socorro/?p=139 We were slow pushing a config change for the latest release, so individual crash reports are not accessible (all other reports are fine). We’re working on correcting this ASAP.

]]> http://blog.mozilla.org/socorro/2011/11/28/individual-crash-reports-temporarily-inaccessible/feed/ 1 Socorro Update this morning http://blog.mozilla.org/socorro/2011/11/07/socorro-update-this-morning/ http://blog.mozilla.org/socorro/2011/11/07/socorro-update-this-morning/#comments Mon, 07 Nov 2011 17:44:27 +0000 jberkus http://blog.mozilla.org/socorro/?p=136 We will be pushing a socorro update at around 10am PST this morning.  This update will enable hang reports and searches on content reports and a few other features.

During the push, expected to last until 10:30AM, some data may appear inconsistent and seach links may be broken.  This situation is expected to correct itself before 11:00AM; if it does not, then please inform the socorro team on irc.mozilla.org #breakpad.

Thank you for your patience!  Lots of great new socorro features are coming over the next three months!

]]> http://blog.mozilla.org/socorro/2011/11/07/socorro-update-this-morning/feed/ 1