Socorro Status Blog » outage http://blog.mozilla.org/socorro Announcements about crash-stats.mozilla.com Wed, 05 Dec 2012 18:18:06 +0000 en-US hourly 1 http://wordpress.org/?v=3.5.1 Socorro HBase connection issues http://blog.mozilla.org/socorro/2011/09/01/socorro-hbase-connection-issues/ http://blog.mozilla.org/socorro/2011/09/01/socorro-hbase-connection-issues/#comments Fri, 02 Sep 2011 02:59:25 +0000 lars http://blog.mozilla.org/socorro/?p=123 Socorro is currently experiencing an HBase connection issue of unknown origin. Processing is down. Web App functions that require fetching actual crashes is down. Web App functions that use the database appear to be working normally.

When I learn more, I’ll say more…

UPDATE: as of 9:30pm PDT, we’re back up and running. All systems should be back to normal. No crashes were lost during this outage.

]]> http://blog.mozilla.org/socorro/2011/09/01/socorro-hbase-connection-issues/feed/ 0 Socorro Disrupted by Data Center Issue. http://blog.mozilla.org/socorro/2011/08/31/socorro-disrupted-by-data-center-issue/ http://blog.mozilla.org/socorro/2011/08/31/socorro-disrupted-by-data-center-issue/#comments Thu, 01 Sep 2011 02:18:21 +0000 lars http://blog.mozilla.org/socorro/?p=115 The Socorro Web App is currently inaccessible due to issues in the Phoenix Data Center. Corey Shields broadcasted this message: “IT is working a network outage in our Phoenix data center right now. This is causing widespread site issues as well as netsplits on IRC. Will keep you posted.”

The outage seem pretty absolute, as I can access none of the Socorro infrastructure to see how it is faring. I will keep you informed as I know more.

]]> http://blog.mozilla.org/socorro/2011/08/31/socorro-disrupted-by-data-center-issue/feed/ 1 Socorro unplanned outage http://blog.mozilla.org/socorro/2011/07/28/socorro-unplanned-outage-2/ http://blog.mozilla.org/socorro/2011/07/28/socorro-unplanned-outage-2/#comments Thu, 28 Jul 2011 11:28:59 +0000 jabba http://blog.mozilla.org/socorro/?p=87 Socorro is experiencing an unplanned outage due to network difficulty in the Phoenix data center. The user interface continues to function for aggregate queries. Calling up individual crashes, along with crash processing is stalled. Crashes coming into the collectors are unaffected other than being delayed in processing. No priority jobs are being processed.

I will report back as soon as I hear more news as to the potential duration of this outage.

Update: HBase has been restarted and all functionality has been restored.

 

]]> http://blog.mozilla.org/socorro/2011/07/28/socorro-unplanned-outage-2/feed/ 0 Socorro Service Restored http://blog.mozilla.org/socorro/2011/03/10/socorro-service-restored/ http://blog.mozilla.org/socorro/2011/03/10/socorro-service-restored/#comments Fri, 11 Mar 2011 01:12:35 +0000 lars http://blog.mozilla.org/socorro/?p=37 IT announces that they’re done mucking about with the infrastructure for now. All Socorro services should be back to normal.

We are notified that this intermittent outage may repeat later this evening.

]]> http://blog.mozilla.org/socorro/2011/03/10/socorro-service-restored/feed/ 0 Socorro Trouble Today http://blog.mozilla.org/socorro/2011/03/10/socorro-trouble-today/ http://blog.mozilla.org/socorro/2011/03/10/socorro-trouble-today/#comments Fri, 11 Mar 2011 00:26:50 +0000 lars http://blog.mozilla.org/socorro/?p=35 Socorro processing has been intermittently adversely affected by infrastructure issues since about 3:30 PST today. First our connection to PostgreSQL failed and, after that came back, our connection to HBase started to get finicky.

IT is on top of the issue, and I’ll post again when we’ve got more of an idea as to when these intermittent problems will be resolved.

]]> http://blog.mozilla.org/socorro/2011/03/10/socorro-trouble-today/feed/ 0