Pushing 2.4.2 database changes tonight
Socorro version 2.4.2 database changes are being pushed tonight, Feb 14, 6pm PST to 9pm PST. This may cause some sluggishness, but will not require a downtime.
Code changes will be pushed on Feb. 15th.
Updates from jberkus@mozilla.com Toggle Comment Threads | Keyboard Shortcuts
-
jberkus
-
jberkus
Downtime planned for Socorro 2.4 Upgrade
Socorro/crash-stats will be down from 16:00 UTC to approximately 18:00 UTC ( 8am to 10am PST ) this Friday, January 13th. This is to accomplish the 2.4 upgrade, whose primary feature is changing the whole socorro database to consistently use UTC times instead of PST/PDT times. Followed by this downtime period will be a somewhat longer ( 2 to 5 hour ) period of degraded performance as we backfill a bunch of data in the background.
-
jberkus
This downtime has begun.
-
jberkus
This downtime has begun
-
jberkus
crash-stats.mozilla.com is back up and being verified by QA. If verification passes, we will resume processing crashes and complete backfill of the reports.
-
jberkus
We are still verifying issues with some of the reports & graphs. 2.4 upgrade is not complete yet.
-
jberkus
Upgrade and verification are complete. We are now completely UTC.
-
-
jberkus
Socorro Update this morning
We will be pushing a socorro update at around 10am PST this morning. This update will enable hang reports and searches on content reports and a few other features.
During the push, expected to last until 10:30AM, some data may appear inconsistent and seach links may be broken. This situation is expected to correct itself before 11:00AM; if it does not, then please inform the socorro team on irc.mozilla.org #breakpad.
Thank you for your patience! Lots of great new socorro features are coming over the next three months!
-
jberkus
Update is taking longer than expected. We will be in transition until around 12:30PM PST.
-
-
jberkus
Socorro 2.3.2 rollout in progress
Version 2.3.2 is currently being deployed onto production, and is expected to be complete tomorrow afternoon (PDT). While this is not expected to block access to crash-stats.mozilla.com at any point, there may be some odd behavior or broken links for Aurora and Nightly builds until data backfilling late tonight is completed. We appreciate your patience.
-
jberkus
Some services down for maintenance 8-23
To support an HBase update, some services will not be available on socorro from 17:30 to 21:30 PDT:
1. Processing of new crash reports
2. Retrieving details and dumps of individual crash reports
Top Crashes By Signature, graphs, and searching lists of crash reports should operate normally during this period.
-
jberkus
Configuration changes May 24
We will be doing some minor configuration changes tonight in preparation for the 1.7.8 deployment. This is not expected to interrupt service, but if it does the interruptions will be less than 5 minutes in length. So if you get a 404, just wait 5 minutes and try again.
-
rhelmer
We’ve hit an unexpected snag with the connection that the web servers use to the database, so crash-stats is currently reporting an error message and returning an HTTP 500.
Collection and processing are unaffected, and we are working to restore the crash-stats service.
-
rhelmer
Everything is back up now, but we’ll need to make some followup changes before the next release (1.7.8). We will likely make those changes tomorrow.
-
-
jberkus
Processing downtime tonight
Processing will be stopped for around 1.5 hours starting at 18:00 PDT tonight, in order to do maintenance on HBase.
Summary statistics will continue to be available on crash-stats during that time, but individual crash reports will not be available.
-
Robert Helmer
Socorro is back up as of 20:30 PDT.
-
-
jberkus
Minor Processing Delays
We are experiencing some minor lag in processing of new reports (one to two minutes). This issue should be fixed as soon as we can arrange an Hbase downtime.
-
jberkus
Deploying 1.7.7 today
The Socorro Team will be deploying version 1.7.7 today from 3pm Pacific to 5pm. This deployment will be done while Socorro is running, and is not expected to take service down. However, it may result in some odd behavior in the Web UI during the deployment itself.
When the upgrade is complete, it will be posted here. Until then, please ignore any issues in the Web UI. Thank you!
-
jberkus
Downtime tonight: 9PM to 10PM
In order to make some database changes, we will have the following downtimes tonight:
- Web UI will be nonfunctional from 9:00pm to 9:15PM
- Crash processing will halt from 9:00PM to around 10:00PM
All times are in PDT
-
jberkus
This downtime is complete.
jberkus is discussing. 
jberkus 6:50 pm on February 14, 2012 Permalink |
2.4.2 database upgrade is complete.