DB Friday, February 24th edition

The overriding theme this week – we are still doing a lot of shifting around of databases pending our data center move.

This week, the database team at Mozilla has:
– Discovered some minor but important issues following a crash. We have increased alerting to ensure that if cleanup after a crash is necessary, the proper folks will be notified.
– Created a new database for Tinderbox Push Log for bug caching. In the process, updated some sorely out-of-date database documentation.
– Added new machines to graphs.mozilla.org

– Cleaned out puppet entries from some database machines that no longer exist.

– Updated database entries in puppet to include or exclude the “read_only” setting as appropriate, and are monitoring that setting with the PalominoDB Nagios Plugin for MySQL.

– Updated a process for loading a Postgres database; instead of four hours, the process only takes a minute.

– Racked the Scalarc appliances for our proof-of-concept (POC) evaluation.

– Started a proof-of-concept (POC) evaluation with ScaleBase.

– In preparation for our data center move:
– We now have redundant backups, both logical and physical, on our most-used development machine as well as the databases backing addons.mozilla.org and support.mozilla.org. We are also monitoring the backups to ensure that no backup is too old. (We are working on testing restores of the major databases as well, but there is only so much we can do every week, and having the backups took priority).

– We have moved our staging site for our Crash Stats site.
– We have turned off the old site for support.mozilla.org in the data center we are moving away from.
– We have ensured that the databases for addons.mozilla.org are no longer being used and are ready to be turned off.