Mozilla DB News, Fri Jun 15th

The theme this week was “data integrity and security”:

  • Upgraded the kernel in our Puppet Dashboard DB master/slave cluster after one of the machines crashed.
  • Verified that none of our databases were running a version of MySQL that was affected by the serious MySQL authentication bypass vulnerability discovered, patched/fixed and reported by MariaDB. (On a personal note, the way they handled finding, fixing and reporting the bug makes me happy that they are such good community stewards)
  • We removed duplicate entries from our Bugzilla instance that were put in when assumptions were made about nonexistent unique keys.
  • Continued the database consistency check for bugzilla.
  • Backed up 2 more database clusters
  • Dealt with our development instance of having some latin1 MyISAM tables – converted them to utf8 InnoDB.
  • Added permissions for the metrics team to read new fields in Bugzilla.
  • Did a database consistency check for addons after a duplicate key error was triggered on several slaves, and re-sync’d the databases that had issues.
  • Repaired a crashed table that looked like it worked, other than not showing recent inserts.
  • Gave the Bugzilla team a place to test development releases of their code (as opposed to production releases)
  • Updated the Firefox Flicks staging database with a fresh copy of the database from development.
  • Decommissioned 2 database servers put into production for Addons due to 2 other machines that were crashing. The other 2 machines are healed, and the temporary servers were running out of disk space (they were smaller because they were temporary). We are back to having an n+2 configuration (without the 2 servers, we were exactly at n servers, meaning we had exactly as many servers as we needed. We like to run n+2, for just such an occasion)