I had planned to take tomorrow off as a fun “play hooky” day, but I still have responsibilities, and with the crazy way this work has been going, I feel like I’ve done 5 days of work in 4 days’ time.
So far this week, I feel like this:
I posted that to our internal IRC channel, and it was submitted by someone else to Mozilla Memes, with the title “Life is always better with awesome DBA’s on staff. I love how Mozilla openly appreciates its employees, in both serious and fun ways.
Now, why was this week so crazy? Well, unlike previous weeks, very little work was done on the data center move (even though we have many, many machines yet to move).
- An HP machine that was a db server for Addons and the new Mozilla Marketplace had a hardware problem and crashed. At one point in time, addons had n+1 slaves – if one slave went down, we were OK. However, we realized that we’re right about capacity now – if one slave goes down, everything else strains to meet the workload. So we added 2 new read slaves to the pool.
- Started graphing the addons database servers with the MySQL Cacti Templates.
- Gave a user read-only access to the Bouncer database for the purpose of debugging a problem.
- Gave the Buildbot application its own database for production use, instead of being on a shared interface.
- Upgraded the aforementioned Buildbot database, including getting puppet to recognize and play nicely with a new version of MySQL.
- Rebuilt and refreshed a developer database cluster with backups (I love testing restores!).
- Created databases/authentication for developer sites (3 different ones)
- Exported databases from Firefox Flicks and support.mozilla.org
It feels like a short list, but all those backup refreshes take a while, and setting up puppet so it can support upgraded MySQL packages took a while. Especially figuring out dependencies for perl-DBD-MySQL!