Skip to content

Category Archives: Ops

BrowserID outage last week

15-May-12

What happened? Last week (9 May 2012 23:24 PDT until the following morning at 07:20), BrowserID had an outage affecting 50% of login requests. Embarrassingly, we didn’t find out until a user filed bug 753728 with us. Root Cause To put it bluntly, human error. Specifically, the accidental draining of all nodes of a load […]

The Sync Platform Meltdown, Explained

25-Oct-11

TL;DR: Recently, the Firefox Sync platform suffered serious performance degradation. We’d like to explain what happened, as well as the steps taken to deal with the issue from the perspective of Mozilla Services Operations (“Ops”). The Sync platform melted down after the release of Firefox 7… …to put it bluntly. This was the worst outage […]

Help Mozilla Services Operations debug what’s wrong with Sync

03-Oct-11

This post is for historical purposes only. Please do not send us any more log data at this time. Thank you!   TL;DR: Paste your most recent about:sync-log into a mozservices.pastebin.mozilla.org and send a link to @mozservices for us to check out. Sometimes when there’s a problem with Firefox Sync, Mozilla Services Operations (@mozservices) needs […]

Redirecting kickstart %pre/%post over serial console

31-Jan-11

To push output over console exec /dev/ttyS1 at the top of your %pre and/or %post sections of your kickstart config.

Working around missing –label option for RAID partitions in Kickstart

19-Jan-11

TL;DR – During kickstart, you can initiate “normal” partitions with a simple --label flag, why not a RAID partition? There are hardware setups that cause the on-disk OS to “see” your drives differently than kickstart did (a situation UUIDs normally solves, but referencing the UUID can be non-ideal in certain cases). This work-around fixes some of those cases.

Incrementing alpha strings with Vim

12-Jan-11

TL;DR: Need to increment a MAC address (for example) in Vim? Do this::set nf=octal,hex,alpha
Now put the cursor over the 2nd char of the octet pair and press <CTRL-A> to increment. Add the ‘set‘ line to your .vimrc for permanence.

PXE booting DOS

07-Jan-11

TL;DR: We still (apparently) need DOS to flash our firmware. Doing this the old way sucks and doesn’t scale. Read on for a network-based solution using PXE (via DHCP + TFTP) on Linux.