Emergency Mozilla Downtime – 03/05/2009, 8pm – 11pm PST (0400 – 0700 03/06/2009 UTC)

mrz

Update:

Sorry, in all that was going on today I forgot today is Thursday and while Thursday’s are normal maintenance nights, Thursdays during release are not.

That said, we’ll be tacking on the following two tasks:

  • 7:00pm PST (0300 UTC) mrapp-stage02 reboot.  See bug 480415 for details.  We’ll be rebooting mrapp-stage02 to pick up changes to Ethernet bonding configuration.  This will affect most *.stage.mozilla.com and *.authstage.mozilla.com staging sites. Duration 15 minutes.
  • 7:00pm PST (0300 UTC) mozilla.org MX changes.  We’ll be changing DNS MX records for mozilla.org.  No downtime expected.

(Apologies for the short notice. We generally want to stick to our normal advertised maintenance windows but sometimes technology works against us.)

We will have an emergency maintenance window tonight from 8pm to 11:00pm PST to fix router configurations after some instability earlier today.

The following work will take place:

  • 8:00pm PST (0400 UTC) Turn up border1 BGP session to Mzima & Any2. See bug 481755 for details. In order to reduce some router CPU load and instability caused earlier this morning we shed several BGP peers. During this window we’ll re-establish peering with upstream transit providers in a controlled environment. No downtime expected.

Let us know if there are any objections as soon as possible.