Mozilla Network Outage Report – 11/12/2009, 5:07am PST – 5:20am PST

mrz

At around 5:07am PST this morning, Mozilla’s primary datacenter suffered an approximately 13 minute outage.

Mozilla uses redundant Cisco Firewall Service Modules in the core Cisco 6509 switches. At about 5:07am the primary unit failed and appears to have crashed within the failover code. This caused an incomplete failover and the standby FWSM never assumed full control.

At 5:19am PST the primary FWSM was manually reset which allowed the standby to complete its failover.

We have grabbed crashinfo and are working with Cisco TAC to diagnose. Specific details will be tracked in bug 528186.

We apologize for any inconvenience this may have caused and will continue to work with Cisco on a remedy.