Mozilla IT

Mozilla IT & Operations

Posts from 2009

Mozilla Scheduled Downtime – 12/17/2009, 6pm – 12am PST (0200 – 0800 12/18/2009 UTC)

We will have a scheduled maintenance window tonight from 6:00pm to 12:00am PST. The following changes will take place: 7:00pm PST (0300 UTC) Geo-location support for bouncer/download.mozilla.org. We’ll be turning on geo-location support to download.mozilla.org tonight. This should improve the download experience by directing users to mirrors that are geographically close to their origin IP.… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 12/10/2009, 6pm – 12pm PST (0200 – 0800 12/11/2009 UTC)

We will have a scheduled maintenance window tonight from 5:00pm to 11:00pm PST. The following changes will take place: 6:00pm PST (0200 UTC) Kernel upgrades. We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each: *ALL* sm-* (development) servers. All IT-administered build services, including build.mozilla.org, try server,… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 12/03/2009, 6pm – 11pm PST (0100 – 0700 12/04/2009 UTC)

We will have a scheduled maintenance window tonight from 5:00pm to 11:00pm PST. The following changes will take place: 6:00pm PST (0100 UTC) Breakpad upgrade. We will be upgrading the breakpad environment to the next release (bugs fixed). In addition, we will also be making changes to the storage layout to help back-end storage scaling.… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 11/19/2009, 5pm – 11pm PST (0100 – 0700 11/20/2009 UTC)

We will have a scheduled maintenance window tonight from 5:00pm to 11:00pm PST. The following changes will take place: 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 529600). Duration 30 minutes. 9:00pm PST (0500 UTC) getpersonas.com GLB DNS change. We’ll be making DNS changes to move getpersonas.com… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 11/15/2009, 6pm – 7:30pm PST (0200 – 0330 11/16/2009 UTC)

We will have a scheduled maintenance window tomorrow from 6:00pm to  7:30pm PST. The following changes will take place: * 6:00pm PST (0200 UTC) bugzilla.mozilla.org update.  We’ll be upgrading Bugzilla from version 3.2.4 to 3.4.3 (bug 503055).  Duration 90 minutes. Feel free to comment directly if you see issues past the planned  downtime.

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 11/12/2009, 5pm – 11pm PST (0100 – 0700 11/13/2009 UTC)

We will have a scheduled maintenance window tonight from 5:00pm to 11:00pm PST. The following changes will take place: 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 528272). Duration 30 minutes. 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug… Read more

Categories: Scheduled Maintenance

Mozilla Network Outage Report – 11/12/2009, 5:07am PST – 5:20am PST

At around 5:07am PST this morning, Mozilla’s primary datacenter suffered an approximately 13 minute outage. Mozilla uses redundant Cisco Firewall Service Modules in the core Cisco 6509 switches. At about 5:07am the primary unit failed and appears to have crashed within the failover code. This caused an incomplete failover and the standby FWSM never assumed… Read more

Categories: Outages

Mozilla Scheduled Downtime – 11/03/2009, 7pm – 11pm PST (0300 – 0700 11/04/2009 UTC)

We will have a scheduled maintenance window tonight from 7:00pm to 11:00pm PST. The following changes will take place: 7:00pm PST (0300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 526284) . Duration 60 minutes. 7:00pm PST (0300 UTC) getpersonas.com update and database upgrade. We’ll be updating getpersonas.com to pick… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Downtime – 10/29/2009, 7pm – 11pm PDT (0200 – 0600 10/30/2009 UTC)

We will have a scheduled maintenance window tonight from 7:00pm to 11:00pm PDT. The following changes will take place: 7:00pm PDT (0200 UTC) labs.mozilla.com migration. We’ll be migrating labs.mozilla.com to mozillalabs.com (bug 489638). Duration 60 minutes. 9:00pm PDT (0400 UTC) addons.mozilla.org backend server maintenance. We’ll be making changes to the backend storage (bug 517015) and… Read more

Categories: Scheduled Maintenance