Mozilla Scheduled Downtime – 12/10/2009, 6pm – 12pm PST (0200 – 0800 12/11/2009 UTC)

mrz

We will have a scheduled maintenance window tonight from 5:00pm to 11:00pm PST. The following changes will take place:

  • 6:00pm PST (0200 UTC) Kernel upgrades. We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
    • *ALL* sm-* (development) servers.
    • All IT-administered build services, including build.mozilla.org, try server, cruncher, AUS staging, and the pageload/graph servers
    • All VPNs and jumphosts in the offices and colos (except Beijing)
    • All office and colo DNS and DHCP servers (except Beijing)
    • All Web-dev and app staging servers
    • support.mozilla.com (chat services and knowledge base)
    • All version control services and related utility services (CVS, CVS-mirror, Hg, SVN, Bonsai, HgWeb, ViewVC, Tinderbox, MXR)
    • ftp.mozilla.org
    • stage.mozilla.org
    • people.mozilla.com
    • intranet.mozilla.org
    • quality.mozilla.org
    • litmus.mozilla.org
    • irc.mozilla.org (multiple servers, will fail over via DNS to keep it up)
    • addons.mozilla.org
    • download.mozilla.org
    • videos.mozilla.org
    • nagios.mozilla.org
    • bugzilla.mozilla.org
    • developer.mozilla.org
    • spreadfirefox.com
    • Internet-facing mail relays, list servers, and outbound mail (mail will be queued)
    • All public and private wikis and blogs administered by Mozilla

    Some services may have more than one outage as various back-end components each get their upgrades (for example, a web application will go down once when the web servers are upgraded and again when its database server is upgraded).

  • 7:00pm PST (0300 UTC) LDAP upgrade. We’ll be upgrading OpenLDAP tonight. Our LDAP infrastructure is redundant and there shouldn’t be any downtime noticed. Duration 30 minutes.
  • 9:00pm PST (0500 UTC) Zeus TM configuration changes. We’ll be turning some SSL cache settings on the production Zeus cluster (bug 533809). No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.