Derek Moore
This week, our Phoenix datacenter fell prey to a series of brief rolling outages which visibly impacted many of Mozilla’s public services. Generally speaking, our datacenter architectures are intentionally simple and spanning tree has served us well. However, as we … Continue reading
ravi
On October 13th at 1324 PST Nagios alerted the start of a network event affecting reachability to the SCL1 data center. SCL1 is configured with redundant internet links where a VPN traverses a redundant firewall at both ends. There is … Continue reading
ravi
Network Operations will be conducting the following maintenance windows: Tuesday July 12 2011 1800 PST (0100 UTC) [Tree Closure] Duration: 3h All Firefox, Fennec & TryServer trees will be closed during this window. The VPN concentrator in SJC1 will be … Continue reading
amilewski
We will have a scheduled maintenance window Tuesday morning, May 31 from 6am to 10am PDT (1300-1700 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 6:00am PDT (1300 UTC) Talos/Pageloader update. RelEng will be landing … Continue reading
amilewski
We will have a scheduled maintenance window Monday morning, May 9 from 6am to 7am PDT (1300-1400 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 6:00am PDT (1300 UTC) Talos/Pageloader update. A new Talos/Pageloader bundle … Continue reading
amilewski
We will have a scheduled maintenance window tomorrow night from 4:00pm to 5:00pm PST. The following changes will take place: 4:00pm PST (0000 UTC) support.mozilla.com. We will be switching over from the San Jose NFS cluster to the Phoenix NFS … Continue reading
mrz
For several hours this morning, Mozilla’s Phoenix data center suffered several intermittent outages. This was fall out from yesterday’s Juniper SRX JunOS upgrade. The following sites/services may have experienced degraded performance or partial/full outages: Firefox Sync Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com) … Continue reading
mrz
We will have an off-schedule maintenance window tomorrow night from 6:00pm to 9:00pm PST. The following changes will take place: 6:00pm PST (0100 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The nature of this upgrade requires … Continue reading
amilewski
We will have a scheduled maintenance window tomorrow (Friday) morning from 7am to 8am PST (1500-1600 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 7:00am PST (1400 UTC) Network hardware reconfiguration. Edge switches in one … Continue reading
amilewski
We will have a scheduled maintenance window Thursday Morning from 6am to 8am PST (1400-1700 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 6:00am PST (1400 UTC) Talos Screen Resolution Change. Screen Resolutions on Fedora … Continue reading