Mozilla IT & Operations
What drives Mozilla IT’s data center growth? You do. I can break down the machines in the data center into three sections (and this is a very simplistic view): Classic infrastructure. All the web sites and infrastructure to support Mozilla and Firefox. Release Engineering. All the infrastructure that builds and tests Firefox. Services. Right now… Read more
This week, our Phoenix datacenter fell prey to a series of brief rolling outages which visibly impacted many of Mozilla’s public services. Generally speaking, our datacenter architectures are intentionally simple and spanning tree has served us well. However, as we have grown to meet demand, some of our more… venerable datacenters have become convoluted as… Read more
The following is a guest post from Tim Guarnieri, Principal, Critical Facilities Practice at Q Builders, Inc. Tim’s been providing assistance and guidance to Mozilla in building out SCL3. In laying out racks in a data center, you can either contain the cold air or contain the hot air (or do nothing). In this post,… Read more
While all of the SCL3 work discussed in this blog goes on, we are still growing in other data centers as well. As I mentioned last week we have a lot of gear going to Phoenix. This week, 3 network engineers, 2 sysadmins, and 1 data center project manager descended on our Phoenix facility to… Read more
Just a short post on our progress. The racks arrived last week… and have been staged out on the floor –
Over the last few weeks we’ve been making a number of small changes to the MXR web tool (https://mxr.mozilla.org/), and it seems about time to highlight some of the bigger changes. Background MXR is the Mozilla Cross-Reference system. It’s basically a convenient way to search a whole lot of Mozilla code for certain things. I’m… Read more
(The short version is over here. This is the long version.) Quick Introduction We’re building out a new 1MW data center in Santa Clara, CA at Vantage Data Centers! Project: SCL3 is our latest data center build out. Ever since we started talking about this internally we knew we wanted to blog about it, to… Read more
Mozilla is growing, and we have to grow our infrastructure right along with it. A lot of this just happens behind the scenes as we are buying and installing new gear every month. For today’s post I thought I would give an insight into the broad range of servers we have in the ordering/shipping process… Read more
We’ve recently been doing a lot of CDN work (see my last post on getpersonas.com), and out of that has come some interesting data as to our world-wide traffic distribution. Here’s a breakdown for of the traffic for getpersonas.com, by region: The CDN traffic for http://mozilla.org/firefox is similar… although Asia ranks a bit higher… Read more
On October 13th at 1324 PST Nagios alerted the start of a network event affecting reachability to the SCL1 data center. SCL1 is configured with redundant internet links where a VPN traverses a redundant firewall at both ends. There is also a point-to-point (p2p) that connects directly to SJC1. The running configuration had the VPN… Read more
