mrz's noise

noise from a mozilla IT/Operations wrangler

Archive for Networking

Christmas Load Balancers

Dear Santa, All I want for Christmas is a load balancer that does not suck.  It should do the following: Terminate and offload SSL sessions at a rate of no less than 20,000/sec Provide in-memory content caching Support web logging, Apache-style (and despite what some hardware vendors thing, web logging is really important) Have some… Read more


Categories: load balancing

border2 upgrade done, one more to go

Completed one of the router upgrades I mentioned the other day lastnight.  One real issue I ran into was the built in sup-bootflash: was too small to hold the IOS image I wanted and I wasted some amount of time deleting/squeezing sup-bootflash: and remembering the boot system syntax to boot off disk0:. Thought I’d share… Read more

Categories: Mozilla, Networking

Router upgrades, San Jose

A couple months ago I mentioned how things have grown in the past two years at Mozilla.  Back then we barely pushed any traffic to the Internet and survived on less than a dozen app servers. Things have changed.  I’ll highlight just a couple of them: Active Firefox users grew from roughly 20 million users… Read more


Categories: Mozilla, Networking

Cisco wireless problems, multicast failures

This post is written entirely out of frustration.  For what seems like months I’ve been on-and-off troubleshooting wireless connectivity issues with Cisco. I’ll give a little background first. At Mozilla’s main campus I’m using a Cisco 3845 ISR with two NM-WLC Wireless LAN controllers and have a total of 9 APs covering two buildings. I… Read more

Categories: Mozilla, Networking

I (Mozilla) need a Network Engineer

When I started at Mozilla two years ago the biggest challenge was handling a release and pushing very close to 100Mbps.  Right around that point the firewalls would fall over.  We had one data center and essentially one provider and I could count the number of app servers on my two hands. That was two… Read more

Categories: Mozilla, Networking

geodns, one week later

Finally got our home-grown GeoDNS deployed last week, one week later than I wanted.  A couple deployment issues and bug 435134 got in the way. We’re finding all sorts of uses for this now and have moved several websites over to this and have a couple more websites/services (like IRC) scheduled for this. Purely out… Read more


Categories: load balancing, Mozilla, Networking

Geo DNS or getting the bits closer to you.

The Mozilla community is blessed with a lot of freely donated bandwidth through our mirror network.  This network handles product downloads through bouncer and a subset of this handles, which carries contains Firefox, Thunderbird, Camino, Seamonkey, and Sunbird releases and Add-ons (and is huge, clocking in around 100-130G). is handled through round-robin DNS. … Read more

Tags: , , , ,

Categories: load balancing, Mozilla, Networking

Hello China.

Over the past year we’ve talked about our market share in China (since last April anyways). Li Gong joined Mozilla and in June we had an office. In August I was in China with Justin to setup office infrastructure (phones, vpn) and shop for data center space. In December I was back in China to… Read more

Tags: ,

Categories: load balancing, Mozilla

China, Amsterdam, San Jose and global load balancing

Mozilla’s current GSLB (global server load balancing[1]) solution (Citrix Netscalers) is a mix of active proximity probes and static map assignments. The algorithm first checks to see if there’s a match in the static maps and then falls back to proximity metrics. If that’s missing, it’ll round-robin through all the GSLB sites (effectively three –… Read more

Tags: , ,

Categories: load balancing, Mozilla, Networking