noise from a mozilla IT/Operations wrangler
I mentioned briefly in Monday’s meeting the performance issues we’re having with our load balancers. Since then, we’ve been hustling to turn up something in the short term to handle Thursday’s Major Update and Firefox 3.0.5/2.0.0.19 release (see here). For a number of months we’ve been looking at Zeus and their ZXTM product. It has… Read more
Completed one of the router upgrades I mentioned the other day lastnight. One real issue I ran into was the built in sup-bootflash: was too small to hold the IOS image I wanted and I wasted some amount of time deleting/squeezing sup-bootflash: and remembering the boot system syntax to boot off disk0:. Thought I’d share… Read more
A couple months ago I mentioned how things have grown in the past two years at Mozilla. Back then we barely pushed any traffic to the Internet and survived on less than a dozen app servers. Things have changed. I’ll highlight just a couple of them: Active Firefox users grew from roughly 20 million users… Read more
This post is written entirely out of frustration. For what seems like months I’ve been on-and-off troubleshooting wireless connectivity issues with Cisco. I’ll give a little background first. At Mozilla’s main campus I’m using a Cisco 3845 ISR with two NM-WLC Wireless LAN controllers and have a total of 9 APs covering two buildings. I… Read more
When I started at Mozilla two years ago the biggest challenge was handling a release and pushing very close to 100Mbps. Right around that point the firewalls would fall over. We had one data center and essentially one provider and I could count the number of app servers on my two hands. That was two… Read more
Among the bazillion link exchange requests I get in Hostmaster’s email, I got this gem – To whomever it may concern, I am interested in your domain names firefox.com,mozillafirefox.com,mozilla.com and was wondering if you would consider selling them to me. I would be willing to go through Escrow.com (or any reputable escrow service of your… Read more
I’m at it again but this time I’m shooting for 80, and yes, I’m that crazy. I’ll be using a 2-post relay rack with 10 single-sided shelves on one side (that’d be 9 Minis per shelf). This means I’ll deal with the following nightmares: 80 freakin’ power bricks, power cords 80 ethernet cables and no… Read more
Finally got our home-grown GeoDNS deployed last week, one week later than I wanted. A couple deployment issues and bug 435134 got in the way. We’re finding all sorts of uses for this now and have moved several websites over to this and have a couple more websites/services (like IRC) scheduled for this. Purely out… Read more
The Mozilla community is blessed with a lot of freely donated bandwidth through our mirror network. This network handles product downloads through bouncer and a subset of this handles releases.mozilla.org, which carries contains Firefox, Thunderbird, Camino, Seamonkey, and Sunbird releases and Add-ons (and is huge, clocking in around 100-130G). releases.mozilla.org is handled through round-robin DNS. … Read more
Just got through a couple weeks of upgrading Mozilla’s entire VMware ESX infrastucture to 3.5 and I have to say, Storage VMotion is simply the coolest thing (well that and VMotion itself and DRS). I feel like I’ll never need downtime anymore! But it’s not without its gripes. Hey VMware – why didn’t you integrate… Read more
