Where in the world is AMO? (Part III: It’s Dead.)

7

Shortly before 12:30am PDT I had to roll back the DNS changes to AMO and serve it only out of San Jose. Around this time, Europe started coming online and pushed traffic loads up, exhausting the capabilities of the Netscalers in Amsterdam.

The Bad
Unfortunately when SSL transactions/second hit nearly 900 a second the CPU was pegged at 100% and the box started failing external health checks and started peforming “oddly”.

SSL Transactions / second

I mentioned elsewhere that the pair in Amsterdam is a pair of Netscaler 7000s without hardware SSL offloading. The glossy material from Citrix says I should be able to get 4400 SSL trans/second. Admittedly the box is doing more than just SSL (caching, compressing, RTT probes), but not even getting to 1/4 of that number sucks.

(We had exactly the same problem with the 9000s (4400 SSL tps) and 10ks (8800 SSL tps) – during release periods we’d easily top out at more than 3k SSL trans/sec, below their 4400/8800 mark, and the boxes would fall over on themselves. We’re now running on the 12ks which have two SSL hardware cards and two CPUs and perform much better but I’m not sure where Citrix get their numbers)

The Good
On the success side, AMO quite quickly started pushing a significant amount of bandwidth out of Amsterdam -

AMO Traffic

I rolled back before peak traffic but during this time frame, a good 11% of AMO traffic was sourced out of Amsterdam and I got a lot of feedback from other channels that performance was quicker.

What’s next?
So what’s the next step? I’ll be shipping out replacement pair of Netscaler 9000s this week that do have an SSL offload card and we’ll re-try this in a couple weeks when they’re online.

While the Netscaler clearly failed to keep up with the load, I should point out that I’m a huge fan of the product. If I had to build out some non-commercial solution using lighttpd or squid or something else to handle AMO (and the SSL traffic and load balancing and GSLB and HA), I’d have spent more than I spent on the Netscalers.

ps. Anyone more local to Amsterdam who wants to help racking?

Categories: Mozilla

7 responses

  1. sam wrote on :

    Please excuse my ignorance, but why is SSL even necessary? Firefox comes pre-loaded with only mozilla sites as able to install XPIs, so security without it isn’t all that bad.

  2. mrz wrote on :

    SSL helps with site verification to make sure that the addons.mozilla.org you goto it really the real addons.mozilla.org (and not some browser or DNS hijack).

    It also verifies that XPIs are coming from a trusted source and there aren’t any man-in-the-middle attacks during an install.

  3. Jon Pritchard wrote on :

    Sounds to me like you need to switch hardware vendors. Those figures are so vastly under the estimated, that is it worth continuing buying hardware from Citrix?

  4. mrz wrote on :

    Two things to keep in mind –

    1. AMO isn’t the typical https website, especially the VersionCheck traffic (used for extention version updates). Those are short lived sessions but there are lots of them. Without knowing how Citrix actually did their test, it’s hard to say if this is relevant or not. Though I’ll concede that I should get more than 25% of what they advertise :)

    2. There really isn’t any alternative solution that includes caching, SSL offloading, compression and load balancing. There are products that do a subset of that are are more expensive than Citrix.

    There is ZXTM from Zeus that I’ve looked at briefly. The scale differently and it’s not clear to me what the total cost would be to get a system that would match the performance of the Netscaler 12000s (ZXTM is a pure software solution using off the self hardware and my Linux distro of choice so the bulk of the expense will be in getting some number of PC servers + the software).

  5. morgamic wrote on :

    Good post, mrz — I hope the 9000s work in a couple weeks. :)

  6. Pingback from mrz’s noise » Blog Archive » How to build a better (SSL proxy) mouse trap (with lighttpd)? on :

    [...] We run our web farm behind a pair of Citrix Netscalers in both San Jose and Amsterdam. What really hits these boxes hard is the SSL offloaded traffic and in certain instances has caused the Netscalers to fall over on themselves. [...]

  7. Pingback from mrz’s noise » Blog Archive » Where in the world is AMO? (Part IV: Take 2) on :

    [...] a month after the first attempt to get AMO (addons.mozilla.org) served out of Amterdam as well as San Jose failed, we’re [...]