Where in the world is AMO? (Part VI: We did it again!)

13

(See Part V.)

I have a confession.  We secretly did something last night (we only barely announced it to Metrics).

No, we didn’t secretly replace the fine coffee at some four-star restaurant but pretty close.

Hot off our 2 second gain in average page load times for addons.mozilla.org, we shaved another 2 seconds off by duplicating The Amsterdam Reboot platform in Singapore (as a proof-of-concept).  Don’t take my word for it – take Gomez’s word:

AMO page load times (Gomez)

A little background

Zeus GLBFor what seems like ages I’ve been trying to figure out how to best serve Asia-Pacific users. It’s a tough case to make because I didn’t have a method to easily measure how much bandwidth traffic I’d need or how it would change page load times or user perceptions.

But I’m a network guy and I’ve had this feeling that we need something in this region. We certainly have a growing population in the area – 5 of the top 9 countries we’ve seen > 20% growth in the last five months have been in Asia. I’ve only been lacking operational data.

AMO .sg bandwidthOver the summer I’ve been playing and testing Voxel’s Silverlining Technology Preview, their global cloud computer platform.  On October 1, it’s going to move out of Preview and their cloud platform would be generally available in all their POPs, including Singapore.

Seemed like a good way to get data…

In a matter of hours I had spun up two Zeus ZXTM and one GLB cloud servers with Voxel in Singapore. I waited till our normal Thursday night window before turning it live.

A couple take away notes on this

  1. We did it right.  We built a proxy/caching platform as part of The Amsterdam Reboot that can easily be replicated anywhere and instantly provide real quantifiable performance benefits.
  2. Clouds make perfect sense to do proof-of-concepts.
  3. IT can move really fast.  We had these three servers ready to go Wednesday morning in a couple hours..
  4. Mozilla’s webdev crew does an awesome job writing extremely cachable webapps. I’m seeing a 91% cache hit rate (350,000 objects, 1.5GB).
  5. If this is a sustainable location, pushing user-focused sites like support.mozilla.com to Singapore are next on my list.
  6. I’d love to run this concept in other geographies like South America. Who does clouds down there?

What I most like about this platform is that it’ll allow us to strategically get content closer to users where it most makes sense.

This is right now just a proof-of-concept. It lets me experiment and get real metrics. I’m very interested in hearing from people who actually live in the area – does this make you happier?

One more thing

I was lucky to have two providers who stepped in and provided resources to let me run this POC. Both deserve a special thanks.

  1. Zeus. These guys are great and quickly issued temporary licenses for both ZXTM and GLB.
  2. Voxel. You guys have a nice platform in the right part of the world. I like working with you.

Tags:

Categories: load balancing

13 responses

  1. Pingback from Where in the world is AMO? (Part VI: We did it again!) < mrz's noise « Noya Khobor on :

    [...] Original post: Where in the world is AMO? (Part VI: We did it again!) < mrz's noise [...]

  2. clouserw wrote on :

    Nice!

  3. David Tenser wrote on :

    Really cool.

    By the way, was the Kampyle survey we ran on SUMO any helpful in gauging the need for more bandwidth Asia-Pacific users?

  4. Pingback from Brion Vibber (brionv) 's status on Saturday, 26-Sep-09 19:57:16 UTC - Identi.ca on :

    [...] http://blog.mozilla.org/mrz/2009/09/25/where-in-the-world-is-amo-part-vi-we-did-it-again/ a few seconds ago from web [...]

  5. Jeff Balogh wrote on :

    What’s generating that world map picture? That looks super cool.

    1. mrz wrote on :

      @jeff That’s the Zeus GLB. It’s not a very good display. When we were testing it with http://www.mozilla.org you saw every DNS lookup show up in the map in near real time. Now, with AMO in there, the lookup rate is around 1500/second and the display can’t keep up.

      Took me several minutes before I could get a screenshot that looked interesting :)

  6. jmdesp wrote on :

    Today downloading my Namoroka nighly update I was only at 130 Kb/s on a line that’s able to do 600 Kb/s.

    Please think about serving the FTP updates from Amsterdam too !

    1. mrz wrote on :

      @jmdesp We sort of do but our download redirector doesn’t quite have any idea about geo-location. That’s something we’re also working on.

  7. reed wrote on :

    @mrz (re #5) — Actually, we don’t. All nightly updates come from http://ftp.mozilla.org, which is only hosted in San Jose. Might be worthwhile to set up a mirror of that in Amsterdam and/or other places.

  8. Jia Jun wrote on :

    Really cool work you guys had done!

  9. Stebs wrote on :

    Whoa, “our download redirector doesn’t quite have any idea about geo-location”, seriously?
    Ever considered Mirrorbrain (or parts of it?)
    http://mirrorbrain.org/

  10. handy aufladen wrote on :

    Hi,
    Thanks for sharing your technical informations.The charts are good to see the real time situation.

  11. jmdesp wrote on :

    Today, downloading firefox-3.7a1pre at 70kb/s from dm-ftp01.mozilla.org in the Pacific Time Zone.

    It would be nice to know the name of the ftp server in Amsterdam, if there’s one for nightly builds, and source code. I’m certain you serve enough copy for European users to make it worth having one.