We have just made a change to a DNS record for getpersonas.com affecting its CDN usage, and thought we’d share some of our findings as to it’s worldwide traffic patterns.
Prior to now, getpersonas-cdn.mozilla.net pointed to a CDN aggregator service at 3crowd. We use this for several things. For example, aus2.mozilla.org (part of the Firefox Automatic Updater Service) goes through 3crowd. It works very well for this type of thing, where what we’re really after is distributing the load across our own infrastructure, in order to gain redundancy.
One of its limitations though is that it is inherently disconnected from the actual CDNs being used. That is, it has no way to know if a particular CDN is really good in one country, but really bad in another. It can’t make that type of determination on-the-fly. By default it just assigns a weight to each CDN, and doles out the traffic in whatever ratios we specify.
For a normal website like getpersonas.com, with broad international appeal, this isn’t quite ideal. For folks in North America or Europe, it works just fine… generally CDNs have pretty good coverage there, and for the most part it won’t make a big difference which one you end up using. Outside of those regions though, CDN coverage is much more hit-and-miss. If you’re in Australia or Argentina, which CDN you get routed to might make a very big difference on your page load times.
For example, here’s a “before” graph of getpersonas.com load time from our Gomez test node in Sydney, Australia (on Telestra, to be precise):
As you can see, it’s quite spikey. If you squint just right you can identify 3 tiers of performance. Not surprisingly, they line up with the 3 different backend services configured in 3crowd.The worst tier is from a local node we control ourselves in San Jose, CA. Clearly, that’s a pretty terrible choice from Australia… and yet it’s getting 1/3 of the overall traffic, simply because 3crowd doesn’t know any better.
This morning I changed this record away from 3crowd towards a new service we’re demoing, Cedexis. This is a similar service, but they maintain a database of CDN provider response times around the world… all the way down to the individual ISP level. Using this database, they can intelligently choose which CDN to use for every single request, theoretically using the fastest one from the user’s location.
Unfortunately it’s too soon to have a useful “after” graph to share. Instead, I’d like to share some information as to just how effective the new service might end up being.
“Global Village” is (as far as our traffic mix is concerned) the 4th biggest ISP in Brazil. They account for approximately 8% of Brazil’s traffic to getpersonas.com. Brazil as a whole is 45% of our South American traffic, which in turn is around 9% of our worldwide traffic. That’s the level at which we can drill down to. Here’s a graph of the “decisions” made by Cedexis for this one ISP in Brazil, over the last 24 hours. Remember, before today it would have been 50/50.
This is 94% vs. 6%. This should give you some idea of just how important choosing the proper CDN could be… far more often than not, we were making the wrong decision for these users. Situations like this generally occur when an ISP has limited peering with upstream providers, and thus does not have a good route to some places. This particular ISP appears to have a good connection to CDNetworks, but a poor one to Edgecast. Obviously this type of thing is more prevalent in smaller ISPs, where they may not be able to get (or afford) more complete peering agreements. You generally won’t see this type of thing with very big ISPs.
Now aggregate this around the world, and you can quickly see that even if the worldwide traffic mix still comes out to approximately 50/50 (and it does… within a few percentage points), making better decisions locally can result in a much better experience for a large group of users.
Over the next few days we should get a good idea of how much this is actually helping… I expect small gains worldwide, and possibly large gains in certain regions. Especially outside of North America, I expect getpersonas.com will be substantially improved, with Asia-Pacific and South America seeing the biggest gains. When we have some data, I’ll post a graph showing the worldwide before-and-after.