{"id":166,"date":"2013-04-18T17:04:38","date_gmt":"2013-04-18T21:04:38","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=166"},"modified":"2013-04-18T17:04:38","modified_gmt":"2013-04-18T21:04:38","slug":"cold-page-load-os-x-10-7-and-talos","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2013\/04\/18\/cold-page-load-os-x-10-7-and-talos\/","title":{"rendered":"cold page load, OS X 10.7, and talos"},"content":{"rendered":"

I’ve been looking at some Talos pageload benchmark data lately. The intention is gleaning enough information to decide whether having a separate cold page load benchmark is worthwhile<\/a>. Our Talos pageload benchmark loads a number of sites a specified number of times (currently 25), records times for all of those, discards the highest time per page (usually the first page load), and averages everything else.\u00a0 The working hypothesis is that the 1st and\/or 2nd iteration of each site load might be worth tracking separately.<\/p>\n

To that end, I’ve been writing a lot of shell scripts to munge benchmark numbers, running those numbers through gnuplot<\/a>, and staring at the output. One of the graph styles I’ve been using is the Q-Q plot<\/a>, which I use for comparing data like so:<\/p>\n

    \n
  1. Sort the page load times for the 1st iteration;<\/li>\n
  2. Sort the page load times for the nth iteration (usually 2nd, 5th, or 10th, the exact number doesn’t seem to matter a great deal);<\/li>\n
  3. Plot numbers from step 1 vs. numbers from step 2 and eyeball the graph.<\/li>\n<\/ol>\n

    There’s a fair amount of data to crunch (8000 runs times 100 sites times 25 runs divided across 8 operating systems), but the most interesting thing to come out of this experiment so far is consistently slower cold page loads on OS X 10.7 on certain sites.\u00a0 Graphing most sites across OS X 10.6, 10.7, and 10.8, the graphs look like this:<\/p>\n

    \"Q-Q<\/a><\/p>\n

    \"Q-Q<\/a><\/p>\n

    These sorts of graphs are exactly what you’d want to see across OS versions: improving load times along warm page load, cold page load, or both.<\/p>\n

    Before pointing out the sites, it’s worth noting that the pages used in the benchmark are <\/strong>representative<\/strong> of these sites as they appeared ~1 year ago: things may have changed and the pages may have been altered slightly to avoid fetching resources off the network, etc.\u00a0 The sites are:<\/p>\n