{"id":1617,"date":"2012-07-09T12:13:26","date_gmt":"2012-07-09T01:13:26","guid":{"rendered":"http:\/\/blog.mozilla.org\/nnethercote\/?p=1617"},"modified":"2012-07-10T10:45:55","modified_gmt":"2012-07-09T23:45:55","slug":"how-to-compare-the-memory-efficiency-of-web-browsers","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nnethercote\/2012\/07\/09\/how-to-compare-the-memory-efficiency-of-web-browsers\/","title":{"rendered":"How to Compare the Memory Efficiency of Web Browsers"},"content":{"rendered":"

TL;DR: Cross-browser comparisons of memory consumption should be avoided.\u00a0 If you want to evaluate how efficiently browsers use memory, you should do cross-browser comparisons of performance<\/em> across several machines featuring a range of memory configurations.<\/strong><\/p>\n

Cross-browser Memory Comparisons are Bad<\/h3>\n

Various tech sites periodically compare the performance of browsers.\u00a0 These often involve some cross-browser comparisons of memory efficiency.\u00a0 A typical one would be this:\u00a0 open a bunch of pages in tabs, measure memory consumption, then close all of them except one and wait two minutes, and then measure memory consumption again.\u00a0 Users sometimes do similar testing.<\/p>\n

I think comparisons of memory consumption like these are (a) very difficult to make correctly, and (b) very difficult to interpret meaningfully.\u00a0 I have suggestions below for alternative ways to measure memory efficiency of browsers, but first I’ll explain why I think these comparisons are a bad idea.<\/p>\n

Cross-browser Memory Comparisons are Difficult to Make<\/h4>\n

Getting apples-to-apples comparisons are really difficult.<\/p>\n

    \n
  1. Browser memory measurements aren’t easy.\u00a0 In particular, all browsers use multiple processes, and accounting for shared memory is difficult.<\/li>\n
  2. Browsers are non-deterministic programs, and this can cause wide variation in memory consumption results.\u00a0 In particular, whether or not the JavaScript garbage collector runs can greatly reduce memory consumption.\u00a0 If you get unlucky and the garbage collector runs just after you measure, you’ll get an unfairly high number.<\/li>\n
  3. Browsers can exhibit adaptive memory behaviour.\u00a0 If running on a machine with lots of free RAM, a browser may choose to take advantage of it;\u00a0 if running on a machine with little free RAM, a browser may choose to discard regenerable data more aggressively.<\/li>\n<\/ol>\n

    If you are comparing two versions of the same browser, problems (1) and (3) are avoided, and so if you are careful with problem (2) you can get reasonable results.\u00a0 But comparing different browsers hits all three problems.<\/p>\n

    Indeed, Tom’s Hardware de-emphasized memory consumption measurements in their latest Web Browser Grand Prix<\/a> due to problem (3).\u00a0 Kudos to them!<\/p>\n

    Cross-browser Memory Comparisons are Difficult to Interpret<\/h4>\n

    Even if you could get the measurements right, memory consumption is still not a good thing to compare.\u00a0 Before I can explain why, I’ll introduce a couple of terms.<\/p>\n