I wrote a while ago about the importance of avoiding page faults for browser performance. Despite this, I’ve been focusing specifically on reducing Firefox’s memory usage. This is not a terrible thing; page fault rates and memory usage are obviously strongly linked. But they don’t have perfect correlation. Not all memory reductions will have equal effect on page faults, and you can easily imagine changes that reduce page fault rates — by changing memory layout and access patterns — without reducing memory consumption.
A couple of days ago, Luke Wagner initiated an interesting email conversation with me about his desire for a page fault benchmark, and I want to write about some of the things we discussed.
It’s not obvious how to design a page fault benchmark, and to understand why I need to first talk about more typical time-based benchmarks like SunSpider. SunSpider does the same amount of work every time it runs, and you want it to run as fast as possible. It might take 200ms to run on your beefy desktop machine, 900ms on your netbook, and 2000ms to run on your smartphone. In all cases, you have a useful baseline against which you can measure optimizations. Also, any optimization that reduces the time on one device has a good chance of reducing time on the other devices. The performance curve across devices is fairly flat.
In contrast, if you’re measuring page faults, these things probably won’t be true on a benchmark that does a constant amount of work. If my desktop machine has 16GB of RAM, I’ll probably get close to zero page faults no matter what happens. But on a smartphone with 512MB of RAM, the same benchmark may lead to a page fault death spiral; the number will be enormous, assuming you even bother waiting for it to finish (or the OS doesn’t kill it). And the netbook will probably lie unhelpfully on one side or the other of the cliff in the performance curve. Such a benchmark will be of limited use.
However, maybe we can instead concoct a benchmark that repeats a sequence of interesting operations until a certain number of page faults have occurred. The desktop machine might get 1000 operations, the netbook 400, the smartphone 100. The performance curve is fairly flat again.
The operations should be representative of realistic browsing behaviour. Obviously, the memory consumption has to increase each time you finish a sequence, but you don’t want to just open new pages. A better sequence might look like “open foo.com in a new tab, follow some links, do some interaction, open three child pages, close two of them”.
And it would be interesting to run this test on a range of physical memory sizes, to emulate different machines such as smartphones, netbooks, desktops. Fortunately, you can do this on Linux; I’m not sure about other OSes.
I think a benchmark (or several benchmarks) like this would be challenging but not impossible to create. It would be very valuable, because it measures a metric that directly affects users (page faults) rather than one that indirectly affects them (memory consumption). It would be great to use as the workload under Julian Seward’s VM simulator, in order to find out which parts of the browser are causing page faults. It might make areweslimyet.com catch managers’ eyes as much as arewefastyet.com does. And finally, it would provide an interesting way to compare the memory “usage” (i.e. the stress put on the memory system) of different browsers, in contrast to comparisons of memory consumption which are difficult to interpret meaningfully.