Snappy, July 19: Telemetry Experiments

For the in-progress work and minor changes that landed see non-meeting notes for this week.

Jeff Muizelaar wrote an interesting blog post work involved in a tab switch on Mac.

Windows Prefetch: Experimental Data vs Reality

I once discovered that Windows Prefetch can adversely affect application startup times, bug 627591. Certain machines were showing performance to be much better with Windows prefetch disabled and using my “manual” dll preload code to warm up the cache. Manual dll preload is a win for loading large applications because it causes xul.dll to be read in sequentially rather than randomly via page-in (see my blog posts from 2010 for details of startup IO uglyness). Unfortunately Windows Prefetch + my preload code measured as a net regression. I found a weird API that seemed to return 0 when prefetch was broken and guarded preload on that.

We have recently backed out above heuristic based on a telemetry study in bug 757215. Perhaps this is why our startup numbers have started getting better in Firefox 16?

Brian Bondy setup a telemetry startup trial to randomly delete prefetch, turn on dll preload. Last week Saptarshi Guha crunched some telemetry numbers, see this bugzilla comment. Turned out Windows Prefetch is a huge win and dll preload is a tiny incremental improvement on top of that (rather than being a regression).

Moral of the story is: do not rely on manual performance testing for workloads involving large amounts of IO. Simulating a “typical” Windows machine is extremely hard without getting noisy numbers. Effort is better spent on analyzing noisy real-world numbers and running experiments in the wild.

Comments are closed.