Measuring Startup Speed Correctly

Until recently our state of the art method for measuring startup was to subtract a timestamp passed via commandline from a new Date() timestamp within a <script> tag. Vlad pioneered this approach, me and others adopted it.

Turns out there are two problems with this approach:

  1. It is cumbersome, especially on Windows where there is no easy way to pass a timestamp via the commandline.
  2. It is wrong. Turns out that Firefox starts loading web pages before the UI is shown. One can’t be sure that the page being loaded is within a visible browser

Our oldest startup benchmark, ts,  has been gathering wrong numbers all along.  This resulted in a class of perverse optimizations that decreased the ts number, but increased the time taken for UI to appear (ie bug 641691). The new tpaint (bug 612190) benchmark should should address this. On my machine measuring pageload vs paint-time results in a 50-100ms difference. See the graph server for more data.

This is why AMO’s complicated method of measuring startup is wrong. Please use our shiny new about:startup extension or if you absolutely want to avoid adding any overhead use getStartupInfo API directly.

4 comments

  1. With Firefox 3.6, I was experiencing low startup and veeeeeeery slow closing. Firefox 4 has definitely improved the second and I think it also improved the first. However stopping can sometimes still take a long time. If I do not wait for Firefox before stopping the computer, then I will get the Firefox crash startup where FIrefox offers me to select a set o ftab to open because Firefox did not close properly last time.

  2. Thanks. I actually tried verifying Talos numbers with about:startup a week ago and the extension (as well as the getStartupInfo API) works great.

    One side effect of the crazy optimizations is apparently that Talos’ ts_shutdown numbers are all wrong as I noticed in https://bugzilla.mozilla.org/show_bug.cgi?id=648749 (my patch there fixes the issue). Talos will try to close the browser immediately when the first page loads – but browser initialization is apparently not finished at this point, consequently you get ts_shutdown numbers that are too high.

  3. I’m with djano big time on this one. I haven’t measured it, but closing Firefox can sometimes take in excess of 2 minutes. During that time, trying to start Firefox is completely impossible — ergo, my “experienced” start time blows out to over 2 minutes in those cases. And I never, ever dare minimize Firefox and leave it in the background. If I do, the only way I can make Firefox responsive again within a sensible amount of time is to kill it and restart. (Note: I regularly use about 50 tabs.)

  4. Hi Voracity, so I am not the only one.

    I too have about a lot of tabs, 15 tabs open all the time (sure I’ll read them one day), plus a lot more of them in Panorama (40?).

    Also note that when my machine hibernates and I bring it back up to life, Firefox is freezing for several minutes (I cannot use it, the tabs disappear and the address bar look weird) until it is responsive again (reacting to mouse overs, or clicks, …).

    Good luck with all of that Tara, the work you are doing and sharing with us is enlightening and unusual, which makes it interesting.