Diagnosing Slow Startup

I spent last week trying to reproduce slow startup on Windows. Some users were reporting >30 second startup, supernova_00 has been feeding me xperf traces on IRC reproducing slow startups.

Startup Bugs

Turns out that if a website uses non-standard font names this can trigger Firefox to start parsing every single font on the system, freezing the browser in the meantime. Turns out facebook does this :(. This is now a blocker bug 600713. This bug has the unfortunate effect of overshadowing any startup improvements in Firefox 4.

We have some code to keep our databases in good shape by VACUUMing them. This is getting revamped in Firefox in bug 541373. In the meantime, for many current users performance suffers due to missing vacuums. If you are suffering from slow Firefox startup, and/or slow Awesomebar try this manual vacuum. This helps in older Firefox releases, but in Firefox 4 this has the effect of supercharging the SQLite database performance by switching to 32K pages.

Scareware

Another fun discovery was the effect of anti-virus software(AVG in this case). Like an annoying pet, AVG has to have a sniff and fondle every file that Firefox opens on startup. Apparently this is a feature called on-demand scanning, yuck.

But the fun doesn’t stop there, Windows has a wonderful prefetch mechanism that speeds up app startup. Unfortunately for supernova_00, \Windows\Prefetch just wouldn’t get populated with Firefox info, meaning that Windows wasn’t optimizing Firefox startup. Once I installed AVG, I ran into the same problem. Uninstalling AVG didn’t help. For whatever reason deleting every file in \Windows\Prefetch fixes that problem. For both of us prefetch got repopulated after being cleaned.

XPerf

Microsoft XPerf makes trivial to optimize cold startup. None of the other OSes have precanned analyses showing how much each individual file access is contributing to slow startup.

If you have a startup problem, I’m much more likely to be able to reproduce it if the report comes with an xperf trace. To get xperf run the Microsoft Platform SDK installer, select “Windows Performance Toolkit”.

To record an IO trace:

  1. Reboot
  2. run cmd.exe as Administrator
  3. xperf -on latency+FILE_IO+FILE_IO_INIT+DISK_IO
  4. run Firefox, reproduce the bug
  5. xperf -d report.etl
  6. Run xperf report.etl to view the report.

Click on “IO Counts” or “Hard faults” graph, select “Summary Table”. “IO Time (ms)” is the interesting column there. To get an idea of the sequence of IO operations, export the summary table to .csv and load it in a spreadsheet/grep/whatever. Every Firefox developer should give xperf a try, addon authors are encouraged too.

9 comments

  1. The on-access anti-virus scanning is basically a standard feature of every anti-virus out there, right?

  2. I once rooted my work desktop because the IT director flatly refused to change the “scan every file on every read” behavior to something more sensible (“scan on write”, perhaps?). A project I was working on required me to regularly load and unload various Java runtimes. The draconian antivirus settings turned a 15-second process into a 3-5 minute waiting game. After a few (dozen) iterations I started to realize how much more time I was spending staring at a screen compared to running the same tests on my own hardware.

    Wouldn’t it be funny if Firefox tossed up a scary-looking prompt?

    “You’re computer is slow because of your antivirus settings!!! Click here for more information!!!”

  3. How does someone on a Firefox trunk nightly perform the manual vacuum without a JS console?

  4. Kurt (supernova_00)

    @ Asa:

    1) Type about:config in the location bar and hit enter
    2) Filter for ‘devtools.errorconsole.enabled’
    3) Toggle this preference to true by double-clicking it (I can’t remember if you have to restart or not)
    4) Go to Tools -> Error Console
    5) Paste the code in the ‘code:’ input box
    6) Click the ‘Evaluate’ button

    Firefox may freeze for a few seconds while the vacuum is performed.

  5. Taras, did you investigate fragmentation as a source for slow startup ?

    Report of UltraDefrag on my disk :
    Number 3 culprit :
    121 C:\Program Files\Mozilla Firefox 3.5\xul.dll -
    Number 4 culprit :
    71 C:\Documents and Settings\USER\Local Settings\Application Data\Mozilla\Firefox\Profiles\XXXXX.XXX\startupCache\startupCache.4.little

    And various Firefox profile related file, as well as cache files, are all over the report.

    I think that firefox updates are ugly for fragmentation (and there’s also something very wrong happening for the startupCache).

  6. @David Naylor yes and it’s too bad for us.

    @Andy yeah, that’d be a funny/cool feature. Perhaps someone should do a perf-analysis firefox addon that complains about stuff like that

    @Kurt, thanks for answering Asa’s question.

    @jmdesp, replied in http://blog.mozilla.org/tglek/2010/07/22/file-fragmentation/#comments

    To everyone, sorry for the huge delay in responding.

  7. In my case, Firefox was taking 6-10 seconds to open. I went in to Tools – Add-Ons and disabled the AVG Safe Search and AVG Security Toolbar. After a restart, Firefox would open in less than one second. Repeatable and consistent. After enabling the AVG Security Toolbar, the start up time was about the same maybe a fraction of a second longer. The long start up delay returned after I enabled AVG Safe Search. Be safe or not to be safe… That is the question best thought about before permanently disabling AVG Safe Search.

  8. Asa, I have a Windows batch file that runs the vacuum command for me:

    @FOR %%G IN (*.sqlite) DO ( sqlite3 %%G “VACUUM” )

    Have to have the Windows sqlite3 binaries installed first though. =)