Chromium vs Minefield: Cold startup performance comparison

Hunting Down Mythical “Slowness”

I recently met a developer who used Chromium instead of Firefox. Chromium’s superior startup speed was his reason for using it.This got me excited because said developer was running Linux, so it was relatively easy to measure cold startup and get a complete IO breakdown.

Turned out Firefox took roughly 23 seconds to start. After much cursing about how I’ve never seen Firefox startup this slow, I eventually gave up on figuring out what’s slowing his startup and instead we measured Chromium startup. It also turned out to also be roughly 23 seconds. The super-slow hard drive made everything slow. Turned out Chromium’s superior startup was a myth in this case.

Measuring Startup

As a result of investigating the startup myth above, my kiwi coworkers encouraged me to post a comparison of Chrome/Firefox startup. I am at linuxconf at the moment so I did the comparison on my laptop.

Laptop configuration:

  • Intel(R) Core(TM)2 Duo CPU  L9400 running at 800Mhz to amplify any performance differences.
  • HITACHI HTS722020K9SA00 harddrive for the user profile and browser binaries
  • OCZ Vertex 30GB SSD for system libraries/configuration.
  • Fedora 12, Minefield 20100119 tarball, chromium-4.0.285.0-0.1.20091230svn35370.fc12.i686
  • sudo sync && sudo sysctl -w vm.drop_caches=3 && sudo sysctl -w vm.drop_caches=0 to clear the disk cache inbetween runs

What am I testing? I am measuring the time between invoking the browser until a JavaScript snippet embedded within a basic webpage is executed (ie Vlad’s approach, with a slightly modified startup.html). The above sysctl command clears disk caches, this creates a similar situation to when one turns on the computer and it hasn’t yet loaded all of the browser libraries from disk into memory. This is a blackbox approach to measuring how long it takes from clicking on the browser icon to get an interactive browser.

Firefox commandline: firefox -profile /mnt/startup/profile/firefox  -no-remote file://`pwd`/startup.html#`python -c ‘import time; print int(time.time() * 1000);’`

Chromium commandline: chromium-browser –user-data-dir=/mnt/startup/profile/chrome  file://`pwd`/startup.html#`python -c ‘import time; print int(time.time() * 1000);’`

Both of these tests are done with an empty profile that was populated and has settled after running the browser a few times.

Results

The following numbers are milliseconds reported by the startup.html above.

Running Chromium five times: 4685, 4168, 4222, 4197, 4232

Running Minefield five times: 3155, 3273, 3352, 3311, 3322

I picked Minefield because that’s the browser that I run and the codebase that I focus on. The linux Chromium channel seems to be the closest parallel to Minefield. I did not test on Windows because it is a bit of a nightmare to measure cold startup there.

Conclusion

On my system Minefield is around 30% faster at starting up with  an empty profile than Chromium (the difference is amplified by running the CPU at 800Mhz). For comparison of Minefield against older Firefox versions, see Dietrich’s post.

I suspect that there is a relatively small difference between the two browsers because we are running into the fundamental limitations of loading large applications into memory (my rant).

26 comments

  1. Running the first line of script is not when the browser *feels* loaded though.

    In Chrome you could be halfway through typing a search before the js engine has even loaded.

    You’d have a hard time getting Firefox to do that. =D

  2. On Windows, Minefield takes AT LEAST fifteen seconds to pop up its window, and an additional 3-5 seconds to render the chrome properly. Chromium, on the other hand, never fails to clock less than 5 seconds.

    Not a Linux user, so no comments about that platform.

  3. Thank you Tara for the data and Fowl to make the fundamental comment.

    It is good that a coldstart of Firefox is very close than the one of Chromium. Though this must be confirmed, and that work on improving the time to start the app must still be focused on, the fundamental point is even if Firefox is as quick, if not quicker, than Chromium, the user perceive it the other way around.

    ‘Why’ is the question to answer now: is it because the main window start up quicker, is it because the UI is (partially?) usable earlier so that the user can start to work earlier (like when logging to Windows it can start entering his username and password before Windows is fully loade)?

    Once the why is answered, actions can be planned to make Firefox feel quicker at startup, it won’t be easy I predict, though.

  4. Also, for reasons of comparison, you probably used clean profiles. I don’t know about Chrome’s performance after a few months, but I do know that ‘dirty’ profiles in Firefox can get extremely slow.
    That is where a lot of benefit could be gained, I think. With a clean profile Firefox performance really is quite good, but the longer you use it, the worse it gets.

  5. When you mentioned Chromium’s startup turned out to be equally 23 seconds, I scrolled down to see if used computer uses SSD. It’s quite known issue of SSD drives, they slow down after some time.
    Thanks Tara for writing this.

  6. Please, please, please, test and optimize against NON-EMPTY profiles.

    Like Pino said, non-clean firefox profiles get super slow… yet that’s what your users have got.

    Testing with clean profiles is a bit meaningless :(

    Nobody has an empty profile ! Or not for long ! That codepath is fast enough !

  7. In the final analysis, what does it matter how long it takes to load as load? A few seconds — even as many as 30 — isn’t nearly as important as how well it works especially since a browser is usually kept available all the time.

  8. Yep, dirty profiles and add-ons are Firefox’s major problems when it comes to startup speed (and UI speed, for that matter).

    Firefox startup had recently blown out to about 30 seconds on my laptop. I disabled all add-ons, cleaned the profile, installed Fx 3.6 and it’s down to about 6 now (on Windows; Chrome’s still a bit quicker, but not much now.)

    UI is much zippier too. My new window time had increased to about 8 seconds! After disabling add-ons, and creating a new profile, this became instant. I’ve since re-added Firebug, Adblock and Weave — the first two means a new window takes about 0.5 second rather than being instant.

    I’m guessing that almost all of the users who have recently come to believe that Firefox is bloated are seeing the effects of dirty profiles and add-ons. So I’d suggest working to a) make startup/UI speed independent of the profile and b) try to make add-ons much more lightweight (Jetpack, maybe? ;) ).

    Also, to make people more aware of the effects of their add-ons, why not show the time taken for Firefox to startup in the status bar? That way, people will take note when they first install Firefox, and can watch in wonder as it gradually slows down over time (especially when they install a new add-on)…

  9. I use Firefox on both Windows and Linux (on the same computer). Cold startup is much much worse on Windows than on Linux. It takes a minute for the browser to start after boot.

    You tested this with a 7200 rpm hard drive, so you probably don’t see as great a difference as we do with slower 5400 (or some even with 4200 rpm) laptop drives. If startup were CPU limited there wouldn’t be such a huge difference between cold and warm starts …

    Please do focus on Windows, because that’s where the big problems are, not Linux. The pain of slow startup is not a “myth”. Chrome (on Windows) starts in 10-15 seconds after boot here, and is immediately usable after the UI shows up. That’s 4 times faster than Firefox. Firefox is not immediately usable after the UI appears.

  10. Seems like you’d want to use Test Pilot data to build a ‘median’ bookmarks/history profile to get more realistic information. (Test Pilot information on the media number of bookmarks and folders is here: https://testpilot.mozillalabs.com/testcases/a-week-life/results.html )

    Still, given that it is really hard to test when it ‘feels’ like it has come up (which is really the right test, per Fowl’s comment) this is still useful and interesting data to have. Would be interesting to compare 3.0, 3.5, and 3.6 using this test as well.

  11. I use Firefox at my workplace. Startup is extremely slow. It might be slower than usual because my profile is served from a network share, and possibly because our internet connection is reasonably slow and our firewall likes to scan all add-ons for malware.

    I don’t have timing information to hand right now. But to give an idea of relative performance, if I’m waiting for Firefox to open I will often get impatient and pop Chrome open, surf to the page I was looking for, read that page, then maybe read another page, all before Firefox appears!

  12. I think the comparison needs to be done with a range of tabs loading at startup (usually about 100 for me…). Being able to manipulate the UI in meaningful ways (like typing searches) before content loads is important to how things feel.

    FYI: I use Firefox 95% of the time and Chrome just to do quick tasks when Firefox is running slowly from having 200 tabs open. I feel that Firefox, because it doesn’t (by default) shrink the tab size to nothing, encourages me to have more tabs open than does Chrome. In Chrome I tend to top out at 30 tabs in two windows. In Firefox I top out at … well, when it becomes unbearable. I wonder if people who use Firefox tend to have more tabs open. (Also, I find that when you have 40 tabs open for a while in Chrome you can feel more lag than under Fx).

  13. One weakness of firefox startup is definitively that it tries to load all saved tabs simultaneously. This is a big mistake. The one that will be shown to the user should have the absolute priority, and the others loaded later, in the order they have last been used, and probably no more than 3 or 4 loaded simultaneously. A astute thing to do would be also, after the 20 first ones last shown, to *stop* preloading the tabs. Maybe still load the bytes from the network, but to parse them only when the user actually selects the tab.

  14. you can get time in nanoseconds using:
    date +%s%N

    and you can modify javascript to get milliseconds from nanoseconds.

    diff = t – (parseInt(h)/1000000);

  15. I think Firefox is fast by default, but becomes slow just by using it. This should be addressed.

    Startup time with a clean profile is important. And I think Firefox is good enough in this department.

    But there is also another equally important part, which is the trend. This thread and the whole Internet is plagued by testimonies of people telling that their Firefox is slow at startup. “I just tested a clean profile and works OK; it is your fault.” is not an answer to this.

    Currently, Firefox converges at auto-destruction. And this is Mozilla’s fault.

    We talking here are nerds who know what a profile is and how to keep it tight. But the real world people don’t. By assigning to them the responsibility of keeping their profiles clean the only you get is them switching to Google Chrome.

  16. Nobody is assigning blame at users for making their profiles dirty. We are actively working on making startup equally fast in the dirty profile case.

    We are also improving our io patterns so Firefox is more responsive during regular browsing.

    Firefox is a big program, there are a lot of pieces that must be correctly arranged for it to perform as well as it can, it takes time.

  17. @Taras Thank you for your concern.

    Yeah, I’ve noticed more responsiveness in Firefox 4 beta 1 and I appreciate it greatly.

    Also I’ve noticed beta 1 being super-stable. I installed it the day after release and so far it has not crashed. No once. I mean with my profile, my add-ons (6 yet enabled), and my 6 to 8 hours of use per day.

    I’m sure all together is not a coincidence. So congratulations on the good work.

  18. SQLite fragmentation. Can it be the cause of that trend?

    After reading these pages:
    http://www.gettingclever.com/2008/06/vacuum-your-firefox-3.html
    http://www.downloadsquad.com/2009/08/24/speed-up-firefox-by-cleaning-out-your-sqlite-databases/
    https://developer.mozilla.org/en/Storage/Performance#Vacuuming_and_zero-fill
    https://addons.mozilla.org/en-US/firefox/addon/11198/
    https://addons.mozilla.org/en-US/firefox/addon/13824/
    https://addons.mozilla.org/en-US/firefox/addon/13878/
    http://www.sqlite.org/lang_vacuum.html

    The fragmentation of the SQLite database seems to be a significant factor of slowness.

    Keeping browsing history causes a constant ratio of inserts and deletes. Eventually this may lead to a total fragmentation of that database.

    On the other hand, Firefox keeps 90 days of browsing history by default. And average users don’t tweak their preferences almost at all.

    As a result, average users end having 90 days of totally fragmented browsing history in their disks.

    Loading that on start-up is painfully slow.

    And that is what Firefox does.

    In one sentence, assessing all the above as true enough, it is probable that the average user is experiencing very slow cold start-ups mainly due to Firefox loading a significantly large, very fragmented, SQLite database.

    Two complementary things can be made to avoid this scenario:
    1.- Don’t load that database on start-up. Or only the 10% of that data that is useful for 90% of the cases. The rest should be loaded only later.
    2.- Run database defragmentation automatically periodically. For example when inactivity is detected.
    Design details can be further discussed for both solutions.

  19. Oh please!! I’ll take Firefox over Chromium any time :)

  20. Tara, have read your article with interest.

    However, have tried all kinds of tips and tricks I found and still found Firefox to be extremely slow on start up.
    I have a normal cold start of around 15 sec (min 10 sec, max 19 sec).
    I have only 5 addons, and 4 GM scripts.
    They are all approved by “Add-on Compatibility Reporter 0.7″

    Here is what I found what speed up at least on my computer:

    Tip no 1:
    I normally use http://www.google.se as my homepage, having that a cold stat takes approx. 15 sec.
    If I choose to have Minefield’s start page (about:home) as my homepage I can reduce the cold start to 6-7 sec
    Don’t ask me why – it just reduce the load time with more than half…

    Tip no 2:
    And here’s the big win, if I do like this below I have a COLD START OF ONLY 2-3 SECONDS.

    1. Create a shortcut of the Firefox icon, it’s OK to change the name such as “Firefox prestart”.
    2. Select the shortcut icon, right-click, choose properties, find the box target (where you probably read this: “C:\Program Files (x86)\Firefox\ firefox.exe”)
    3. Add the following parameter -silent afterwards, do not forget a space after the quotation mark. Ie “C:\Program Files (x86)\Firefox\firefox.exe” -silent
    4. Click OK to save the change.
    5. Then drag the icon to the Autostart folder (see “All Programs”, “Start Button”) and drop the icon there.
    6. Restart the computer.

    Now, Firefox will be loaded when Windows starts up. But after being “booted” minimized, Firefox will be closed down again and do not stay in memory.
    That process adds approx. 2 sec to your total windows boot time. But as a normal Win boot takes 50-55 sec at my computer, it’s barely notable.

    The result is that you can “cold start” Firefox as if it were a “Warm start”.
    And this without using other programs.

    On my computer, I can now cold start Firefox in 2-3 seconds, not bad compared to before 10-18 seconds …..
    Do you have an SSD, Firefox will probably start more or less instantly ….

    Check it out :)

    And Tara, if you find my suggestions of any interest, please forward it to Mozilla’s developers for further investigation.

  21. @Superputte

    My name is “Taras”. This is a good tip, but there is something seriously wrong with your setup. Your 2-3second warm startup is about 4-6times slower than it should be. Addon compatibility reporter thing doesn’t actually test startup performance, so it’s probably some addon causing super-slow startup speed.

  22. I have tried “everything”
    Incl new profile, ramdisk, reinstallation, no addons, disabled antivirus/firewall (NOD32), etc without any progress.
    I have even reinstalled win 7 (x64 ultimate) with nothing but latest FF/minefield prebeta 8 installed – no change……

    I can improve the coldstart a nice bit from approx 12-15 sec to 6-8 sec by change my default homepage from http://www.google.se to the preinstalled (?) by Firefox about:home.
    I find that odd… but this is the only thing I have found reduce the cold start.

    Nevertheless, there is an odd thing at my computer……

    If I use the ResMon I can when I right-click at firefox.exe, and “analyse process”, I get following comment:
    “One or more threads of firefox.exe is waiting to finish network I/O”
    Underneath in the box says; “firefox.exe (Processor-ID: 3064) Thread: 3332″
    (the numbers above are always random, if I check next time, there is some other numbers)

    Have tried to google what that mean, but did not found any answer.

    Only Firefox have this comment, no other process. If I run other browsers, they doesn’t neither have this comment.
    My network and drivers, etc seams to be correct installed.

    Except the slow load time, FF is acting normal.
    It’s perceived snappy as IE9 or Chrome.
    Also at benchmarks I get nice results.
    I e Sunspider 0.9 = 220 ms (compared with Chrome 200), etc – completely normal.

    But the slow startup does nagging me…..

    I can if of interest post printscreens, etc if you tell me what kind of information you would like to investigate.

    Rgds

  23. I forgot to mentioned that I have tried also the 64-bit version of Minefield.

    Compared latest builds with the 32 bit version, the 64 version starts 1-2 sec faster at cold start, 0,5-1 sec faster at warm start.
    The network I/O thread (as I wrote abound before, is still there.
    Generally it feels a little bit snappier when browsing, but benchmarks are approx 10% worse than the 32- bit version (which I think is normal).

  24. Those that are suffering slow startup on windows, please send me an xperf trace. See instructions in https://blog.mozilla.org/tglek/2010/10/04/diagnosing-slow-startup/

  25. Like SuperPutte, I’ve been experiencing the same issues with FF, only they don’t seem to be at startup, they happpen every now and then.

    I’ve stripped out every extension, run in safe mode, it still happens.

    And while it’s been mostly affecting FF, it’s also been affecting Chrome, Digsby, and even Explorer and my text editor Uedit32. My install is about two years old, so I’m thinking about blowing the whole C: partition away and starting from scratch again.

    This is the only hit when I search for “resmon” “waiting to finish network I/O”

  26. Fixed!

    Couldn’t get xperf to run, so I googled the error message.

    That article told me what the problem was (I was running ProcExp at the same time, a no-no). It also described my problem (sort of, there was no indication that my CPU core was maxed out at the time), but it referenced a KB.

    The easiest thing for me in the KB was to go to the advanced properties of my ethernet adapter, and turn off the LMHOSTS check mark.

    Haven’t had the problem since!