Firefox 4: jar jar jar

Opening files is relatively expensive. There is a small syscall overhead and a higher overhead of fetching data from disk. Depending on physical data layout and disk type, this can leave modern CPUs twiddling their thumbs for a long time while the disk skips around fetching all of the different file pieces.

Optimization #1: Fewer naked files

About two years ago I started gathering naked files on disk and shoving them into jars (eg bug 508421). We made jar reading as efficient as possible by cleaning up code and switching to mmap. Eventually all application data files read from disk during “normal” startup ended up in jars. Unfortunately we ended up with four jars (toolkit, chrome + 2 locale jars), which felt silly. Due to limitations in XPCOM, a lot of naked files were still read from disk on version upgrades and extension installation.

Optimization #2: One jar to rule them all

Recently Michael Wu unleashed a can of omnijar whoopass. This was a massive effort driven by Android packaging requirements. Now application startup data is always being read from one file. This implies better data locality, less seeking, less waiting. One benefit of packing files tightly is that the OS speculatively reads data from disk in chunks that are usually larger than what the application requests. This makes reading nearby files free. Unfortunately there was no good way to predict the order that files will be accessed in without actually running Firefox, so there was more room for improvement.

Optimization #3: Optimized jar layout

So now that all of our data was in one file, the next logical step was to pack it intelligently. The only way to do this is to profile Firefox startup and then order the jar according to that. Unfortunately even once one lays out all of the jar entries sequentially we were still doing our io suboptimally. This was due to the fact that the zip index (jars are zip files) is traditionally located on the end of the file. Wikipedia entry has pictures to illustrate this.

In order to maximize readahead benefits and minimize disk seeks it would be nice to have the file index in the front of the file. So I changed our zip layout from

<entry1><entry2>…<entryN><central directory><end of central directory>

to

<offset of the last entry read on startup><central directory><end of central directory><entry1><entry2>…<entryN><end of central directory>

So all I did was change the offset in <end of central directory> to always be 4 (it can’t be 0 because anal zip programs balk at “NULL” central directory offsets). Then I added a second identical <end of central directory> entry to keep the the rule that the central directory is always followed by one. I also used the extra space forced upon me by overly vigilant zip programs to store a number indicating how much data we can preread on startup.

This yielded a 2-3x reduction in disk io over an unoptimized omnijar. This is on top of a >30-100x reduction achieved by going from naked files to omnijar.

The downside of my interpretation of the zip spec is that some zip programs expect zip files to be more rigid than the spec allows. Older versions of Firefox, Microsoft zip support in windows, WinRAR, unix zip programs, etc accept my optimized jars. 7zip, broken antivirus (it’s a security risk to be overly picky) fail.

Trivia: this isn’t the first time we got tripped up by picky zip reading code. For example, the Android apk reader irritatingly insists at having a zip entry at byte zero of an Android package. This means that one can’t use apks to do the Android equivalent of self-extracting .exe files on Windows. Michael Wu is writing a custom library loader to deal with that :)

Optimization #4: More Omnijar

Feeling that omnijar wasn’t awesome enough, Michael Wu went ahead and omnijared extensions. Most extensions will no longer need to be unpacked from xpi files. This also means that extension authors can opt to use the optimized jar format above to further speed up Firefox startup.

Other jar optimizations

Switching to jars via startup cache will allow us to further optimize our first startup. There is option of halving our jar IO further by actually making use of that readahead integer I added to optimized jars.

16 comments

  1. I wonder if people will finally stop complaining about sluggishness and bloat of Firefox with version 4.

  2. Kurt (supernova_00)

    Great work!

  3. “There is option of halving our jar IO further by actually making use of that readahead integer I added to optimized jars.”

    Where’s the bug tracking this?

    Great work, btw. :D

  4. We may want to consider packaging the user’s profile into a single jar file, I have some notes here: https://bugzilla.mozilla.org/show_bug.cgi?id=592479

  5. Congratulations on work well done! I am afraid however that I found one more application that is unable to open such JAR files – Total Commander. This makes looking around in Firefox code a lot more complicated for me :-(

  6. Correction to the above – it can open them, it will simply not recognize them as ZIP files automatically. Meaning that I have to rename them first. Ok, can live with that…

  7. Two things:

    1. No matter how you paint it, these “optimized” JAR files aren’t ZIP files anymore. The spec is quite clear, the central directory comes at the end.

    There’s nothing wrong with what you’re doing, but instead of trying to pretend they’re ZIPs and hoping for apps not to complain, you’d be much better off changing their extension and providing tools to optimize and de-optimize them manually.

    I’m pretty sure 7-Zip for example will never supports these files, as it’s by design exhaustively picky.

    2. I was experiencing slow cold startup in FX (specially slow compared with other browsers) and decided to take a look at it. Even with a clean profile it was still quite a bit slower. One of the things I noticed is that it loads 18 DLL files of its own on startup (all sitting in the Firefox/Minefield directory). This is quite bad, not just for the number of files, but also on Windows you usually have got security software (Windows Defender comes by default, and usually some AV too).

    On my machine, scanning a file takes about 50ms and doesn’t increase much with file size. So for Firefox that’s a second of startup time. Note that this doesn’t show in warm tests as security software usually caches on-demand scan results and won’t reanalyze until restart or until the definition files are updated.

    Certain other browser uses just 2 DLLs on startup and loads some others on demand when needed (video playback, webgl…), and it seems like that’s working a lot better for them.

    I searched and found some bugs about Linux but nothing about Windows. Since there’s one big DLL (xul.dll) and a lot of tiny ones (which are unconditionally loaded), it appears it’d make sense to try and pack everything together.

  8. Ben,
    You are right about shipping to many dlls. That’s what https://bugzilla.mozilla.org/show_bug.cgi?id=561842 is about. Unfortunately, we we unable to finish that in time for ff4, should get that in a subsequent release.

    I would love to get some more data on what effect antiviral software has on startup.

  9. Well, as Ben has pointed out, these jars aren’t zips anymore–at least in a strict sense. But since you changed their layout for a reason, it would be nice if you could let us know how (–> with what zipper and which options) you achieved this.

    Cheers, Chris

  10. Hi Taras,

    thanks a lot for the script–works fine.

    BTW: Am I right that the jars of the latest Linux nightlies are not (yet) optimized? Are they going to be?

    Cheers, Chris

  11. Can you explain the purpose of the log file used by that python script (I don’t know python). The script doesn’t work unless I create an empty omni.jar.log file but not exactly sure if there is suppose to be something in it first.
    Thanks

  12. great script.
    now only if we had the faintest idea of the arguments required – and their effect

  13. With a little experimentation I was able to use optimizejars.py to unpack (“deoptimize”) your omni.jar files to normal .zip files.
    As well as to repack (“optimize”) the resulting unmodified .zip files, even after renaming them.
    However it does not repack modified zip files. At least those with the modifications that I made.
    These same modifications worked very well with previous versions of mozilla (seamonkey). (There are generally only very minor changes with different versions, as was the case this time.)

    Error messages include :
    —-
    import: unable to grab mouse `': Ressource temporairement non disponible @ error/xwindow.c/XSelectWindow/9048.
    ./optimizejars.py: ligne40: local_file_header : commande introuvable
    ./optimizejars.py: ligne41: Erreur de syntaxe près du symbole inattendu « , »
    ./optimizejars.py: ligne41: ` (“signature”, “uint32″),’
    —-

    maybe due to the statement :
    import sys, os, subprocess, struct

    not finding one of the referenced routines ?
    or requiring some sort of signature in the (unsigned) zip file ?
    There was no signature requirement in previous versions, in the .jar files modified.
    The mouse cursor was changed as usual during processing.

  14. This ‘ugly’ ZIP-Format hinders contributors.

    How much benefit/speed brings it to the end user when Startup Cache is used?

  15. This change is being discussed at https://bugzilla.mozilla.org/show_bug.cgi?id=605524, including the fact the resulting file is not a valid ZIP file. It’s not a problem with the tools, the optimized format is not a valid ZIP.

    Why not stuff the index inside an initial file and also leave the index at the end? That will make it ZIP valid while still giving you an index at the front also.

    Or, you can also just accept its not a JAR file anymore and change the suffix.

    Folks can go to the above bugzilla link to vote on fixing this…