mwu's blog
A tale of a fearless coder
Omnijar. How does it work?
August 13th, 2010 by mwu
What is omnijar?
Omnijar is a new packaging format. It’s expected to be enabled by default for Firefox 4.0b5.
The difference that omnijar makes can been seen comparing the file listings of builds before and after omnijar:
Trunk Linux build without omnijar:
firefox/application.ini
firefox/blocklist.xml
firefox/chrome/browser.jar
firefox/chrome/en-US.jar
firefox/chrome/icons/default/default16.png
firefox/chrome/icons/default/default32.png
firefox/chrome/icons/default/default48.png
firefox/chrome/localized.manifest
firefox/chrome.manifest
firefox/chrome/nonlocalized.manifest
firefox/chrome/pippki.jar
firefox/chrome/toolkit.jar
firefox/components/addonManager.js
firefox/components/amContentHandler.js
firefox/components/amWebInstallListener.js
firefox/components/browser.xpt
firefox/components/components.manifest
firefox/components/contentAreaDropListener.js
firefox/components/contentSecurityPolicy.js
firefox/components/crypto-SDR.js
firefox/components/FeedConverter.js
firefox/components/FeedProcessor.js
firefox/components/FeedWriter.js
firefox/components/fuelApplication.js
firefox/components/GPSDGeolocationProvider.js
firefox/components/interfaces.manifest
firefox/components/jsconsole-clhandler.js
firefox/components/libbrowsercomps.so
firefox/components/libdbusservice.so
firefox/components/libmozgnome.so
firefox/components/NetworkGeolocationProvider.js
firefox/components/nsBadCertHandler.js
firefox/components/nsBlocklistService.js
firefox/components/nsBrowserContentHandler.js
firefox/components/nsBrowserGlue.js
firefox/components/nsContentDispatchChooser.js
firefox/components/nsContentPrefService.js
firefox/components/nsDefaultCLH.js
firefox/components/nsDownloadManagerUI.js
firefox/components/nsFilePicker.js
firefox/components/nsFormAutoComplete.js
firefox/components/nsFormHistory.js
firefox/components/nsHandlerService.js
firefox/components/nsHelperAppDlg.js
firefox/components/nsINIProcessor.js
firefox/components/nsLivemarkService.js
firefox/components/nsLoginInfo.js
firefox/components/nsLoginManager.js
firefox/components/nsLoginManagerPrompter.js
firefox/components/nsMicrosummaryService.js
firefox/components/nsPlacesAutoComplete.js
firefox/components/nsPlacesDBFlush.js
firefox/components/nsPlacesExpiration.js
firefox/components/nsPrivateBrowsingService.js
firefox/components/nsPrompter.js
firefox/components/nsProxyAutoConfig.js
firefox/components/nsSafebrowsingApplication.js
firefox/components/nsSearchService.js
firefox/components/nsSearchSuggestions.js
firefox/components/nsSessionStartup.js
firefox/components/nsSessionStore.js
firefox/components/nsSetDefaultBrowser.js
firefox/components/nsSidebar.js
firefox/components/nsTaggingService.js
firefox/components/nsTryToClose.js
firefox/components/nsUpdateService.js
firefox/components/nsUpdateServiceStub.js
firefox/components/nsUpdateTimerManager.js
firefox/components/nsUrlClassifierLib.js
firefox/components/nsUrlClassifierListManager.js
firefox/components/nsURLFormatter.js
firefox/components/nsWebHandlerApp.js
firefox/components/PlacesProtocolHandler.js
firefox/components/storage-Legacy.js
firefox/components/storage-mozStorage.js
firefox/components/txEXSLTRegExFunctions.js
firefox/components/WeaveCrypto.js
firefox/components/Weave.js
firefox/components/WebContentConverter.js
firefox/crashreporter
firefox/crashreporter.ini
firefox/crashreporter-override.ini
firefox/defaults/autoconfig/platform.js
firefox/defaults/autoconfig/prefcalls.js
firefox/defaults/pref/channel-prefs.js
firefox/defaults/pref/firefox-branding.js
firefox/defaults/pref/firefox.js
firefox/defaults/pref/firefox-l10n.js
firefox/defaults/pref/reporter.js
firefox/defaults/pref/services-sync.js
firefox/defaults/profile/bookmarks.html
firefox/defaults/profile/chrome/userChrome-example.css
firefox/defaults/profile/chrome/userContent-example.css
firefox/defaults/profile/localstore.rdf
firefox/defaults/profile/mimeTypes.rdf
firefox/defaults/profile/prefs.js
firefox/dependentlibs.list
firefox/dictionaries/en-US.aff
firefox/dictionaries/en-US.dic
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/icon.png
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/preview.png
firefox/firefox
firefox/firefox-bin
firefox/greprefs.js
firefox/icons/document.png
firefox/icons/mozicon128.png
firefox/icons/updater.png
firefox/libfreebl3.chk
firefox/libfreebl3.so
firefox/libmozalloc.so
firefox/libmozjs.so
firefox/libmozsqlite3.so
firefox/libnspr4.so
firefox/libnss3.so
firefox/libnssckbi.so
firefox/libnssdbm3.chk
firefox/libnssdbm3.so
firefox/libnssutil3.so
firefox/libplc4.so
firefox/libplds4.so
firefox/libsmime3.so
firefox/libsoftokn3.chk
firefox/libsoftokn3.so
firefox/libssl3.so
firefox/libxpcom.so
firefox/libxul.so
firefox/modules/AddonLogging.jsm
firefox/modules/AddonManager.jsm
firefox/modules/AddonRepository.jsm
firefox/modules/AddonUpdateChecker.jsm
firefox/modules/CertUtils.jsm
firefox/modules/CrashSubmit.jsm
firefox/modules/CSPUtils.jsm
firefox/modules/ctypes.jsm
firefox/modules/debug.js
firefox/modules/distribution.js
firefox/modules/DownloadLastDir.jsm
firefox/modules/DownloadPaths.jsm
firefox/modules/DownloadUtils.jsm
firefox/modules/FileUtils.jsm
firefox/modules/Geometry.jsm
firefox/modules/HUDService.jsm
firefox/modules/InlineSpellChecker.jsm
firefox/modules/ISO8601DateUtils.jsm
firefox/modules/LightweightThemeConsumer.jsm
firefox/modules/LightweightThemeManager.jsm
firefox/modules/Microformats.js
firefox/modules/NetUtil.jsm
firefox/modules/NetworkPrioritizer.jsm
firefox/modules/openLocationLastURL.jsm
firefox/modules/PerfMeasurement.jsm
firefox/modules/PlacesDBUtils.jsm
firefox/modules/PlacesUIUtils.jsm
firefox/modules/PlacesUtils.jsm
firefox/modules/PluginProvider.jsm
firefox/modules/PluralForm.jsm
firefox/modules/PopupNotifications.jsm
firefox/modules/Services.jsm
firefox/modules/services-sync/auth.js
firefox/modules/services-sync/base_records/collection.js
firefox/modules/services-sync/base_records/crypto.js
firefox/modules/services-sync/base_records/keys.js
firefox/modules/services-sync/base_records/wbo.js
firefox/modules/services-sync/constants.js
firefox/modules/services-sync/engines/bookmarks.js
firefox/modules/services-sync/engines/clients.js
firefox/modules/services-sync/engines/forms.js
firefox/modules/services-sync/engines/history.js
firefox/modules/services-sync/engines.js
firefox/modules/services-sync/engines/passwords.js
firefox/modules/services-sync/engines/prefs.js
firefox/modules/services-sync/engines/tabs.js
firefox/modules/services-sync/ext/Observers.js
firefox/modules/services-sync/ext/Preferences.js
firefox/modules/services-sync/ext/StringBundle.js
firefox/modules/services-sync/ext/Sync.js
firefox/modules/services-sync/identity.js
firefox/modules/services-sync/log4moz.js
firefox/modules/services-sync/notifications.js
firefox/modules/services-sync/resource.js
firefox/modules/services-sync/service.js
firefox/modules/services-sync/status.js
firefox/modules/services-sync/stores.js
firefox/modules/services-sync/trackers.js
firefox/modules/services-sync/type_records/bookmark.js
firefox/modules/services-sync/type_records/clients.js
firefox/modules/services-sync/type_records/forms.js
firefox/modules/services-sync/type_records/history.js
firefox/modules/services-sync/type_records/passwords.js
firefox/modules/services-sync/type_records/prefs.js
firefox/modules/services-sync/type_records/tabs.js
firefox/modules/services-sync/util.js
firefox/modules/SpatialNavigation.js
firefox/modules/stylePanel.jsm
firefox/modules/tabview/AllTabs.jsm
firefox/modules/tabview/groups.jsm
firefox/modules/tabview/utils.jsm
firefox/modules/utils.js
firefox/modules/WindowDraggingUtils.jsm
firefox/modules/XPCOMUtils.jsm
firefox/modules/XPIProvider.jsm
firefox/mozilla-xremote-client
firefox/platform.ini
firefox/plugin-container
firefox/README.txt
firefox/removed-files
firefox/res/contenteditable.css
firefox/res/designmode.css
firefox/res/dtd/mathml.dtd
firefox/res/dtd/xhtml11.dtd
firefox/res/EditorOverride.css
firefox/res/entityTables/html40Latin1.properties
firefox/res/entityTables/html40Special.properties
firefox/res/entityTables/html40Symbols.properties
firefox/res/entityTables/htmlEntityVersions.properties
firefox/res/entityTables/mathml20.properties
firefox/res/entityTables/transliterate.properties
firefox/res/fonts/mathfont.properties
firefox/res/fonts/mathfontStandardSymbolsL.properties
firefox/res/fonts/mathfontSTIXNonUnicode.properties
firefox/res/fonts/mathfontSTIXSize1.properties
firefox/res/fonts/mathfontSTIXSizeOneSym.properties
firefox/res/fonts/mathfontUnicode.properties
firefox/res/grabber.gif
firefox/res/html/folder.png
firefox/res/langGroups.properties
firefox/res/language.properties
firefox/res/svg.css
firefox/res/table-add-column-after-active.gif
firefox/res/table-add-column-after.gif
firefox/res/table-add-column-after-hover.gif
firefox/res/table-add-column-before-active.gif
firefox/res/table-add-column-before.gif
firefox/res/table-add-column-before-hover.gif
firefox/res/table-add-row-after-active.gif
firefox/res/table-add-row-after.gif
firefox/res/table-add-row-after-hover.gif
firefox/res/table-add-row-before-active.gif
firefox/res/table-add-row-before.gif
firefox/res/table-add-row-before-hover.gif
firefox/res/table-remove-column-active.gif
firefox/res/table-remove-column.gif
firefox/res/table-remove-column-hover.gif
firefox/res/table-remove-row-active.gif
firefox/res/table-remove-row.gif
firefox/res/table-remove-row-hover.gif
firefox/run-mozilla.sh
firefox/searchplugins/amazondotcom.xml
firefox/searchplugins/answers.xml
firefox/searchplugins/creativecommons.xml
firefox/searchplugins/eBay.xml
firefox/searchplugins/google.xml
firefox/searchplugins/wikipedia.xml
firefox/searchplugins/yahoo.xml
firefox/Throbber-small.gif
firefox/update.locale
firefox/updater
firefox/updater.ini
That’s 259 files. The total number of files differ by platform, but not by much.
Trunk Linux build with omnijar:
firefox/application.ini
firefox/blocklist.xml
firefox/chrome/icons/default/default16.png
firefox/chrome/icons/default/default32.png
firefox/chrome/icons/default/default48.png
firefox/chrome.manifest
firefox/components/binary.manifest
firefox/components/libbrowsercomps.so
firefox/components/libdbusservice.so
firefox/components/libmozgnome.so
firefox/crashreporter
firefox/crashreporter.ini
firefox/crashreporter-override.ini
firefox/dependentlibs.list
firefox/dictionaries/en-US.aff
firefox/dictionaries/en-US.dic
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/icon.png
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf
firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/preview.png
firefox/firefox
firefox/firefox-bin
firefox/icons/document.png
firefox/icons/mozicon128.png
firefox/icons/updater.png
firefox/libfreebl3.chk
firefox/libfreebl3.so
firefox/libmozalloc.so
firefox/libmozjs.so
firefox/libmozsqlite3.so
firefox/libnspr4.so
firefox/libnss3.so
firefox/libnssckbi.so
firefox/libnssdbm3.chk
firefox/libnssdbm3.so
firefox/libnssutil3.so
firefox/libplc4.so
firefox/libplds4.so
firefox/libsmime3.so
firefox/libsoftokn3.chk
firefox/libsoftokn3.so
firefox/libssl3.so
firefox/libxpcom.so
firefox/libxul.so
firefox/mozilla-xremote-client
firefox/omni.jar
firefox/platform.ini
firefox/plugin-container
firefox/README.txt
firefox/removed-files
firefox/run-mozilla.sh
firefox/searchplugins/amazondotcom.xml
firefox/searchplugins/answers.xml
firefox/searchplugins/creativecommons.xml
firefox/searchplugins/eBay.xml
firefox/searchplugins/google.xml
firefox/searchplugins/wikipedia.xml
firefox/searchplugins/yahoo.xml
firefox/Throbber-small.gif
firefox/update.locale
firefox/updater
firefox/updater.ini
There are 61 files with omnijar. The omnijar build has 76.4% less files. Or, as I like to think of it, the non-omnijar build has 324.6% more files.
How does it work?
A comparison of different packaging formats in Mozilla
- Flat packaging. The original. Just files in directories.
- Jar packaging. A more sophisticated way. Files in the chrome directory are packed into jars.
- Omnijar packaging. The third generation packaging format. Every file that can be placed into a jar is packed into omni.jar.
All the files that are now gone have been packed into a file named omni.jar.
Why?
It’s faster.
Storing related files in a jar is more efficient than scattering files across the filesystem. Flat packaging causes a sort of high level fragmentation where related files might be stored far apart on the disk and result in wasted time seeking. Keeping everything in a file minimizes fragmentation. Startup time is generally improved. Omnijar also paves the way for other startup optimizations.
It’s useful for developers
Omnijar improves performance for users, but it also provides a handy feature for developers – omnijar packaging can be easily converted into flat packaging. To do so, unzip the omni.jar file and delete it. Any file can then be directly edited.
Developers building Firefox with omnijar enabled (–enable-chrome-format=omni) should know that the build in the dist/bin directory is flat packaged. Files are only packed into the omnijar after packaging with |make package|.
how much faster is this?
[...] This post was mentioned on Twitter by Planet Mozilla, Planet Repeater. Planet Repeater said: Michael Wu: Omnijar. How does it work? http://dlvr.it/3mNnK [...]
It depends on the platform, but I’ve seen 10-20% cold start improvements on osx and windows while linux is a bit less than 10%. On starts with fast start cache invalidation, the improvement is much higher though I don’t remember how much better things got for that case.
You and Taras Glek are heroes for working through this!
It’s not in nightly builds yet.
A suggestion: give us a jar: link to an omni.jar that we can explore, e.g. jar:http://mwu.com/files/omni.jar!/ (Firefox’s ability to browse ZIP file contents via the jar: protocol is a well-kept secret.)
A question: will this mean every .mar update will contain an entire new omni.jar even if only one or two script files change? I wonder if nightly update could be made smart and reliable enough to unpack the existing omni.jar, slot in a new file, and repack it to be bit-for-bit identical with the new build.
Next: you need to train the boot PROM to execute .jar files directly, and get rid of that crufty operating system layer
Well, it wasn’t intentional, but you can browse the omnijar through “resource:/”. Right now that points to the firefox directory, but it’ll point to omni.jar once omnijar is turned on.
The way the zip format is, the impact on mars size should be low as long as the order of files within the omnijar doesn’t change too much. (changes in one file do not affect others in zips) Taras wants to keep the contents of the omni.jar uncompressed, so that could also reduce the size of incremental updates.
We can tweak jars to make them load faster (see Bug 559961) and omnijar makes those tricks more effective.
Are there any gains other than start up?
There may since IO performance should generally be improved. Startup time was the only thing I measured.