Cleaning Up Startup Disk IO
August 20th, 2009 by tglek
Maintaining a module, killing off another one
I was granted ownership of the jar module. Today, I resumed my quest to kill off the barely limping stopwatch module. Together with nuking STANDALONE mode in jar stuff, I will have landed 75KB worth of -ve diffs this month. It feels so good to delete code.
IO Report
Currently I am focusing on application IO (excluding libraries and IO caused by libraries).
From my empirical measurements, opening individual files on a 7200RPM hard drive costs around 0-40ms. This is on Linux. I presume files open quickly when they are located near previously opened files and slower if a full disk seek is required for them. Combining files is usually a significant win in terms of throughput. It turns out that even warm starts and reading from SSDs can benefit from combined IO. Currently small file throughput ranges from <1KB/s to <200KB/s for files < 500K. Combining files into memory mapped jars bumps that up to 1-1.5MB/s (currently jar files are relatively small, making them responsible for a higher proportion of IO should boost that further).
The biggest gains are to be had on Windows Mobile where almost every seemingly trivial filesystem operation takes 2-3ms.
I would like to reduce the number of files read on startup to a dozen or so to be able to crank up disk throughput. Unfortunately, there is a lot to be done, I could use a great deal of help.
Below is a long list of files gathered by stracing firefox-bin, and what I know about them:
/home/taras/builds/minefield/dist/bin/application.ini
/home/taras/builds/minefield/dist/bin/browserconfig.properties
/home/taras/builds/minefield/dist/bin/platform.ini
Haven’t investigated these yet. Perhaps, instead of reading these files should be compiling this info into the xulrunner launcher app?
Update: ted says application.ini used to be compiled in: bug 383167
/home/taras/builds/minefield/dist/bin/plugins
I’m ignoring plugins at the moment.
/home/taras/builds/minefield/dist/bin/chrome
/home/taras/builds/minefield/dist/bin/chrome/browser.jar
/home/taras/builds/minefield/dist/bin/chrome/browser.manifest
/home/taras/builds/minefield/dist/bin/chrome/comm.manifest
/home/taras/builds/minefield/dist/bin/chrome/en-US.jar
/home/taras/builds/minefield/dist/bin/chrome/en-US.manifest
/home/taras/builds/minefield/dist/bin/chrome/pageloader.manifest
/home/taras/builds/minefield/dist/bin/chrome/pippki.manifest
/home/taras/builds/minefield/dist/bin/chrome/reftest.manifest
/home/taras/builds/minefield/dist/bin/chrome/toolkit.jar
/home/taras/builds/minefield/dist/bin/chrome/toolkit.manifest
/home/taras/builds/minefield/dist/bin/chrome/xslt-qa.manifest
The .manifest files describe how to find stuff in chrome jars. Parsing .manifest files is inefficient due to disk seeks. Additionally, current .manifest parsing code appears to be slow on mobile devices. bug 506392
I would like to move .manifest files within jars(and look for manifests within jars when a manifest isn’t found alongside the .jar in the filesystem)
.jar files are a little bit of a cpu hog when it comes to parse the zip file index. This can be done lazily: bug 511754
/home/taras/builds/minefield/dist/bin/chrome/icons/default/default16.png
/home/taras/builds/minefield/dist/bin/chrome/icons/default/default32.png
/home/taras/builds/minefield/dist/bin/chrome/icons/default/default48.png
These icons are parsed by gnome libs 3 times during startup. I wonder if we are initializing some gnome thing 3 times?
/home/taras/builds/minefield/dist/bin/components
/home/taras/builds/minefield/dist/bin/components/aboutCertError.js
/home/taras/builds/minefield/dist/bin/components/aboutPrivateBrowsing.js
/home/taras/builds/minefield/dist/bin/components/aboutRights.js
/home/taras/builds/minefield/dist/bin/components/aboutRobots.js
/home/taras/builds/minefield/dist/bin/components/aboutSessionRestore.js
/home/taras/builds/minefield/dist/bin/components/FeedConverter.js
/home/taras/builds/minefield/dist/bin/components/FeedWriter.js
/home/taras/builds/minefield/dist/bin/components/fuelApplication.js
/home/taras/builds/minefield/dist/bin/components/httpd.js
/home/taras/builds/minefield/dist/bin/components/NetworkGeolocationProvider.js
/home/taras/builds/minefield/dist/bin/components/nsAddonRepository.js
/home/taras/builds/minefield/dist/bin/components/nsBadCertHandler.js
/home/taras/builds/minefield/dist/bin/components/nsBlocklistService.js
/home/taras/builds/minefield/dist/bin/components/nsBrowserContentHandler.js
/home/taras/builds/minefield/dist/bin/components/nsBrowserGlue.js
/home/taras/builds/minefield/dist/bin/components/nsContentDispatchChooser.js
/home/taras/builds/minefield/dist/bin/components/nsContentPrefService.js
/home/taras/builds/minefield/dist/bin/components/nsDefaultCLH.js
/home/taras/builds/minefield/dist/bin/components/nsDownloadManagerUI.js
/home/taras/builds/minefield/dist/bin/components/nsExtensionManager.js
/home/taras/builds/minefield/dist/bin/components/nsFilePicker.js
/home/taras/builds/minefield/dist/bin/components/nsFormAutoComplete.js
/home/taras/builds/minefield/dist/bin/components/nsHandlerService.js
/home/taras/builds/minefield/dist/bin/components/nsHelperAppDlg.js
/home/taras/builds/minefield/dist/bin/components/nsLivemarkService.js
/home/taras/builds/minefield/dist/bin/components/nsLoginInfo.js
/home/taras/builds/minefield/dist/bin/components/nsLoginManager.js
/home/taras/builds/minefield/dist/bin/components/nsLoginManagerPrompter.js
/home/taras/builds/minefield/dist/bin/components/nsMicrosummaryService.js
/home/taras/builds/minefield/dist/bin/components/nsPlacesAutoComplete.js
/home/taras/builds/minefield/dist/bin/components/nsPlacesDBFlush.js
/home/taras/builds/minefield/dist/bin/components/nsPlacesTransactionsService.js
/home/taras/builds/minefield/dist/bin/components/nsPrivateBrowsingService.js
/home/taras/builds/minefield/dist/bin/components/nsProgressDialog.js
/home/taras/builds/minefield/dist/bin/components/nsProxyAutoConfig.js
/home/taras/builds/minefield/dist/bin/components/nsSafebrowsingApplication.js
/home/taras/builds/minefield/dist/bin/components/nsSample.js
/home/taras/builds/minefield/dist/bin/components/nsSearchService.js
/home/taras/builds/minefield/dist/bin/components/nsSearchSuggestions.js
/home/taras/builds/minefield/dist/bin/components/nsSessionStartup.js
/home/taras/builds/minefield/dist/bin/components/nsSessionStore.js
/home/taras/builds/minefield/dist/bin/components/nsSetDefaultBrowser.js
/home/taras/builds/minefield/dist/bin/components/nsSidebar.js
/home/taras/builds/minefield/dist/bin/components/nsTaggingService.js
/home/taras/builds/minefield/dist/bin/components/nsTryToClose.js
/home/taras/builds/minefield/dist/bin/components/nsUpdateService.js
/home/taras/builds/minefield/dist/bin/components/nsUrlClassifierLib.js
/home/taras/builds/minefield/dist/bin/components/nsUrlClassifierListManager.js
/home/taras/builds/minefield/dist/bin/components/nsURLFormatter.js
/home/taras/builds/minefield/dist/bin/components/nsWebHandlerApp.js
/home/taras/builds/minefield/dist/bin/components/pluginGlue.js
/home/taras/builds/minefield/dist/bin/components/reftest-cmdline.js
/home/taras/builds/minefield/dist/bin/components/storage-Legacy.js
/home/taras/builds/minefield/dist/bin/components/storage-mozStorage.js
/home/taras/builds/minefield/dist/bin/components/tp-cmdline.js
/home/taras/builds/minefield/dist/bin/components/txEXSLTRegExFunctions.js
/home/taras/builds/minefield/dist/bin/components/WebContentConverter.js
Above files are only read in during the “slow” startup and are “fastloaded” afterward. However, they are stat()ed to make sure the fastload stuff isn’t stale. This is bad: bug 511761. Making these depend on .autoreg seems reasonable.
/home/taras/builds/minefield/dist/bin/extensions
This probably should depend on .autoreg too
/home/taras/builds/minefield/dist/bin/updates
/home/taras/builds/minefield/dist/bin/updates/0/update.test
/home/taras/builds/minefield/dist/bin/update.test
This too?
/home/taras/builds/minefield/dist/bin/greprefs
/home/taras/builds/minefield/dist/bin/greprefs/all.js
/home/taras/builds/minefield/dist/bin/greprefs/security-prefs.js
/home/taras/builds/minefield/dist/bin/greprefs/xpinstall.js
Reading a directory full of pref files that never change is crazy.
bug 507288 provides 10-100x throughput improvement in reading gre prefs.
/home/taras/builds/minefield/dist/bin/defaults/pref
/home/taras/builds/minefield/dist/bin/defaults/pref/channel-prefs.js
/home/taras/builds/minefield/dist/bin/defaults/pref/firefox-branding.js
/home/taras/builds/minefield/dist/bin/defaults/pref/firefox.js
/home/taras/builds/minefield/dist/bin/defaults/pref/firefox-l10n.js
/home/taras/builds/minefield/dist/bin/defaults/pref/reporter.js
This is similar to gre pref situation, but worse because there are more pref files. This is also harder to refactor. I propose having 2 hardcoded pref names for xulrunner applications: app.js burried in a jar file speed and l10n.js for localization convenience.
/home/taras/builds/minefield/dist/bin/dictionaries
/home/taras/builds/minefield/dist/bin/dictionaries/en-US.aff
/home/taras/builds/minefield/dist/bin/dictionaries/en-US.dic
For some reason only one of my computers is reading the spellcheck dictionary at startup, but it is costing me 70ms. I wonder if these can be moved to a locale jar.
/home/taras/builds/minefield/dist/bin/modules/distribution.js
/home/taras/builds/minefield/dist/bin/modules/DownloadLastDir.jsm
/home/taras/builds/minefield/dist/bin/modules/ISO8601DateUtils.jsm
/home/taras/builds/minefield/dist/bin/modules/NetUtil.jsm
/home/taras/builds/minefield/dist/bin/modules/utils.js
/home/taras/builds/minefield/dist/bin/modules/XPCOMUtils.jsm
These modules should be moved into toolkit.jar: bug 509755
/home/taras/builds/minefield/dist/bin/res/broken-image.png
/home/taras/builds/minefield/dist/bin/res/charsetalias.properties
/home/taras/builds/minefield/dist/bin/res/charsetData.properties
/home/taras/builds/minefield/dist/bin/res/forms.css
/home/taras/builds/minefield/dist/bin/res/hiddenWindow.html
/home/taras/builds/minefield/dist/bin/res/html.css
/home/taras/builds/minefield/dist/bin/res/loading-image.png
/home/taras/builds/minefield/dist/bin/res/quirk.css
/home/taras/builds/minefield/dist/bin/res/ua.css
These need to be in a jar too: bug 508421
/home/taras/builds/minefield/dist/bin/searchplugins
/home/taras/builds/minefield/dist/bin/searchplugins/amazondotcom.xml
/home/taras/builds/minefield/dist/bin/searchplugins/answers.xml
/home/taras/builds/minefield/dist/bin/searchplugins/creativecommons.xml
/home/taras/builds/minefield/dist/bin/searchplugins/eBay.xml
/home/taras/builds/minefield/dist/bin/searchplugins/google.xml
/home/taras/builds/minefield/dist/bin/searchplugins/wikipedia.xml
/home/taras/builds/minefield/dist/bin/searchplugins/yahoo.xml
I think these should be in a jar too.
This concludes the list of Firefox application files read at startup. I haven’t touched the stuff in the profile directories or libraries. I think it is realistic to get a tenfold reduction in physical files read by Firefox 3.6.
10 Responses to “Cleaning Up Startup Disk IO”
“I haven’t touched the stuff in the profile directories or libraries.” I assume that there are similar wins to be found in profile loading, and perhaps smaller wins in library loading?
The default16.png stuff is coming from nsWindow::SetIcon(). Are there multiple windows (hidden window?)? Or maybe we’re just dumb and call it from few different places.
looking at this from the perspective of a complete outsider, is there a reason that so much data has to be stored in files and in separate files at that?
Has anyone considered throwing a lot more into SQLite database files? For example preferences? Would this be any faster?
It seems it took a lot of time to build Places into the codebase and perhaps that is still not perfect (awesome bar perf lags with too much history to churn through?). Would a similar level of resources be required to move a lot of file data to SQLite database files?
Should we maybe be running the JS files through a minimizer and the XML files at least through a comment stripper? At build time, I mean.
> Perhaps, instead of reading these files should be compiling this info into the xulrunner launcher app?
Please don’t. That would make using the plain xulrunner stub impossible.
> For some reason only one of my computers is reading the spellcheck dictionary at startup, but it is costing me 70ms. I wonder if these can be moved to a locale jar
Please don’t. Or at least, still allow for the spellcheck dictionaries to be files in a system-wide directory, shared with OOo (which is done on Linux distros). Please avoid requiring patches to libhunspell, too, so that system-wide libhunspell can still be used.
application.ini and platform.ini are required outside of the app – l10n uses them to correctly process repacks and for other reasons. You’ll need to provide the info somewhere else if you compile it in. You also can’t compile application.ini in afaik as that is the main config part of a xulrunner app.
If you remove the stat check on the components you’ll break the majority of developers – many devs use the fact these are symlinked (on mac & linux) and can change the code without even recompiling. Removing this would break devs significantly. Also I think Thunderbird doesn’t even use .autoreg at the moment as Benjamin said it wasn’t necessary.
You’re also conflicting yourself. You said you’ll check for a .manifest inside the .jar after checking alongside the .jar. That would be a wasted stat in most cases.
There’s a similar developer issue with extensions/ – dev through extensions may start to be more popular. Additionally, if someone installs an extension globally, they’ll have to update the .autoreg file to get it to be picked up – that doesn’t sound like something most people would know to do.
Prefs – don’t forget about the possibility of a user.js or even other .js files (I think MCD has some).
Please also remember to notify comm-central/other apps of these changes or at least keep a live list of what has changed wrt startup. It will make it much easier for us to pick these changes up and we may then be able to do it at the same time as you land or before if appropriate – it saves us firefighting time.
Yeah, baby! Firefox 3.5 startup on OS X 10.5 is *so* slow.
This makes me think that extension packaging should be redesigned in order to make it more efficient. Don’t forget that extensions are a major reason for startup slowness.
My take on this is that extensions shouldn’t be unpacked in the extensions dir; the XPI should be there instead. Perhaps also the practice of packaging all chrome files in chrome.jar should be discontinued as well, given that in this case there would be one zip nested in another one.
Jorge: XPI files are compressed so FF will have to pay the decompression tax while running. Jar files are uncompressed so get the same benefits that Toolkit.jar & browser.jar get (particularly after the recent work Taras has been doing on Jar files).
A lot of extensions package within chrome.jar, but more could (like Stylish etc.)
The extension manager and app update hitting the disk is mainly due to write access check though the hits beyond 1 by app update are mainly do to the way it was original coded. Filed bug 512650 for this