mwu's blog

mozilla-central: Now with 88.7% less PRBool!

mwu — Thu, 29 Sep 2011 10:01:55 +0000

3263 files changed, 30182 insertions(+), 30183 deletions(-)

Bug 675553 has landed. According to a simple grep, we’ve gone from 31656 PRBools to 3562 PRBools and from 1914 PRPackedBool to 14 PRPackedBool. The remaining ones are mostly in C code or code passing data to C.

PR_TRUE (18774 -> 17953) and PR_FALSE (22973 -> 20195) haven’t been affected as much, but now it’s your turn. Feel free to correct any PR_TRUEs or PR_FALSEs you see along the way.

A script is available to help unbitrot your mercurial patch queue. More info is available in the post on dev-planning.

Another script is available to convert any PR_TRUEs or PR_FALSEs on the lines your patches touch. It operates in the same way as the first script.

The Twelve Booleans of Mozilla

mwu — Fri, 29 Jul 2011 06:15:49 +0000

In order to switch Mozilla away from PRBools and to real boolean types, it’s necessary to first fix any code that relies on PRBool being a plain integer. This can be accomplished by ensuring every assignment to a PRBool or return from a function returning PRBool is in fact a boolean.

I wrote a clang plugin to do this called BoolCheck. It’s like Taras’ Prcheck, except based on clang instead of elsa. BoolCheck borrows Prcheck’s tests but everything else was written from scratch.

BoolCheck can check code like this:

typedef void (*prboolFuncPointerType)(PRBool i, double j, PRBool k);

prboolFuncPointerType* f;

void foo() {
  int foo = 12;
  PRBool zoo = 13;
  (*f)(foo, 44, 99);
}

and output errors like this:
% clang++ -c -Xclang -load -Xclang ../BoolCheck.so -Xclang -add-plugin -Xclang boolcheck -Xclang -plugin-arg-boolcheck -Xclang -load-type-list -Xclang -plugin-arg-boolcheck -Xclang test-bools func_pointer_bad.cc

func_pointer_bad.cc:8:16: error: Literal is not a valid boolean
  PRBool zoo = 13;
               ^~
func_pointer_bad.cc:9:8: error: Non-boolean expression
  (*f)(foo, 44, 99);
       ^~~
func_pointer_bad.cc:9:17: error: Literal is not a valid boolean
  (*f)(foo, 44, 99);
                ^~
3 errors generated.

The most complicated part of this is determining whether something is a valid boolean or not. As it turns out, PRBool isn’t the only boolean type in the mozilla code. Every once in a while, another (custom) boolean type gets assigned to a PRBool. In order to make sure that type is always used as a boolean and not a integer, that type is added to the list of types being checked for correctness.

In the end, we need to check about twelve different boolean types to ensure PRBool is only ever assigned booleans.

PRBool
PRPackedBool
JSBool
JSPackedBool
cairo_bool_t
pixman_bool_t
gboolean
PKIX_Boolean
CK_BBOOL
mdb_bool
mork_bool
realGLboolean

The actual number can vary by platform since types like gboolean would only be seen on Linux/gtk, but other platforms usually introduce their own boolean type in place of gboolean. The plugin takes a file containing a list of these types to check for boolean correctness so it can easily be used to help other projects switch to real booleans.

Symlinks enabled on omnijar builds

mwu — Wed, 27 Jul 2011 03:39:27 +0000

Bug 664907 has landed on mozilla-central. This enables symlinks in chrome packaging while omnijar is enabled. So, if you are:

Working on files which are in the chrome directory (listed in a jar manifest)
Working on files which are not preprocessed
Not on Windows
Running builds out of dist/bin (or the OSX equivalent)

then running make after changes won’t be necessary. Depending on the type of files you’re working on, starting Firefox with -purgecaches is probably required, but it’s still faster than running make.

Killing PRBool with clang and a little bit of fire

mwu — Sat, 23 Jul 2011 07:30:41 +0000

Once upon a time, there was a tool called prcheck which checked usage of PRBools and made sure only righteous 1s and 0s were assigned to PRBools. The goal was to correct all the uses of PRBool and hopefully turn PRBool into a Real Boolean.

This didn’t happen, though many abuses of PRBool were corrected along the way.

But, there may be hope now. A new PRBool checking tool based on clang has lead to a number of patches correcting various misuses of PRBool across the Mozilla codebase. The abuses of PRBool are varied and creative, but they all tend to fall under two categories:

Using PRBool when some other type would be more appropriate or vice versa. (bug 671190 and bug 671417)
Returning NS_ERROR_* codes in functions that return PRBool. This is particularly bad as error codes are likely to be treated as true, and true usually means success. (bug 671185)

With the help of this clang plugin, a bit of manual build fixing, and some hours lost in gdb, we can typedef PRBool as a (real) bool. The local build here is capable of starting up, passing make check, and passing all but one xpcshell test.

More updates and code release of this clang plugin to come.

Extensions now installed packed

mwu — Sat, 11 Sep 2010 00:14:11 +0000

Bug 533038 has landed. Extensions are no longer unpacked by default when installed. If an extension with an id such as {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} is installed, it will show up in the profile extensions directory as a file named {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}.xpi instead of a directory named {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}. This was done to reduce the startup impact of having too many files from extensions. It will also make it possible to fix startup cache/fastload invalidation by stat’ing the extensions directory.

Extension developers should note that they can opt back into unpacked installation by adding true to their install.rdf. Extensions with dictionaries, binary components, or window icons are most likely to be incompatible with packed installation.

Due to the entire extension being packed, developers no longer need to pack their chrome files into jars for better startup performance. If a developer chooses to keep packing the chrome files into jars, any jars inside the xpi should not be compressed since compressed jars inside the xpi use more memory to access. The files within the jar within the xpi, however, can continue to be compressed without much impact.

Omnijar. How does it work?

mwu — Sat, 14 Aug 2010 00:28:59 +0000

What is omnijar?

Omnijar is a new packaging format. It’s expected to be enabled by default for Firefox 4.0b5.

The difference that omnijar makes can been seen comparing the file listings of builds before and after omnijar:

Trunk Linux build without omnijar:

firefox/application.ini firefox/blocklist.xml firefox/chrome/browser.jar firefox/chrome/en-US.jar firefox/chrome/icons/default/default16.png firefox/chrome/icons/default/default32.png firefox/chrome/icons/default/default48.png firefox/chrome/localized.manifest firefox/chrome.manifest firefox/chrome/nonlocalized.manifest firefox/chrome/pippki.jar firefox/chrome/toolkit.jar firefox/components/addonManager.js firefox/components/amContentHandler.js firefox/components/amWebInstallListener.js firefox/components/browser.xpt firefox/components/components.manifest firefox/components/contentAreaDropListener.js firefox/components/contentSecurityPolicy.js firefox/components/crypto-SDR.js firefox/components/FeedConverter.js firefox/components/FeedProcessor.js firefox/components/FeedWriter.js firefox/components/fuelApplication.js firefox/components/GPSDGeolocationProvider.js firefox/components/interfaces.manifest firefox/components/jsconsole-clhandler.js firefox/components/libbrowsercomps.so firefox/components/libdbusservice.so firefox/components/libmozgnome.so firefox/components/NetworkGeolocationProvider.js firefox/components/nsBadCertHandler.js firefox/components/nsBlocklistService.js firefox/components/nsBrowserContentHandler.js firefox/components/nsBrowserGlue.js firefox/components/nsContentDispatchChooser.js firefox/components/nsContentPrefService.js firefox/components/nsDefaultCLH.js firefox/components/nsDownloadManagerUI.js firefox/components/nsFilePicker.js firefox/components/nsFormAutoComplete.js firefox/components/nsFormHistory.js firefox/components/nsHandlerService.js firefox/components/nsHelperAppDlg.js firefox/components/nsINIProcessor.js firefox/components/nsLivemarkService.js firefox/components/nsLoginInfo.js firefox/components/nsLoginManager.js firefox/components/nsLoginManagerPrompter.js firefox/components/nsMicrosummaryService.js firefox/components/nsPlacesAutoComplete.js firefox/components/nsPlacesDBFlush.js firefox/components/nsPlacesExpiration.js firefox/components/nsPrivateBrowsingService.js firefox/components/nsPrompter.js firefox/components/nsProxyAutoConfig.js firefox/components/nsSafebrowsingApplication.js firefox/components/nsSearchService.js firefox/components/nsSearchSuggestions.js firefox/components/nsSessionStartup.js firefox/components/nsSessionStore.js firefox/components/nsSetDefaultBrowser.js firefox/components/nsSidebar.js firefox/components/nsTaggingService.js firefox/components/nsTryToClose.js firefox/components/nsUpdateService.js firefox/components/nsUpdateServiceStub.js firefox/components/nsUpdateTimerManager.js firefox/components/nsUrlClassifierLib.js firefox/components/nsUrlClassifierListManager.js firefox/components/nsURLFormatter.js firefox/components/nsWebHandlerApp.js firefox/components/PlacesProtocolHandler.js firefox/components/storage-Legacy.js firefox/components/storage-mozStorage.js firefox/components/txEXSLTRegExFunctions.js firefox/components/WeaveCrypto.js firefox/components/Weave.js firefox/components/WebContentConverter.js firefox/crashreporter firefox/crashreporter.ini firefox/crashreporter-override.ini firefox/defaults/autoconfig/platform.js firefox/defaults/autoconfig/prefcalls.js firefox/defaults/pref/channel-prefs.js firefox/defaults/pref/firefox-branding.js firefox/defaults/pref/firefox.js firefox/defaults/pref/firefox-l10n.js firefox/defaults/pref/reporter.js firefox/defaults/pref/services-sync.js firefox/defaults/profile/bookmarks.html firefox/defaults/profile/chrome/userChrome-example.css firefox/defaults/profile/chrome/userContent-example.css firefox/defaults/profile/localstore.rdf firefox/defaults/profile/mimeTypes.rdf firefox/defaults/profile/prefs.js firefox/dependentlibs.list firefox/dictionaries/en-US.aff firefox/dictionaries/en-US.dic firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/icon.png firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/preview.png firefox/firefox firefox/firefox-bin firefox/greprefs.js firefox/icons/document.png firefox/icons/mozicon128.png firefox/icons/updater.png firefox/libfreebl3.chk firefox/libfreebl3.so firefox/libmozalloc.so firefox/libmozjs.so firefox/libmozsqlite3.so firefox/libnspr4.so firefox/libnss3.so firefox/libnssckbi.so firefox/libnssdbm3.chk firefox/libnssdbm3.so firefox/libnssutil3.so firefox/libplc4.so firefox/libplds4.so firefox/libsmime3.so firefox/libsoftokn3.chk firefox/libsoftokn3.so firefox/libssl3.so firefox/libxpcom.so firefox/libxul.so firefox/modules/AddonLogging.jsm firefox/modules/AddonManager.jsm firefox/modules/AddonRepository.jsm firefox/modules/AddonUpdateChecker.jsm firefox/modules/CertUtils.jsm firefox/modules/CrashSubmit.jsm firefox/modules/CSPUtils.jsm firefox/modules/ctypes.jsm firefox/modules/debug.js firefox/modules/distribution.js firefox/modules/DownloadLastDir.jsm firefox/modules/DownloadPaths.jsm firefox/modules/DownloadUtils.jsm firefox/modules/FileUtils.jsm firefox/modules/Geometry.jsm firefox/modules/HUDService.jsm firefox/modules/InlineSpellChecker.jsm firefox/modules/ISO8601DateUtils.jsm firefox/modules/LightweightThemeConsumer.jsm firefox/modules/LightweightThemeManager.jsm firefox/modules/Microformats.js firefox/modules/NetUtil.jsm firefox/modules/NetworkPrioritizer.jsm firefox/modules/openLocationLastURL.jsm firefox/modules/PerfMeasurement.jsm firefox/modules/PlacesDBUtils.jsm firefox/modules/PlacesUIUtils.jsm firefox/modules/PlacesUtils.jsm firefox/modules/PluginProvider.jsm firefox/modules/PluralForm.jsm firefox/modules/PopupNotifications.jsm firefox/modules/Services.jsm firefox/modules/services-sync/auth.js firefox/modules/services-sync/base_records/collection.js firefox/modules/services-sync/base_records/crypto.js firefox/modules/services-sync/base_records/keys.js firefox/modules/services-sync/base_records/wbo.js firefox/modules/services-sync/constants.js firefox/modules/services-sync/engines/bookmarks.js firefox/modules/services-sync/engines/clients.js firefox/modules/services-sync/engines/forms.js firefox/modules/services-sync/engines/history.js firefox/modules/services-sync/engines.js firefox/modules/services-sync/engines/passwords.js firefox/modules/services-sync/engines/prefs.js firefox/modules/services-sync/engines/tabs.js firefox/modules/services-sync/ext/Observers.js firefox/modules/services-sync/ext/Preferences.js firefox/modules/services-sync/ext/StringBundle.js firefox/modules/services-sync/ext/Sync.js firefox/modules/services-sync/identity.js firefox/modules/services-sync/log4moz.js firefox/modules/services-sync/notifications.js firefox/modules/services-sync/resource.js firefox/modules/services-sync/service.js firefox/modules/services-sync/status.js firefox/modules/services-sync/stores.js firefox/modules/services-sync/trackers.js firefox/modules/services-sync/type_records/bookmark.js firefox/modules/services-sync/type_records/clients.js firefox/modules/services-sync/type_records/forms.js firefox/modules/services-sync/type_records/history.js firefox/modules/services-sync/type_records/passwords.js firefox/modules/services-sync/type_records/prefs.js firefox/modules/services-sync/type_records/tabs.js firefox/modules/services-sync/util.js firefox/modules/SpatialNavigation.js firefox/modules/stylePanel.jsm firefox/modules/tabview/AllTabs.jsm firefox/modules/tabview/groups.jsm firefox/modules/tabview/utils.jsm firefox/modules/utils.js firefox/modules/WindowDraggingUtils.jsm firefox/modules/XPCOMUtils.jsm firefox/modules/XPIProvider.jsm firefox/mozilla-xremote-client firefox/platform.ini firefox/plugin-container firefox/README.txt firefox/removed-files firefox/res/contenteditable.css firefox/res/designmode.css firefox/res/dtd/mathml.dtd firefox/res/dtd/xhtml11.dtd firefox/res/EditorOverride.css firefox/res/entityTables/html40Latin1.properties firefox/res/entityTables/html40Special.properties firefox/res/entityTables/html40Symbols.properties firefox/res/entityTables/htmlEntityVersions.properties firefox/res/entityTables/mathml20.properties firefox/res/entityTables/transliterate.properties firefox/res/fonts/mathfont.properties firefox/res/fonts/mathfontStandardSymbolsL.properties firefox/res/fonts/mathfontSTIXNonUnicode.properties firefox/res/fonts/mathfontSTIXSize1.properties firefox/res/fonts/mathfontSTIXSizeOneSym.properties firefox/res/fonts/mathfontUnicode.properties firefox/res/grabber.gif firefox/res/html/folder.png firefox/res/langGroups.properties firefox/res/language.properties firefox/res/svg.css firefox/res/table-add-column-after-active.gif firefox/res/table-add-column-after.gif firefox/res/table-add-column-after-hover.gif firefox/res/table-add-column-before-active.gif firefox/res/table-add-column-before.gif firefox/res/table-add-column-before-hover.gif firefox/res/table-add-row-after-active.gif firefox/res/table-add-row-after.gif firefox/res/table-add-row-after-hover.gif firefox/res/table-add-row-before-active.gif firefox/res/table-add-row-before.gif firefox/res/table-add-row-before-hover.gif firefox/res/table-remove-column-active.gif firefox/res/table-remove-column.gif firefox/res/table-remove-column-hover.gif firefox/res/table-remove-row-active.gif firefox/res/table-remove-row.gif firefox/res/table-remove-row-hover.gif firefox/run-mozilla.sh firefox/searchplugins/amazondotcom.xml firefox/searchplugins/answers.xml firefox/searchplugins/creativecommons.xml firefox/searchplugins/eBay.xml firefox/searchplugins/google.xml firefox/searchplugins/wikipedia.xml firefox/searchplugins/yahoo.xml firefox/Throbber-small.gif firefox/update.locale firefox/updater firefox/updater.ini

That’s 259 files. The total number of files differ by platform, but not by much.

Trunk Linux build with omnijar:

firefox/application.ini firefox/blocklist.xml firefox/chrome/icons/default/default16.png firefox/chrome/icons/default/default32.png firefox/chrome/icons/default/default48.png firefox/chrome.manifest firefox/components/binary.manifest firefox/components/libbrowsercomps.so firefox/components/libdbusservice.so firefox/components/libmozgnome.so firefox/crashreporter firefox/crashreporter.ini firefox/crashreporter-override.ini firefox/dependentlibs.list firefox/dictionaries/en-US.aff firefox/dictionaries/en-US.dic firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/icon.png firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/install.rdf firefox/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/preview.png firefox/firefox firefox/firefox-bin firefox/icons/document.png firefox/icons/mozicon128.png firefox/icons/updater.png firefox/libfreebl3.chk firefox/libfreebl3.so firefox/libmozalloc.so firefox/libmozjs.so firefox/libmozsqlite3.so firefox/libnspr4.so firefox/libnss3.so firefox/libnssckbi.so firefox/libnssdbm3.chk firefox/libnssdbm3.so firefox/libnssutil3.so firefox/libplc4.so firefox/libplds4.so firefox/libsmime3.so firefox/libsoftokn3.chk firefox/libsoftokn3.so firefox/libssl3.so firefox/libxpcom.so firefox/libxul.so firefox/mozilla-xremote-client firefox/omni.jar firefox/platform.ini firefox/plugin-container firefox/README.txt firefox/removed-files firefox/run-mozilla.sh firefox/searchplugins/amazondotcom.xml firefox/searchplugins/answers.xml firefox/searchplugins/creativecommons.xml firefox/searchplugins/eBay.xml firefox/searchplugins/google.xml firefox/searchplugins/wikipedia.xml firefox/searchplugins/yahoo.xml firefox/Throbber-small.gif firefox/update.locale firefox/updater firefox/updater.ini

There are 61 files with omnijar. The omnijar build has 76.4% less files. Or, as I like to think of it, the non-omnijar build has 324.6% more files.

How does it work?

A comparison of different packaging formats in Mozilla

Flat packaging. The original. Just files in directories.
Jar packaging. A more sophisticated way. Files in the chrome directory are packed into jars.
Omnijar packaging. The third generation packaging format. Every file that can be placed into a jar is packed into omni.jar.

All the files that are now gone have been packed into a file named omni.jar.

Why?

It’s faster.

Storing related files in a jar is more efficient than scattering files across the filesystem. Flat packaging causes a sort of high level fragmentation where related files might be stored far apart on the disk and result in wasted time seeking. Keeping everything in a file minimizes fragmentation. Startup time is generally improved. Omnijar also paves the way for other startup optimizations.

It’s useful for developers

Omnijar improves performance for users, but it also provides a handy feature for developers – omnijar packaging can be easily converted into flat packaging. To do so, unzip the omni.jar file and delete it. Any file can then be directly edited.

Developers building Firefox with omnijar enabled (–enable-chrome-format=omni) should know that the build in the dist/bin directory is flat packaged. Files are only packed into the omnijar after packaging with |make package|.