Category Archives: Firefox

San Francisco Oxidation meeting notes

At last week’s Mozilla All Hands meeting in San Francisco we had an Oxidation meeting about the use of Rust in Firefox. It was low-key, being mostly about status and progress. The notes are here for those who are interested.

A New Preferences Parser for Firefox

Firefox’s preferences system uses data files to store information about default preferences within Firefox, and user preferences in a user’s profile (such as prefs.js, which records changes to preference values, and user.js, which allows users to override default preference values).

A new parser

These data files use a custom format, and therefore Firefox has a custom parser for them. I recently rewrote the parser. The new parser has the following benefits over the old parser.

  • It is faster (raw parsing speed is close to 2x faster).
  • It is safer (because it’s written in Rust rather than C++).
  • It is more correct and better tested (the old one got various obscure edge cases wrong).
  • It is more readable, and easier to modify.
  • It issues no warnings, only errors.
  • It is slightly stricter (e.g. doesn’t allow any malformed input, and it catches integer overflow).
  • It has error recovery and better error messages (including correct line numbers).


Modifiability was the prime motivation for the change. I wanted to make some adjustments to the preferences file grammar, but this would have been very difficult in the old parser, because it was written in an awkward style.

It was essentially a single loop containing a giant switch statement on a state variable. This switch was executed for every single char in a file. The states held by the state variable had names like PREF_PARSE_QUOTED_STRING, PREF_PARSE_UNTIL_OPEN_PAREN, PREF_PARSE_COMMENT_BLOCK_MAYBE_END. It also had a second state variable, because in some places a single one wasn’t enough; the parser had to return to the previous state after exiting the current state. Furthermore, lexing and parsing were not separate, so code to handle comments and whitespace was spread around in various places.

The new parser is a recursive descent parser — even though the grammar doesn’t actually have any recursion — in which the structure of the code reflects the structure of the grammar. Lexing is distinct from parsing. As a result, the new parser is much easier to read and modify. In particular, after landing it I added error recovery without too much effort; that would have been almost impossible in the old parser.

Note that the idea of error recovery for preferences parsing was first proposed in bug 107264, filed in 2001! After landing it, I tweeted the following.

Amazingly enough, the original reporter is on Twitter and responded!


The new parser is slightly stricter and rejects some malformed input that the old parser accepted.

Junk chars

Disconcertingly, the old parser allowed arbitrary junk between preferences (including at the start and end of the prefs file) so long as that junk didn’t include any of the following chars: ‘/’, ‘#’, ‘u’, ‘s’, ‘p’. This means that lines like these:

!foo@bar&pref("prefname", true);
ticky_pref("prefname", true);    // missing 's' at start
User_pref("prefname", true);     // should be 'u' at start

would all be treated the same as this:

pref("prefname", true);

The new parser disallows such junk because it isn’t necessary and seems like an unintentional botch by the old parser. In practice, this caught a couple of prefs that accidentally had an extra ‘;’ at the end.

SUB char

The old parser allowed the SUB (0x1a) character between tokens and treated it like ‘\n’.

The new parser does not allow this character. SUB was used to indicate end-of-file (not end-of-line) in some old operating systems such as MS-DOS, but this doesn’t seem necessary today.

Invalid escapes

The old parser tolerated (with a warning) invalid escape sequences within  string literals — such as “\q” (not a valid escape) and “\x1” and “\u12″(both of which have insufficient hex digits) — accepting them literally.

The new parser does not tolerate invalid escape sequences because it doesn’t seem necessary and would complicate things.

NUL char

The old parser tolerated the NUL character (0x00) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs will almost certainly consider those chars as end-of-string markers.

The new parser treats the NUL character as end-of-file, to avoid this danger. (The escape sequences “\x00” and “\u0000” are also disallowed.)

Integer overflow

The old parser allowed integer literals to overflow, silently wrapping them.

The new parser treats integer overflow as a parse error. This seems better,
and it caught overflows of several existing prefs.


Error recovery minimizes the risk of data loss caused by the increased strictness because malformed pref lines in prefs.js will be removed but well-formed pref lines afterwards are preserved.

Nonetheless, please keep an eye out for any other problems that might arise from this change.


I mentioned before that I wanted to make some adjustments to the preferences file grammar. Specifically, I changed the grammar used by default preference files (but not user preference files) to support annotating each preference with one or more boolean attributes. The attributes supported so far are ‘sticky’ and ‘locked’. For example:

pref("sticky.pref", true, sticky);
pref("locked.pref", 123, locked);
pref("sticky-and-locked-pref", "blah", sticky, locked);

Note that the addition of the ‘locked’ attribute fixed a 10 year old bug.

When will this ship?

All of these changes are on track to ship in Firefox 60, which is due to release on May 9th.

Nicer commands for content processes

The current Firefox release creates content processes with very long and ugly commands. Here’s an example, as reported by ‘ps’ on my Linux64 machine (with wrapping to make it show up entirely on your screen):

/home/njn/moz/mi8/o64/dist/bin/firefox -contentproc -childID 2 -isForBrowser -in
tPrefs 6:50|7:-1|20:0|35:1000|43:20|44:5|45:10|52:0|58:128|59:10000|64:0|66:400|
300|310:30| -boolPrefs 1:0|2:0|4:1|5:1|25:1|28:1|29:1|30:1|32:1|33:1|34:1|37:1|3
|295:1|304:1|305:1|306:0|307:0|308:0| -stringPrefs 3:7;default|215:3;1.0|226:332
;  ¼½¾ǃː??։֊׃״؉؊٪۔܁܂܃܄ᅟᅠ᜵           ???‐ ’․‧??????? ‹›⁁⁄⁒ ⅓⅔⅕⅖⅗⅘⅙⅚?⅜⅝⅞⅟∕∶⎮╱⧶⧸⫻⫽>
⿰⿱⿲⿳⿴⿵⿶⿷⿸⿹⿺⿻ 。〔〕〳゠ㅤ㈝㈞㎮㎯㏆㏟꞉︔︕︿﹝﹞?./。ᅠ???�|227:4;
high|280:36;97c47c17-80b5-497c-b5ee-1c5db2d06af6| -schedulerPrefs 0001,2 -appdir
 /home/njn/moz/mi8/o64/dist/bin/browser 4500 true tab

That’s 1733 characters! Most of it is for three flags, called -intPrefs, -boolPrefs, and -stringPrefs, which are used to pass values of prefs that are required very early in the content process’s existence, before the first normal IPC message is received. These pref values are encoded compactly, but there are enough of them that they make the command quite long.

Furthermore, there is one pref (network.IDN.blacklist_chars) that contains a bunch of unusual characters. That explains the gobbledygook on the 4th and 3rd last lines. Unsurprisingly, this gobbledygook was reported as a bug… which has been duplicated five times.

The good news is that I recently fixed this, by changing things so that only pref values that have changed since startup get passed in this way. (Content processes are able to see the startup values, and so don’t need to be told them.) On my machine and profile, this reduces the number of pref values passed via the command from ~222 to ~3. It also reduces the number sent later via IPC from ~3165 to ~180.

The command now looks like this:

/home/njn/local/install/firefox/firefox -contentproc -childID 4 -isForBrowser -b
oolPrefs 37:1|235:1|257:1|294:1| -stringPrefs 280:36;8bdf7a6e-d39a-494c-b55a-829
2c4fc6254| -schedulerPrefs 0001,2 -greomni /home/njn/local/install/firefox/omni.
ja -appomni /home/njn/local/install/firefox/browser/omni.ja -appdir /home/njn/lo
cal/install/firefox/browser 3237 true tab

That’s 361 characters, all of them ASCII. Much better! And I have a plan to reduce things even further.

This change is in Firefox 60, which is on track to be released around May 9th.

Faster Firefox startup & shutdown with add-ons present

Thanks to some recent improvements by Kris Maglione, add-ons built with Firefox’s add-ons SDK are now much faster to start up and shut down. For users with multiple add-ons installed, this can make a large difference. For example, one user reported the following.

I’ve noticed a dramatic improvement in startup and shutdown time since the fixes landed on aurora.

Configuration: Two windows with 15 tabs each and 22 addons.

  • Startup: 25 seconds -> 14 seconds (44% reduction)
  • Shutdown: 30 seconds -> 5 seconds (83% reduction)

Shutdown used to take so long it was triggering a minidump.

Your mileage may vary, but these improvements have the potential to benefit many users. They are present in Firefox 50 (currently in Beta, and due for release on November 15) and all later versions.

How to get localized Firefox Nightly builds

One of the easiest and best ways that someone can help Mozilla and Firefox is to run Firefox Nightly. I’ve been doing it on my Windows, Mac and Linux machines for the past couple of months. It requires daily restarts, but otherwise it has been a smooth experience for me.

Unfortunately the number of Nightly users has been steadily dropping for some time, which hurts our ability to catch crashes and other regressions early. Pascal Chevrel and Marcia Knous are leading efforts underway to reverse this trend.

One problem with Nightly builds has been their visibility. In particular, finding localized (non-English) builds was difficult. That situation has just improved: thanks to Kohei Yoshino there is now a single page containing Nightly builds for all platforms and locales. As far as I know there are no other pages that currently link to that page, but perhaps that will happen as part of the planned work to give Nightly builds a place on

If you have friends and family who would like to help Mozilla and are willing to use pre-release versions of Firefox, please suggest Firefox Nightly to them.

How to switch to a 64-bit Firefox on Windows

I recently wrote about 64-bit Firefox builds on Windows, explaining why you might want to switch — it can reduce the likelihood of out-of-memory crashes — and also some caveats.

However, I didn’t explain how to switch, so I will do that now.

First, if you want to make sure that you aren’t already running a 64-bit Firefox, type “about:support” in the address bar and then look at the User Agent field in the Application Basics table near the top of the page.

  • If it contains the string “Win64”, you are already running a 64-bit Firefox.
  • If it contains the string “WOW64“, you are running a 32-bit Firefox on a 64-bit Windows installation, which means you can switch to a 64-bit build.
  • Otherwise, you are running a 32-bit Firefox on a 32-bit Windows installation, and cannot switch to a 64-bit Firefox.

Here are links to pages contain 64-bit builds for all the different release channels.

  • Release
  • Beta
  • Developer Edition
  • Nightly:  This is a user-friendly page, but it only has the en-US locale.
  • Nightly: This is a more intimidating page, but it has all locales. Look for a file with a name of the form firefox-<VERSION>.<LOCALE>.win64.installer.exe, e.g. for Nightly 50 in German.

By default, 32-bit Firefox and 64-bit Firefox are installed to different locations:

  • C:\Program Files (x86)\Mozilla Firefox\
  • C:\Program Files\Mozilla Firefox\

If you are using a 32-bit Firefox and then you download and install a 64-bit Firefox, by default you will end up with two versions of  Firefox installed. (But note that if you let the 64-bit Firefox installer add shortcuts to the desktop and/or taskbar, these shortcuts will replace any existing shortcuts to 32-bit Firefox.)

Both the 32-bit Firefox and the 64-bit Firefox will use the same profile, which means all your history, bookmarks, extensions, etc., will be available in either version. You’ll be able to run both versions, though not at the same time with the same profile. If you decide you don’t need both versions you can simply remove the unneeded version through the Windows system settings, as normal; your profile will not be touched when you do this.

Finally, there is a plan to gradually roll out 64-bit Firefox to Windows users in increasing numbers.

Firefox 64-bit for Windows can take advantage of more memory

By default, on Windows, Firefox is a 32-bit application. This means that it is limited to using at most 4 GiB of memory, even on machines that have more than 4 GiB of physical memory (RAM). In fact, depending on the OS configuration, the limit may be as low as 2 GiB.

Now, 2–4 GiB might sound like a lot of memory, but it’s not that unusual for power users to use that much. This includes:

  • users with many (dozens or even hundreds) of tabs open;
  • users with many (dozens) of extensions;
  • users of memory-hungry web sites and web apps; and
  • users who do all of the above!

Furthermore, in practice it’s not possible to totally fill up this available space because fragmentation inevitably occurs. For example, Firefox might need to make a 10 MiB allocation and there might be more than 10 MiB of unused memory, but if that available memory is divided into many pieces all of which are smaller than 10 MiB, then the allocation will fail.

When an allocation does fail, Firefox can sometimes handle it gracefully. But often this isn’t possible, in which case Firefox will abort. Although this is a controlled abort, the effect for the user is basically identical to an uncontrolled crash, and they’ll have to restart Firefox. A significant fraction of Firefox crashes/aborts are due to this problem, known as address space exhaustion.

Fortunately, there is a solution to this problem available to anyone using a 64-bit version of Windows: use a 64-bit version of Firefox. Now, 64-bit applications typically use more memory than 32-bit applications. This is because pointers, a common data type, are twice as big; a rough estimate for 64-bit Firefox is that it might use 25% more memory. However, 64-bit applications also have a much larger address space, which means they can access vast amounts of physical memory, and address space exhaustion is all but impossible. (In this way, switching from a 32-bit version of an application to a 64-bit version is the closest you can get to downloading more RAM!)

Therefore, if you have a machine with 4 GiB or less of RAM, switching to 64-bit Firefox probably won’t help. But if you have 8 GiB or more, switching to 64-bit Firefox probably will help the memory usage situation.

Official 64-bit versions of Firefox have been available since December 2015. If the above discussion has interested you, please try them out. But note the following caveats.

  • Flash and Silverlight are the only supported 64-bit plugins.
  • There are some Flash content regressions due to our NPAPI sandbox (for content that uses advanced features like GPU acceleration or microphone APIs).

On the flip side, as well as avoiding address space exhaustion problems, a security feature known as ASLR works much better in 64-bit applications than in 32-bit applications, so 64-bit Firefox will be slightly more secure.

Work is being ongoing to fix or minimize the mentioned caveats, and it is expected that 64-bit Firefox will be rolled out in increasing numbers in the not-too-distant future.

UPDATE: Chris Peterson gave me the following measurements about daily active users on Windows.

  • 66.0% are running 32-bit Firefox on 64-bit Windows. These users could switch to a 64-bit Firefox.
  • 32.3% are running 32-bit Firefox on 32-bit Windows. These users cannot switch to a 64-bit Firefox.
  • 1.7% are running 64-bit Firefox already.

UPDATE 2: Also from Chris Peterson, here are links to 64-bit builds for all the channels:

More compacting GC

Jon Coppeard recently extended SpiderMonkey’s compacting GC abilities. Previously, the GC could only compact GC arena containing JavaScript objects. Now it can also compact arenas containing shapes (a data structure used within SpiderMonkey which isn’t visible to user code) and strings, which are two of the largest users of memory in the GC heap after objects.

These improvements should result in savings of multiple MiBs in most workloads, and they are on track to ship in Firefox 48, which will be released in early August. Great work, Jon!

I rewrote Firefox’s BMP decoder

Recently I’ve been deliberately working on some areas of Firefox I’m unfamiliar with, particular relating to graphics. This led me to rewriting Firefox’s BMP decoder and learn a number of interesting things along the way.

Image decoding

Image decoding is basically the process of taking an image encoded in a file and extracting its pixels. In principle it’s simple. You start by reading some information about the image, such as its size and colour depth, which typically comes in some kind of fixed-size header. Then you read the pixel data, which is variable-sized.

This isn’t hard if you have all the data available at the start. But in the context of a browser it makes sense to decode incrementally as data comes in over the network. In that situation you have to be careful and constantly check if you have enough data yet to safely read the next chunk of data. This checking is error-prone and tends to spread itself all over the image decoder.

For this reason, Seth Fowler recently wrote a new class called StreamingLexer that encapsulates this checking and exposes a nice state-based interface to image decoders. When a decoder changes state (e.g. it finishes reading the header) it tells StreamingLexer how many bytes it needs to safely enter the next state (e.g. to read the first row of pixels) and StreamingLexer won’t return control to the decoder until that many bytes are available.

Another consideration when decoding images is that you can’t trust them. E.g. an image might claim to be 100 x 100 pixels but actually contain less data than that. If you’re not careful you could easily read memory you shouldn’t, which could cause crashes or security problems. StreamingLexer helps with this, too.

StreamingLexer makes image decoders simpler and safer, and converting the BMP decoder to use it was my starting point.

The BMP format

The BMP format comes from Windows. On the web it’s mostly used on the web for favicons though it can be used for normal images.

There’s no specification for BMP. There are eight in-use versions of the format that I know of, with later versions mostly(!) extending earlier versions. If you’re interested, you can read the brief description of all these versions that I wrote in a big comment at the top of nsBMPDecoder.cpp.

Because the format is so gnarly I started getting nervous that my rewrite might  introduce bugs in some of the darker corners, especially once Seth told me that our BMP test coverage wasn’t that good.

So I searched around and found Jason Summers’ wonderful BMP Suite, which exercises pretty much every corner of the BMP format. Version 2.3 of the BMP Suite contains 57 images, 23 of which are “good” (obviously valid), 14 of which are “bad” (obviously invalid) and 20 of which are “questionable” (not obviously valid or invalid). The presence of this last category demonstrates just how ill-specified BMP is as a format, and some of the “questionable” tests have two or three reference images, any of which could be considered a correct rendering. (Furthermore, it’s possible to render a number of the “bad” images in a reasonable way.)

This test suite was enormously helpful. As well as giving me greater confidence in my changes, it immediately showed that we had several defects in the existing BMP decoder, particular relating to the scaling of 16-bit colors and an almost complete lack of transparency handling. In comparison, Chrome rendered pretty much all the images in BMP suite reasonably, and Safari and Edge got a few wrong but still did better than Firefox.

Fixing the problems

So I fixed these problems as part of my rewrite. The following images show a number of test images that Firefox used to render incorrectly; in each case a correct rendering is on the left, and our old incorrect rendering is on the right.

bad-bmp-2 bad-bmp-3 bad-bmp-4 bad-bmp-5

It’s clear that the old defects were mostly related to colour-handling, though the first pair of images shows a problem relating to the starting point of the pixel data.

(These images are actually from an old version of Firefox with version 2.4 of BMP Suite, which I just discovered was released only a few days ago. I just filed a bug to update the copy we use in automated testing. Happily, it looks like the new code does reasonable things with all the images added in v2.4.)

These improvements will ship in Firefox 44, which is scheduled to be released in late January, 2016. And with that done I now need to start thinking about rewriting the GIF decoder

moz-icon: a curious corner of Firefox

Here’s an odd little feature in Firefox. Enter the following text into the address bar.


On my Linux box, it shows the following icon image.


On my Windows laptop, it shows a different icon image.


On my Mac Laptop, that URL doesn’t work but if I change the “128” to “64” it shows this icon image.


In each case we get the operating system’s icon for a PDF file.

Change the “size” (up to a maximum of 255) value and you’ll get a different size. Except on Mac the limit seems to be lower, probably due to the retina screen in some way.

Change the file extension and you’ll get a different icon. You can try all sorts, like “.xml”, “.cpp”, “.js”, “.py”, “.doc”, etc.

How interesting. So what’s this for? As far as I understand, it’s used to make local directory listing pages look nice. E.g. on my Linux box if I type “file:///home/njn/” into the address bar I get a listing of my home directory, part of which looks like the following.


That listing uses this mechanism to show the appropriate system icon for each file type.

Furthermore, this feature is usable from regular web pages! Just put a “moz-icon” URL into an <image> tag and you’ll get OS-specific icons in your webpage.

That’ll only work on Firefox, though. Chrome actually has a similar mechanism, though it’s not usable from regular web pages. (For more detail — much more — read here.) As far as I know Safari, IE and Edge do not have such a mechanism; I’m not sure if they support listing of local directories in this fashion.