Categories
Firefox Prefs

A New Preferences Parser for Firefox

Firefox’s preferences system uses data files to store information about default preferences within Firefox, and user preferences in a user’s profile (such as prefs.js, which records changes to preference values, and user.js, which allows users to override default preference values).

A new parser

These data files use a custom format, and therefore Firefox has a custom parser for them. I recently rewrote the parser. The new parser has the following benefits over the old parser.

  • It is faster (raw parsing speed is close to 2x faster).
  • It is safer (because it’s written in Rust rather than C++).
  • It is more correct and better tested (the old one got various obscure edge cases wrong).
  • It is more readable, and easier to modify.
  • It issues no warnings, only errors.
  • It is slightly stricter (e.g. doesn’t allow any malformed input, and it catches integer overflow).
  • It has error recovery and better error messages (including correct line numbers).

Modifiability

Modifiability was the prime motivation for the change. I wanted to make some adjustments to the preferences file grammar, but this would have been very difficult in the old parser, because it was written in an awkward style.

It was essentially a single loop containing a giant switch statement on a state variable. This switch was executed for every single char in a file. The states held by the state variable had names like PREF_PARSE_QUOTED_STRING, PREF_PARSE_UNTIL_OPEN_PAREN, PREF_PARSE_COMMENT_BLOCK_MAYBE_END. It also had a second state variable, because in some places a single one wasn’t enough; the parser had to return to the previous state after exiting the current state. Furthermore, lexing and parsing were not separate, so code to handle comments and whitespace was spread around in various places.

The new parser is a recursive descent parser — even though the grammar doesn’t actually have any recursion — in which the structure of the code reflects the structure of the grammar. Lexing is distinct from parsing. As a result, the new parser is much easier to read and modify. In particular, after landing it I added error recovery without too much effort; that would have been almost impossible in the old parser.

Note that the idea of error recovery for preferences parsing was first proposed in bug 107264, filed in 2001! After landing it, I tweeted the following.

Amazingly enough, the original reporter is on Twitter and responded!

Strictness

The new parser is slightly stricter and rejects some malformed input that the old parser accepted.

Junk chars

Disconcertingly, the old parser allowed arbitrary junk between preferences (including at the start and end of the prefs file) so long as that junk didn’t include any of the following chars: ‘/’, ‘#’, ‘u’, ‘s’, ‘p’. This means that lines like these:

!foo@bar&pref("prefname", true);
ticky_pref("prefname", true);    // missing 's' at start
User_pref("prefname", true);     // should be 'u' at start

would all be treated the same as this:

pref("prefname", true);

The new parser disallows such junk because it isn’t necessary and seems like an unintentional botch by the old parser. In practice, this caught a couple of prefs that accidentally had an extra ‘;’ at the end.

SUB char

The old parser allowed the SUB (0x1a) character between tokens and treated it like ‘\n’.

The new parser does not allow this character. SUB was used to indicate end-of-file (not end-of-line) in some old operating systems such as MS-DOS, but this doesn’t seem necessary today.

Invalid escapes

The old parser tolerated (with a warning) invalid escape sequences within  string literals — such as “\q” (not a valid escape) and “\x1” and “\u12″(both of which have insufficient hex digits) — accepting them literally.

The new parser does not tolerate invalid escape sequences because it doesn’t seem necessary and would complicate things.

NUL char

The old parser tolerated the NUL character (0x00) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs will almost certainly consider those chars as end-of-string markers.

The new parser treats the NUL character as end-of-file, to avoid this danger. (The escape sequences “\x00” and “\u0000” are also disallowed.)

Integer overflow

The old parser allowed integer literals to overflow, silently wrapping them.

The new parser treats integer overflow as a parse error. This seems better,
and it caught overflows of several existing prefs.

Consequences

Error recovery minimizes the risk of data loss caused by the increased strictness because malformed pref lines in prefs.js will be removed but well-formed pref lines afterwards are preserved.

Nonetheless, please keep an eye out for any other problems that might arise from this change.

Attributes

I mentioned before that I wanted to make some adjustments to the preferences file grammar. Specifically, I changed the grammar used by default preference files (but not user preference files) to support annotating each preference with one or more boolean attributes. The attributes supported so far are ‘sticky’ and ‘locked’. For example:

pref("sticky.pref", true, sticky);
pref("locked.pref", 123, locked);
pref("sticky-and-locked-pref", "blah", sticky, locked);

Note that the addition of the ‘locked’ attribute fixed a 10 year old bug.

When will this ship?

All of these changes are on track to ship in Firefox 60, which is due to release on May 9th.

Categories
Firefox Prefs

Nicer commands for content processes

The current Firefox release creates content processes with very long and ugly commands. Here’s an example, as reported by ‘ps’ on my Linux64 machine (with wrapping to make it show up entirely on your screen):

/home/njn/moz/mi8/o64/dist/bin/firefox -contentproc -childID 2 -isForBrowser -in
tPrefs 6:50|7:-1|20:0|35:1000|43:20|44:5|45:10|52:0|58:128|59:10000|64:0|66:400|
67:1|68:0|69:0|70:100|75:0|76:120|77:120|162:2|163:1|167:60|168:30|169:512000|17
8:5000|180:6|194:8192|195:524288|196:5|209:10000|230:24|231:32768|233:0|234:0|24
3:5|247:1048576|249:100|250:5000|252:600|253:4|254:1|262:20|279:4|291:60000|309:
300|310:30| -boolPrefs 1:0|2:0|4:1|5:1|25:1|28:1|29:1|30:1|32:1|33:1|34:1|37:1|3
8:1|39:1|42:1|46:1|47:0|48:0|49:1|50:1|51:1|53:0|56:1|57:1|60:1|61:0|62:0|63:0|6
5:0|71:1|72:1|73:1|74:1|78:1|79:1|80:0|81:0|82:1|83:1|84:1|85:1|86:1|89:0|90:0|9
3:1|94:1|98:1|99:1|100:0|101:1|102:1|103:1|104:0|105:0|107:0|108:0|109:1|110:1|1
11:1|114:1|115:1|116:1|117:1|118:1|119:0|120:0|121:1|122:0|123:0|124:1|125:1|126
:0|127:1|128:1|129:0|131:1|132:0|133:0|134:1|135:1|136:0|137:0|138:0|139:1|140:1
|141:1|142:1|143:0|144:1|145:1|146:1|147:1|148:1|149:1|150:0|151:1|152:1|153:0|1
54:1|155:0|156:0|157:0|158:1|159:1|160:1|161:1|164:1|165:0|172:1|175:0|176:0|177
:1|181:0|183:1|184:0|185:1|187:1|189:0|190:0|193:0|197:1|198:0|199:1|200:1|201:0
|204:1|208:1|210:1|211:0|213:1|216:0|228:0|229:0|232:1|235:0|237:1|238:1|240:1|2
41:0|248:1|251:1|256:0|257:0|258:0|259:1|260:1|261:0|266:1|269:1|270:1|271:1|272
:1|273:1|274:0|275:1|281:1|284:0|285:1|286:1|287:0|288:1|289:1|290:1|292:0|293:0
|295:1|304:1|305:1|306:0|307:0|308:0| -stringPrefs 3:7;default|215:3;1.0|226:332
;  ¼½¾ǃː??։֊׃״؉؊٪۔܁܂܃܄ᅟᅠ᜵           ???‐ ’․‧??????? ‹›⁁⁄⁒ ⅓⅔⅕⅖⅗⅘⅙⅚?⅜⅝⅞⅟∕∶⎮╱⧶⧸⫻⫽>
⿰⿱⿲⿳⿴⿵⿶⿷⿸⿹⿺⿻ 。〔〕〳゠ㅤ㈝㈞㎮㎯㏆㏟꞉︔︕︿﹝﹞?./。ᅠ???�|227:4;
high|280:36;97c47c17-80b5-497c-b5ee-1c5db2d06af6| -schedulerPrefs 0001,2 -appdir
 /home/njn/moz/mi8/o64/dist/bin/browser 4500 true tab

That’s 1733 characters! Most of it is for three flags, called -intPrefs, -boolPrefs, and -stringPrefs, which are used to pass values of prefs that are required very early in the content process’s existence, before the first normal IPC message is received. These pref values are encoded compactly, but there are enough of them that they make the command quite long.

Furthermore, there is one pref (network.IDN.blacklist_chars) that contains a bunch of unusual characters. That explains the gobbledygook on the 4th and 3rd last lines. Unsurprisingly, this gobbledygook was reported as a bug… which has been duplicated five times.

The good news is that I recently fixed this, by changing things so that only pref values that have changed since startup get passed in this way. (Content processes are able to see the startup values, and so don’t need to be told them.) On my machine and profile, this reduces the number of pref values passed via the command from ~222 to ~3. It also reduces the number sent later via IPC from ~3165 to ~180.

The command now looks like this:

/home/njn/local/install/firefox/firefox -contentproc -childID 4 -isForBrowser -b
oolPrefs 37:1|235:1|257:1|294:1| -stringPrefs 280:36;8bdf7a6e-d39a-494c-b55a-829
2c4fc6254| -schedulerPrefs 0001,2 -greomni /home/njn/local/install/firefox/omni.
ja -appomni /home/njn/local/install/firefox/browser/omni.ja -appdir /home/njn/lo
cal/install/firefox/browser 3237 true tab

That’s 361 characters, all of them ASCII. Much better! And I have a plan to reduce things even further.

This change is in Firefox 60, which is on track to be released around May 9th.