Greg has been leading the way technically in build system work for a
while now, so in my mind this is more of a formality than anything else.
You have undoubtedly already interacted with the great work he’s been
doing: mach, the fantastic new front-end driver for the build system, or
moz.build, our Makefile replacement for build system data. I expect that
this will continue, so making him module owner was an easy decision for me.

I will continue to serve as a peer of the Build config module, but Greg
will now have final say on all questions regarding build system work.

[extensions]
color =
pager =

[pager]
pager = LESS=FRSXQ less
quiet = True

The pager configuration simply makes less handle the colored output properly, and also behave like cat if the output fits on the screen.

I frequently find myself using hg export to view the contents of patches in my mq, so I also added an alias that I could use instead to get nice colored and pagered diffs:

[alias]
show = log --patch --verbose --rev

[pager]
attend = diff,status,log,qdiff,blame,annotate,pdiff,glog,show

The pager configuration is necessary because by default pager only works on a whitelist of commands, so you need to add this new show alias. Then you can simply use it like hg show tip to view the contents of your topmost mq patch.

I’ve only tested this configuration on Ubuntu 12.04 with Mercurial 2.0.2, so I’d be interested to hear if it works elsewhere.

How is this relevant to Firefox Mobile? Currently the builds we’re producing only run on processors that support ARMv7. This is partially because we’ve been working on performance for quite a while, and it’s much harder to get acceptable performance on a slower processor, so targeting only faster processors makes sense. (This is the same reason that Chrome for Android only runs on Android phones running the latest version of Android.) It’s also partially because all modern JavaScript engines ship with a JIT, which is a highly specialized piece of code that needs to know intimate details about the type of processor it’s running on. We used to produce additional builds that supported ARMv6 alongside our ARMv7 builds, but we saw lots of ARMv6-specific crashes in our crash reporting system, and we didn’t have the resources to tackle them all. Additionally, we were focused on making Firefox Mobile run well on ARMv7 processors; so making it run well on ARMv6 seemed like a stretch at the time.

Coming back to the present, we’ve got a revitalized mobile team working on a revamped Firefox Mobile that’s much faster than previous versions, so the performance target seems much more within reach. We also had people attending MozCamps and other Mozilla events across the globe last year. Dietrich visited Nairobi for some Mozilla Kenya events and found that the most widely used Android phones in Kenya are all ARMv6 devices. In addition, there are lots of Android phones being sold in China that are ARMv6. Even in the USA there are some low-end Android devices being released that are still ARMv6, like the LG Optimus Hub, which shipped in October of 2011. As of that date roughly 58% of the Android install base was comprised of ARMv6 phones. That’s a huge segment of the market that we’re not supporting.

Because of this, during the Firefox Mobile revamp Doug roped me in and asked if I would look into getting our ARMv6 builds back up and running. I started working on it figuring it wouldn’t be too bad since we used to produce builds. As it turns out, I was wrong. We managed to break things in quite a few ways since we disabled those builds. A few of them were simple fixes in our build configuration (although one of those took Mike Hommey and I a solid week of debugging to track down), but I also ran into a few problems with our custom linker. Firefox Mobile ships with a replacement for the system dynamic linker on Android. It’s pretty complicated, but this is the reason that Firefox only takes up about 15MB, whereas Chrome for Android takes up nearly 50MB after installation. Being a complicated piece of code there were some hard-to-diagnose bugs in it. Thankfully, with some input from Jacob Bramley from ARM we were able to track down the remaining problem and get builds working again.

With all the setbacks and other issues it’s not unreasonable to ask why we’re doing this. Clearly this isn’t the end of the process by any means. We still have to get automated builds back up and running on our build farm. We will undoubtedly have to shake out more ARMv6-specific bugs in our JavaScript engine and elsewhere. We’ll almost assuredly have to do some work to make performance acceptable. It’s a lot of work and it will take time, but this seems like the right thing to do given the number of users we can reach. You can follow along in Bugzilla if you’re interested in this work.

We decided to make the manifests support boolean expressions, similar to what reftest manifests allow, except with a restricted grammar, and not “all of JavaScript”. To make this useful, we had to offer a set of values to test, so Jeff Hammel buckled down and wrote a Python module called mozinfo that had been in discussion for a long time. I wrote a few bits to hook this all up between the build system and the xpcshell test harness, and it all landed on mozilla-central this morning.

I’ll update the MDN documentation later today, but for a preview, a sample manifest entry might look like:

[test_foo.js]
skip-if = os == 'win' || os == 'linux'

If you look in your object directory in a build containing these patches (or in the xpcshell directory of a test package from a tinderbox build), you’ll find a mozinfo.json, which is where most of the values you can use in these expressions come from. For example, the mozinfo.json for my 64-bit Linux build looks like:

{'os': 'linux', 'toolkit': 'gtk2', 'crashreporter': false, 'debug': false, 'bits': 64, 'processor': 'x86_64'}

You can annotate tests with “skip-if, run-if and fail-if” conditions currently. “skip-if” indicates that a test should not be run if the condition evaluates to true, “run-if” indicates that a test should only be run if the condition evaluates to true and “fail-if” indicates that the test is known to fail if the condition is true. Tests marked fail-if will produce TEST-KNOWN-FAIL output if they fail, and TEST-UNEXPECTED-PASS (which is treated as a failure) if they pass.

Hopefully this work will enable developers to more easily work with xpcshell tests. We’d appreciate any feedback you have on these changes!

Dietrich has done some work on this, writing an add-on that measures the time that various UI actions take. This covers the direct case, where a user initiates an action and expects a response in a reasonable amount of time. Clearly we want to make sure that individual actions don’t take an extraordinary amount of time.

I took the opposite tack, with an eye on being able to detect when the application was not responsive to user input regardless of what actions the user was taking. Building on some work by Chris Jones and Alon Zakai, I wrote some code that instruments the main thread event loop to find out how long it takes to respond to events, which ought to be a reasonable proxy for measuring responsiveness. When the instrumentation detects that the event loop takes too long to respond (more than 50 milliseconds, currently) it writes a data point to a log giving the current timestamp and the amount of time the event loop was not responsive.

When I implemented this I had my eye on Talos integration, where we could run the browser through some automated UI tests with this instrumentation enabled, and then correlate “UI actions” with “unresponsive periods” and ensure that the browser did not become unresponsive during those actions. Talos integration has been shifted off as a longer-term goal, with the more immediate goal being “find UI actions that are the worst offenders of unresponsiveness”. To that end we’ve filed some other bugs about correlating this unresponsiveness data with JavaScript execution, and correlating the data with C++ execution. If you’ve got any ideas please feel free to contribute to those bugs!

If you’d like to try out the responsiveness instrumentation I implemented, it landed on mozilla-central a while ago, and there’s some reasonably complete documentation in the source code. There are implementations for Windows, Linux/GTK2 and OS X currently. (And a patch for an Android implementation in a bug.)

$ cut -f1 -d, 20110613-modulelist.txt  | sort -u | wc -l
10385

There are over 10,000 unique filenames in a single day’s worth of crash reports. That sure seems like a lot! Now, certainly, a lot of these modules look like they’ve been randomly named, which probably indicates that they’re some kind of virus (like 0eYZf0QFDSGEAbTRWD3F.dll, for example), so those are likely to inflate the number. There’s a bug on file asking that we collect MD5 hashes of every DLL in our crash reports so we could more easily detect malware/virus DLLs that use these tactics, as well as integrate with lists of known malware and viruses from antivirus vendors.

In the past, we have had problems with plugins and extensions causing crashes for many Firefox users. We have ways of mitigating those through blacklisting. We can also blacklist specific DLLs from loading in the Firefox process, which is not used as often because it’s harder to get right and provides little feedback to users about what’s been disabled. However, given the sheer number of possible things that can be loaded in our process, it’s unlikely that we’ll ever be able to block all software that causes crashes for users. This is unfortunate, because any one of these pieces of software can cause a crash in Firefox, and all the user sees is “Firefox crashed“. I suppose we now know how Microsoft feels when users blame Windows for crashes caused by faulty drivers.

hg bzexport [REV] [BUG]

where REV is the name of a currently applied patch from your queue, and BUG is a bug number to attach it to. Of course, the extension is smarter than just that, and if you leave off those parameters it will default to working on the topmost patch in your queue, and it will attempt to deduce the bug number from the commit message in the patch. (Standard notation used at Mozilla should work, i.e. “bug 12345 – some text”, “b=12345 some text” etc.)

bzexport attempts to be clever and borrow your Bugzilla login cookies from your default Firefox profile so that you don’t have to provide authentication details. This does not currently work on the Mercurial shipped with MozillaBuild, so you’ll have to provide your username and password in your .hgrc as described in the README if you intend to use it under MozillaBuild’s hg.

bzexport relies on Gerv‘s excellent BzAPI, so hats off to Gerv for that! Also, patches for additional functionality are welcome.