MemShrink progress, week 105–108

This is the first of the every-four-weeks MemShrink reports that I’m now doing.  The 21 bugs fixed in the past four weeks include 11 leak fixes, which is great, but I won’t bother describing them individually.  Especially when I have several other particularly impressive fixes to describe…

Image Handling

Back in March I described how Timothy Nikkel had greatly improved Firefox’s handling of image-heavy pages.  Unfortunately, the fix had to be disabled in Firefox 22 and Firefox 23 because it caused jerky scrolling on pages with lots of small images, such as Pinterest.

Happily, Timothy has now fixed those problems, and so his previous change has been re-enabled in Firefox 24.  This takes a big chunk out of the #1 item on the MemShrink big ticket items list.  Fantastic news!

Lazy Bytecode Generation

Brian Hackett finished implementing lazy bytecode generation.  This change means that JavaScript functions don’t have bytecode generated for them until they run.  Because lots of websites use libraries like jQuery, in practice a lot of JS functions are never run, and we’ve found this can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.

Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

The completion of this finished off item #5 on the old Memshrink big ticket items list.  Great stuff.  This will be in Firefox 24.

Add-on Memory Reporting

Nils Maier implemented add-on memory reporting in about:memory.  Here’s some example output from my current session.

├───33,345,136 B (05.08%) -- add-ons
│   ├──18,818,336 B (02.87%) ++ {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
│   ├──11,830,424 B (01.80%) ++ {59c81df5-4b7a-477b-912d-4e0fdf64e5f2}
│   └───2,696,376 B (00.41%) ++ treestyletab@piro.sakura.ne.jp/js-non-window/zones/zone(0x7fbd7bf53800)

It’s obvious that Tree Style Tabs is taking up 2.7 MB.  What about the other two entries?  It’s not immediately obvious, but if I look in about:support at the “extensions” section I can see that they are AdBlock Plus and ChatZilla.

If you’re wondering why those add-ons are reported as hex strings, it’s due to a combination of the packaging of each individual add-on, and the fact that the memory reporting code is C++ and the add-on identification code is JS and there aren’t yet good APIs to communicate between the two.  (Yes, it’s not ideal and should be improved, but it’s a good start.)  Also, not all add-on memory is reported, just that in JS compartments;  old-style XUL add-ons in particular can have their memory consumption under-reported.

Despite the shortcomings, this is a big deal.  Users have been asking for this information for years, and we’ve finally got it.  (Admittedly, the fact that we’ve tamed add-on leaks makes it less important than it used to be, but it’s still cool.)  This will also be in Firefox 24.

b2g

Gregor Wagner has landed a nice collection of patches to help the Twitter and Notes+ apps on B2G.

While on the topic of B2G, in today’s MemShrink meeting we discussed the ongoing problem of slow memory leaks in the main B2G process.  Such leaks can cause the phone to crash or become flaky after its been running for hours or days or weeks, and they’re really painful to reproduce and diagnose.  Our partners are finding these leaks when doing multi-hour stress tests as part of their QA processes.  In contrast, Mozilla doesn’t really have any such testing, and as a result we are reacting, flat-footed, to external reports, rather than catching them early ourselves.  This is a big problem because users will rightly expect to have their phones run for weeks (or even months) without rebooting.

Those of us present at the meeting weren’t quite sure how we can improve our QA situation to look for these leaks.  I’d be interested to hear any suggestions.  Thanks!

12 Responses to MemShrink progress, week 105–108

  1. Is the addons memory reports more accurate than the addon for the purpose called “about:addons-memory“, or is it the same functionality?

    That addon provides the names of the addons and has a nice and clean interface. So, if they both provide the same accuracy, I think I will stick with that addon.

    • Nicholas Nethercote

      Same author, and much the same approach, as far as I know.

    • about:addons-memory seems more accurate in one case (see below about https-everywhere). Both implementations are by Nils Maier. I do prefer looking at the built-in interface, since it shows everything and the add-ons I keep are low on the list (though they are easily actionable, unlike a lot of the report).

  2. https-everywhere does have a compartment (…/components/https-everywhere.js), but it doesn’t get reparented to the add-on hierarchy. After a bit of testing I see it’s because I’m running it from git (https-everywhere@eff.org is a plain file containing the path to the git clone).

  3. I’m not a developer and I don’t know what’s possible or not possible, so I’m just going to throw out stupid stuff that would hopefully spur thinking…

    You said you guys weren’t “quite sure how we can improve our QA situation to look for these leaks.” I was going to say that our partners found them, so let’s do whatever they did. But did you mean that they merely observed that there *were* leaks? Not really identifying the roots?

    Say, it wouldn’t be possible to emulate the hardware and then watch/record everything, would it?
    I was thinking the other day that maybe it’d be interesting/helpful if I could create a virtual machine in VirtualBox for B2G hardware.
    Might be nice if you could just freeze everything and then take a sort of snapshot, then wait a bit and do it again. Would that even be helpful? If so maybe it could be build it into the OS itself.
    Perhaps what’s necessary would seem extreme so it’s hard to think of. Like, maybe attaching a secondary storage to a device and writing out massive logs.

    Hope my baseless, nonsensical words (in my sleepy stupor) spurred at least a thought.

    • Nicholas Nethercote

      I must not have explained myself clearly. The problem is more a personnel/organizational one than a technical one. I.e. who should be doing this testing? Which tests should they run? Etc.

  4. Thanks as always for the update
    It’s particularly good news that some of the big items will be in FF24 as that will be the basis of the next enterprise release. There will be a lot of people using that release until the middle of next year.

    • Nicholas Nethercote

      Good point. And it (or maybe FF25) will also be the basis of a future Firefox OS point release!

  5. Could the period before commenting is disabled be extended so that the most recent memshrink report is always open instead of being locked down for several weeks before the next one is published?

  6. The generational garbage collector has recently made it’s way into Are We Fast Yet test runs. Do you have any idea when it might be ready for Are We Slim Yet testing?

    • Nicholas Nethercote

      AWSY just tests a single configuration, whereas AWFY tests multiple configurations. So the short answer is “when it’s enabled for good”.

      Having said that, it’s possible to do trial runs on AWSY, and it’s a good idea — I’ll try doing one next week with GGC enabled.

  7. Ken Roberts

    Nicholas,
    I find your posts a joy to read, each one a little present.