The Old Big Ticket Items List

#5: Better Script Handling

This had two parts.  The first part was the sharing of immutable parts of scripts, which Till Schneidereit implemented.  It can save multiple MiBs of memory, particular if you have numerous tabs open from the same website.

The second part is lazy bytecode generation, which Brian Hackett has implemented and landed, though it hasn’t yet enabled.  Brian is working out the remaining kinks and hopes to land by the end of the current (v24) cycle.    Hopefully he will, because measurements have shown that it can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.  Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

So the first part is done and the second is imminent, which is enough to cross this item off the list.  [Update:  Brian just enabled lazy bytecode on trunk!]

#4: Regain compartment-per-global losses

Bill McCloskey implemented zones, which restructured the heap to allow a certain amount of sharing between zones. This greatly reduced wasted space and reduced memory consumption in common cases by ~5%.

Some more could still be done for this issue.  In particular, it’s not clear if things have improved enough that many small JSMs can be used without wasting memory.  Nonetheless, things have improved enough that I’m happy to take this item off the list.

#3: Boot2Gecko

This item was about getting about:memory (or something similar) working on B2G, and using it to optimize memory.  This was done some time ago and the memory reporters (plus DMD) were enormously helpful in improving memory consumption.  Many of the fixes fell under the umbrella of Operation Slim Fast.

So I will remove this particular item from the list, but memory optimizations for B2G are far from over, as we’ll see below.

#2: Compacting Generational GC

See below.

#1: Better Foreground Tab Image Handling

See below.

The New Big Ticket Items List

#5: pdf.js

pdf.js was recently made the default way of opening PDFs in Firefox, replacing plug-ins such as Adobe Reader.  While this is great in a number of respects, such as security, it’s not as good for memory consumption, because pdf.js can use a lot of memory in at least some cases.  We need to investigate the known cases and improve things.

#4: Dev tools

While we’ve made great progress with memory profiling tools that help Firefox developers, the situation is not nearly as good for web developers.  Google Chrome has heap profiling tools for web developers, and Firefox should too.  (The design space for heap profilers is quite large and so Firefox shouldn’t just copy Chrome’s tools.)  Alexandre Poirot has started work on a promising prototype, though there is a lot of work remaining before any such prototype can make it into a release.

#3: B2G Nuwa

Cervantes Yu and Thinker Li have been working on Nuwa, which aims to give B2G a pre-initialized template process from which every subsequent process will be forked.  This might sound esoteric, but the important part is that it greatly increases the ability for B2G processes to share unchanging data.  In one test run, this increased the number of apps that could be run simultaneously from five to nine, which is obviously a big deal.  The downside is that getting it to work requires some extremely hairy fiddling with low-level code.  Fingers crossed it can be made to work reliably!

Beyond Nuwa, there is still plenty of other ways that B2G can have its memory consumption optimized, as you’d expect in an immature mobile OS.  Although I won’t list anything else in the big ticket items list, work will continue here, as per MemShrink’s primary aim: “MemShrink is a project that aims to reduce the memory consumption of Firefox (on desktop and mobile) and Firefox OS.”

#2: Compacting Generational GC

Generational GC will reduce fragmentation, reduce the working set size, and speed up collections.  Great progress has been made here — the enormous task of exactly rooting the JS engine and the browser is basically finished, helped along greatly by a static analysis implemented by Brian Hackett.  And Terrence Cole and others are well into converting the collector to be generational.  So we’re a lot closer than we were, but there is still some way to go.  So this item is steady at #2.

#1: Better Foreground Tab Image Handling

Firefox still uses much more memory than other browsers on image-heavy pages.  Fortunately, a great deal of progress has been made here.  Timothy Nikkel fixed things so that memory usage when scrolling through image-heavy pages is greatly reduced.  However, this change caused some jank on pages with lots of small images, so it’s currently disabled on the non-trunk branches.  Also, there is still a memory spike when memory-heavy pages are first loaded, which needs to be fixed before this item can be considered done.  So this item remains at #1.

Summary

Three items from the old list (#3, #4, #5) have been ticked off.  Two items remain (#1, #2) — albeit with good progress having been made — and keep their positions on the list.  Three items have been added to the new list (#3, #4, #5).
Let me know if I’ve omitted anything important!

Fixes

areweslimyet.com proved its value again, identifying a 40 MiB regression relating to layer animations.  Joe Drew fixed the problem quickly.  Thanks, Joe!

Ben Turner changed the GC heuristics for web workers on B2G so that less garbage is allowed to accumulate.  This was a MemShrink:P1.

Bas Schouten fixed a gfx top-crasher that was caused by inappropriate handling of an out-of-memory condition.  This was also a MemShrink:P1.

Randell Jesup fixed a leak in WebRTC code.

Changes to these progress reports

I’ve written over sixty MemShrink progress reports in the past two years.  When I started writing them, I had two major goals.  First, to counter the notion that nobody at Mozilla cares about memory consumption — this was a common comment on tech blogs two years ago.  Second, to keep interested people informed of MemShrink progress.

At this point, I think the first goal has been comprehensively addressed.  (More due to the  fact that we actually have improved Firefox’s memory consumption than because I wrote a bunch of blog posts!)  As a result, I feel like the value of these posts has diminished.  I’m also tired of writing them;  long-time readers may have noticed that they are much terser than they used to be.

Therefore, going forward, I’m only going to post one report every four weeks.  (The MemShrink team will still meet every two weeks, however.)  I’ll also be more selective about what I write about — I’ll focus on the big improvements, and I won’t list every little leak that has been fixed, though I might post links to Bugzilla searches that list those fixes. Finally, I won’t bother recording the MemShrink bug counts.  (In fact, I’ve started that today.)  At this point it’s pretty clear that the P1 list hovers between 15 and 20 most of the time, and the P2 and P3 lists grow endlessly.

Apologies to anyone disappointed by this, but rest easy that it will give me more time to work on actual improvements to Firefox

I look forward to a mountain of bugmail when I return.

Measurements aren’t taken away;  you have to click on the “Measure” button for that to happen.  Also, adding ?verbose to the URL no longer has an effect.  If you want verbose output, you need to click on the “verbose” checkbox.

The motivation for this change was that about:memory’s functionality has expanded greatly over the past two years, and cramming more functionality into the existing UI was a bad idea.  There are numerous advantages to the new UI.

• No need to control behaviour via the URL, which is admittedly an odd way to do it.
• You can switch between verbose and non-verbose modes without having to reload the page.
• All the buttons are at the top of the page instead of the bottom, so you don’t have to scroll down.
• You can trigger a GC/CC/minimize memory usage event without having do a memory measurement.
• When you save reports to file, it’s clearer that a new measurement will be taken, rather than saving any measurements that are currently displayed on the screen.
• The buttons are grouped by function, which makes them easier to understand.

There’s also the “Load and diff…” button, which lets you easily find the difference between two saved memory report files.  Here’s some example output taken after closing a tab containing a Google search result.

You can see that all the measurements are negative.  The [-] markers on leaf nodes in the tree indicate that they are present in the first file but not in the second file.  Corresponding [+] markers are used for measurements present in the second file but not the first.

Kyle Huey made image.src='' discard the image immediately even if the image is not in the document.  This provides a way for image memory to be discarded immediately, which is important for some B2G apps such as the Gallery app.  This was a MemShrink:P1 bug.

Justin Lebar fixed a bad leak in AudioChannelAgent.

Mike Habicher fixed an image-related leak that was manifesting on B2G.

Other Fixes

I exposed the existing JSON memory report dumping functionality in about:memory.  As a result, gzipped JSON is now the preferred format for attaching memory report data to bugs.  This was a MemShrink:P1 bug.

Honza Bambas overhauled the DOM storage code.  Apparently this might reduce memory consumption, but I fully admit to not knowing the details.

Nicolas Silva fixed a leak in layers code relating to OMTC.

I removed the MEMORY_EXPLICIT telemetry measurement.  I didn’t want to do this, but it’s been causing hangs for some users, and those hangs are hard to avoid due to the complexity of reporting memory consumption from web workers.  Furthermore, in practice the measurement was sufficiently noisy that we’ve never gotten anything useful from it.

Help Needed

Mozilla code uses a mixture of fallible and infallible allocations.  In theory, any allocation that could allocate a large amount of memory (e.g. a few hundred KiB or more) should use a fallible allocator.  We’ve seen recently some cases where large allocations were being made infallibly, which led to OOM crashes.  Bug 862592 proposes adding an assertion to the infallible allocators that the request size isn’t too large.  This is an easy bug to get started with, if anyone is interested.  Details are in the bug.

Bug Counts

Here are the current bug counts.

• P1: 15 (-4/+4)
• P2: 145 (-0/+7)
• P3: 131 (-2/+4)
• Unprioritized: 5 (-1/+5)

Interregnum

I will be on vacation during May.  As a result, there will be no MemShrink reports for the next three fortnights.  (Although I’ll be away for just over four weeks, that period covers three MemShrink meetings.)  I’ll be back with the next report on June 11.  See you then!

about:memory is our main memory profiling tool.  You can copy and paste the memory reports shown by about:memory and they will reproduce nicely — so long as the displaying text is a fixed-width font — as the following example shows.

Main Process

Explicit Allocations
536.22 MB (100.0%) -- explicit
├──217.45 MB (40.55%) -- window-objects
│  ├───91.55 MB (17.07%) -- top(https://mail.google.com/mail/u/0/#label/A-moz%2Fbugz/13d41da77907a707, id=16)/active
│  │   ├──61.18 MB (11.41%) -- window(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  ├──56.29 MB (10.50%) -- js
│  │   │  │  ├──56.11 MB (10.46%) -- compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  │  │  ├──35.23 MB (06.57%) -- gc-heap
│  │   │  │  │  │  ├──18.17 MB (03.39%) -- objects
│  │   │  │  │  │  │  ├───9.88 MB (01.84%) ── dense-array
│  │   │  │  │  │  │  ├───6.08 MB (01.13%) ── ordinary
│  │   │  │  │  │  │  └───2.21 MB (00.41%) ++ (2 tiny)
│  │   │  │  │  │  ├──10.71 MB (02.00%) ++ (6 tiny)
│  │   │  │  │  │  └───6.35 MB (01.18%) ── unused-gc-things
│  │   │  │  │  ├──12.01 MB (02.24%) ++ (7 tiny)
│  │   │  │  │  └───8.86 MB (01.65%) ++ objects-extra
│  │   │  │  └───0.19 MB (00.04%) ++ compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1, about:blank)
│  │   │  └───4.89 MB (00.91%) ++ (4 tiny)

Accordingly, until now the preferred way of reporting about:memory’s data in bug reports has been to include a copy of the entire contents of about:memory?verbose.

This worked well when about:memory contained a few dozen lines.  But the level of detail in the underlying memory reporters has increased greatly since then.  This is a good thing — more data is helpful — but it means that a full about:memory?verbose is now typically thousands of lines.  When viewing the data directly in about:memory?verbose this isn’t a problem, because you can easily collapse and expand sub-trees to focus on the interesting parts.  However, when reading pasted data as text, it’s overwhelming, even for experts.

New: Gzipped JSON

With the landing of bug 848560, about:memory can now import and export memory reports as gzipped JSON.  Use the “Write reports to a file” and “Read reports from a file” buttons at the bottom of about:memory.  (Note: the UI will likely change soon, but not substantively.)

(The JSON schema is documented in a comment near the top of xpcom/base/nsIMemoryInfoDumper.idl, for those who are interested.)

This is now the preferred format for attaching memory report data to bug reports.  It has two important advantages over text.

• When you load one of these files into about:memory, you can do the aforementioned collapsing and expanding of sub-trees.
• If you have two of these files, you can diff them using toolkit/components/aboutmemory/tools/diff-memory-reports.js.  (For the moment you’ll have to ungzip both files first, which is stupid.  Hopefully this’ll be fixed soon.)

The ability to compute memory report diffs will be extremely useful for diagnosing cases where Firefox’s memory consumption increases unexpectedly.  (There are already two bugs that are waiting on this feature for a better diagnosis.)

There is a bug open to implement a “Load reports from URL” feature in about:memory, which will make the viewing memory reports attached to bugs even easier.  It’s currently unassigned;  please email me or comment in the bug if you are interested in implementing it.

Finally, although the gzipped JSON format is now the preferred format for transferring the full memory report data, copy and paste is still useful for communicating snippets of about:memory’s contents.  Support for it won’t be disappearing.

Hot damn, those are some impressive projects.

AWSY

areweslimyet.com has been proving its worth.

Jonathan Kew reduced the amount of memory taken by fonts on Fennec at start-up, which was detected by AWSY/mobile.  Jonathan also reverted a change that AWSY detected as increasing Fennec memory consumption, and filed a follow-up to investigate further.

Joe Drew fixed a bad regression on AWSY relating to image decoding.  It’s not clear to me if this was a genuine regression that users would have seen, or if it was an artifact of the way AWSY does its measurements.  Either way, it’s good that it was fixed, and props to Joe for doing it so quickly.

Finally, we closed bug 833518, which was for an AWSY regression caused by the new DOM bindings.  This was previously improved by an Aurora-only hack, but enough cases have been translated to the new bindings that we’re naturally down almost to where we were.

Miscellaneous

The mobile team abandoned their goal of making Fennec work on phones with only 256 MiB of memory.  The rationale is that Android phones with only 256 MiB of RAM are uncommon, whereas low-end phones that meet the current minimum of 384 MiB are much more common.  The mobile team will of course continue to look for ways to improve memory consumption in order to make life for users with 384 MiB phones.

I modified the JS engine so that it doesn’t emit bytecode for asm.js functions in the normal case.  This reduced the memory consumption of the Unreal 3 demo used at GDC by about 100 MiB.  I also added a memory reporter for array buffers used by asm.js, which are often quite large and weren’t being measured on 64-bit platforms.

Alexandre Poirot fixed a leak relating to dev tools.

Randell Jesup fixed a small leak in WebRTC.

Help Needed

I’m working on adding a button to about:memory trigger the dumping of memory reporter data to file.  I have a patch awaiting review, but I’m getting a test failure on Windows.  The test saves gzipped memory reports to file, and then immediately loads that saved file (and uncompresses it) and checks the data looks as expected.  This works fine on Mac and Linux, but on Windows I’m sometimes getting incomplete data in the load step.  The file is quite short (just 253 bytes compressed, and 620 bytes uncompressed) and the truncation point varies between runs;  in the most severe occurrence only 9 bytes of uncompressed data were loaded, though the cut-off point seems to vary randomly.

I suspect there’s a file synchronization problem between the save and the load, even though gzclose() is called on the save file before the loading occurs.  If anyone has ideas about what the problem might be, I’d love to hear them.

Update: Nils Maier and an anonymous commenter pointed out the problem — I was using “r” instead of “rb” for the file mode.  On Windows, this causes mangling of EOL chars.

Also, we’re seeing some strange behaviour on Mac OS X where memory managed by jemalloc doesn’t appear to be released back to the OS as it should.  This is both alarming and hard to understand, which is not a good combination.

Good First Bugs

I have two easy bugs assigned to me that I probably won’t get around to for some time.  Both of them would be good first (or second, or third…) bugs.

• Bug 857382 is about making about:memory handle memory report diffs more elegantly.
• Bug 798914 is just a minor code clean-up.  Nothing too exciting, but first bugs often aren’t!

Please email or comment in one of the bugs if you are interested in helping.

Bug Counts

Here are the current bug counts.

• P1: 15 (-0/+2)
• P2: 138 (-4/+8)
• P3: 129 (-2/+7)
• Unprioritized: 1 (-3/+0)

There’s no way to specify this combination in a single try server push, but I realized I could do it with two separate pushes.  TryChooser told me that Linux64 was the least-loaded platform, so I did a try run with all the tests on that platform:

try: -b do -p linux64 -u all -t none

And then I did a try run that did builds (but no tests) on all platforms except Linux64:

try: -b do -p linux,macosx64,win32,android,android-armv6,android-noion,ics_armv7a_gecko,panda,otoro,unagi -u none -t none

Interesting.  I wonder if this is a combination that other people have considered.

The following graph from areweslimyet.com shows the impact of zones — about 5/6 of the way along the graph there’s a distinct drop, most noticeable in the dark blue line.  The light green line (settled start-up) showed a ~6 MiB drop, which is ~10%.  Note that the fraction of JS memory in areweslimyet.com is less than that in typical sites, so the drop in the higher lines is smaller than the improvements normal users should see.

Avi Halachmi fixed a problem where badly managed gradients could cause spikes of 10s of MiBs when tab animations occurred.  This was a MemShrink:P1 bug.  The fix has been backported to Aurora.

Jed Parsons fixed excessive memory consumption by Persona after visiting the B2G marketplace.  At least, I think that’s what happened;  I won’t pretend to genuinely understand what went on in that bug.  This was a MemShrink:P1 bug.

Fabrice Desré fixed a bad B2G leak relating to error messages.  This bug was fixed before it was triaged in a MemShrink meeting, but it probably would have been a MemShrink:P1 because it could cause the phone to OOM after a few hours.

I removed all uses of nsFixedSizeAllocator.  This was only a small memory consumption win (a few 10s of KiBs) but it cut several hundred lines of code, and removed another low-quality custom allocator (and attractive nuisance) from the codebase.

I added a “js-main-runtime-temporary-peak” memory reporter, which measures the peak size of the main JSRuntime’s “temporary” pool, which mostly holds parse nodes.  These are short-lived but they can be quite large — 10s of MiBs in normal browsing, and we’ve seen it exceed 1.5 GiB on large asm.js examples.  Relatedly, I reduced the size of parse nodes by 12.5% on 64-bit platforms, and 16.7% on 32-bit platforms.

Interesting Open Bugs

Sometimes, especially on B2G, we have excessive memory consumption due to JS strings.  It would be useful to be able to dump the contents of all the strings in memory, to look for ones that shouldn’t be there.  Bug 852010 has been filed for this.  It shouldn’t be too hard, as it would fit in with the existing JSON memory report dumping infrastructure.  This is currently blocking bug 850893, a B2G MemShrink:P1 bug.  Please comment in the bug or contact me if you want to get involved.

Bug 846616 and bug 850545 both are about high “heap-unclassified” values.  Reducing “heap-unclassified” is getting very difficult, because the memory is often allocated in ways we can’t measure by third-party code such as Cairo and graphics drivers.  I suppose in the Cairo case we could put effort into adding some memory profiling infrastructure and try to get it upstreamed, but the driver situation seems pretty hopeless.

Bug Status

Here are the current bug counts.

• P1: 13 (-3/+4)
• P2: 134 (-2/+8)
• P3: 124 (-4/+6)
• Unprioritized: 4 (-6/+4)