Category Archives: about:memory

Gzipped JSON is now the preferred format for attaching memory report data to bugs

Old: Text

about:memory is our main memory profiling tool.  You can copy and paste the memory reports shown by about:memory and they will reproduce nicely — so long as the displaying text is a fixed-width font — as the following example shows.

Main Process

Explicit Allocations
536.22 MB (100.0%) -- explicit
├──217.45 MB (40.55%) -- window-objects
│  ├───91.55 MB (17.07%) -- top(https://mail.google.com/mail/u/0/#label/A-moz%2Fbugz/13d41da77907a707, id=16)/active
│  │   ├──61.18 MB (11.41%) -- window(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  ├──56.29 MB (10.50%) -- js
│  │   │  │  ├──56.11 MB (10.46%) -- compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  │  │  ├──35.23 MB (06.57%) -- gc-heap
│  │   │  │  │  │  ├──18.17 MB (03.39%) -- objects
│  │   │  │  │  │  │  ├───9.88 MB (01.84%) ── dense-array
│  │   │  │  │  │  │  ├───6.08 MB (01.13%) ── ordinary
│  │   │  │  │  │  │  └───2.21 MB (00.41%) ++ (2 tiny)
│  │   │  │  │  │  ├──10.71 MB (02.00%) ++ (6 tiny)
│  │   │  │  │  │  └───6.35 MB (01.18%) ── unused-gc-things
│  │   │  │  │  ├──12.01 MB (02.24%) ++ (7 tiny)
│  │   │  │  │  └───8.86 MB (01.65%) ++ objects-extra
│  │   │  │  └───0.19 MB (00.04%) ++ compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1, about:blank)
│  │   │  └───4.89 MB (00.91%) ++ (4 tiny)

Accordingly, until now the preferred way of reporting about:memory’s data in bug reports has been to include a copy of the entire contents of about:memory?verbose.

This worked well when about:memory contained a few dozen lines.  But the level of detail in the underlying memory reporters has increased greatly since then.  This is a good thing — more data is helpful — but it means that a full about:memory?verbose is now typically thousands of lines.  When viewing the data directly in about:memory?verbose this isn’t a problem, because you can easily collapse and expand sub-trees to focus on the interesting parts.  However, when reading pasted data as text, it’s overwhelming, even for experts.

New: Gzipped JSON

With the landing of bug 848560, about:memory can now import and export memory reports as gzipped JSON.  Use the “Write reports to a file” and “Read reports from a file” buttons at the bottom of about:memory.  (Note: the UI will likely change soon, but not substantively.)

(The JSON schema is documented in a comment near the top of xpcom/base/nsIMemoryInfoDumper.idl, for those who are interested.)

This is now the preferred format for attaching memory report data to bug reports.  It has two important advantages over text.

  • When you load one of these files into about:memory, you can do the aforementioned collapsing and expanding of sub-trees.
  • If you have two of these files, you can diff them using toolkit/components/aboutmemory/tools/diff-memory-reports.js.  (For the moment you’ll have to ungzip both files first, which is stupid.  Hopefully this’ll be fixed soon.)

The ability to compute memory report diffs will be extremely useful for diagnosing cases where Firefox’s memory consumption increases unexpectedly.  (There are already two bugs that are waiting on this feature for a better diagnosis.)

There is a bug open to implement a “Load reports from URL” feature in about:memory, which will make the viewing memory reports attached to bugs even easier.  It’s currently unassigned;  please email me or comment in the bug if you are interested in implementing it.

Finally, although the gzipped JSON format is now the preferred format for transferring the full memory report data, copy and paste is still useful for communicating snippets of about:memory’s contents.  Support for it won’t be disappearing.

MemShrink progress, week 93–94

After lots of activity in the previous month, the past two weeks have been fairly quiet for MemShrink.

AWSY

areweslimyet.com has been proving its worth.

Jonathan Kew reduced the amount of memory taken by fonts on Fennec at start-up, which was detected by AWSY/mobile.  Jonathan also reverted a change that AWSY detected as increasing Fennec memory consumption, and filed a follow-up to investigate further.

Joe Drew fixed a bad regression on AWSY relating to image decoding.  It’s not clear to me if this was a genuine regression that users would have seen, or if it was an artifact of the way AWSY does its measurements.  Either way, it’s good that it was fixed, and props to Joe for doing it so quickly.

Finally, we closed bug 833518, which was for an AWSY regression caused by the new DOM bindings.  This was previously improved by an Aurora-only hack, but enough cases have been translated to the new bindings that we’re naturally down almost to where we were.

Miscellaneous

The mobile team abandoned their goal of making Fennec work on phones with only 256 MiB of memory.  The rationale is that Android phones with only 256 MiB of RAM are uncommon, whereas low-end phones that meet the current minimum of 384 MiB are much more common.  The mobile team will of course continue to look for ways to improve memory consumption in order to make life for users with 384 MiB phones.

I modified the JS engine so that it doesn’t emit bytecode for asm.js functions in the normal case.  This reduced the memory consumption of the Unreal 3 demo used at GDC by about 100 MiB.  I also added a memory reporter for array buffers used by asm.js, which are often quite large and weren’t being measured on 64-bit platforms.

Alexandre Poirot fixed a leak relating to dev tools.

Randell Jesup fixed a small leak in WebRTC.

Help Needed

I’m working on adding a button to about:memory trigger the dumping of memory reporter data to file.  I have a patch awaiting review, but I’m getting a test failure on Windows.  The test saves gzipped memory reports to file, and then immediately loads that saved file (and uncompresses it) and checks the data looks as expected.  This works fine on Mac and Linux, but on Windows I’m sometimes getting incomplete data in the load step.  The file is quite short (just 253 bytes compressed, and 620 bytes uncompressed) and the truncation point varies between runs;  in the most severe occurrence only 9 bytes of uncompressed data were loaded, though the cut-off point seems to vary randomly.

I suspect there’s a file synchronization problem between the save and the load, even though gzclose() is called on the save file before the loading occurs.  If anyone has ideas about what the problem might be, I’d love to hear them.

Update: Nils Maier and an anonymous commenter pointed out the problem — I was using “r” instead of “rb” for the file mode.  On Windows, this causes mangling of EOL chars.

Also, we’re seeing some strange behaviour on Mac OS X where memory managed by jemalloc doesn’t appear to be released back to the OS as it should.  This is both alarming and hard to understand, which is not a good combination.

Good First Bugs

I have two easy bugs assigned to me that I probably won’t get around to for some time.  Both of them would be good first (or second, or third…) bugs.

  • Bug 857382 is about making about:memory handle memory report diffs more elegantly.
  • Bug 798914 is just a minor code clean-up.  Nothing too exciting, but first bugs often aren’t!

Please email or comment in one of the bugs if you are interested in helping.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-0/+2)
  • P2: 138 (-4/+8)
  • P3: 129 (-2/+7)
  • Unprioritized: 1 (-3/+0)

MemShrink progress, week 83–84

Fixed

Justin Lebar made it so that memory reports could be collected on production B2G phones (i.e. not just developer phones with root access).  This was a MemShrink:P1, because getting these reports is crucial.

Gregor Wagner tuned the GC heuristics used by workers.  This is important for B2G, which uses workers extensively.

Andrew McCreight fixed a leak involving audio contexts.

I added a memory reporter for event targets, which includes XHRs.  This can measure multiple MiB of memory when running Gmail.

I added a memory reporter for data held by the JS engine’s regexp JIT compiler.  It usually measures insignificant amounts.

I fixed an inaccuracy in the “resident” memory report tree, which is visible in about:memory when running on Linux, which was caused by a change in recent kernels.

AWSY

The recent results on AWSY have been ugly.  There were two bad regressions in December, as the following graph makes clear.

areweslimyet.com, december 2012

John Schoenick did some work to improve AWSY to make regression hunting easier, and as a result we finally know which changes caused these regressions.

  • A refactoring of images code caused the bigger regression, on December 18.  Seth Fowler is looking into this.
  • Two changes relating to the new DOM bindings caused the smaller regression on December 11/12.  This is largely because many more JS getter/setter functions are present.  It’s not clear yet how to win back this memory, though it should be possible to turn these changes off in the short-term.

These regressions have made it to the Aurora branch, which means there is some urgency now to either fix them or back out/disable them soon.  We don’t want them to reach Beta.

Bug Counts

Here are the current bug counts.

  • P1: 21 (-1/+7)
  • P2: 125 (-2/+11)
  • P3: 104 (-1/+3)
  • Unprioritized: 4 (-17/+3)

The changes are larger than usual because we had a big log of untriaged bugs to go through, due to the six week break since the last MemShrink meeting.

MemShrink progress, week 79–82

I skipped the last MemShrink report due to Christmas, so we have four weeks’ worth of bug fixes today.

LEAKS

Joe Walker fixed a bad leak found by Jesse Ruderman:  if you closed a browser window with the developer toolbar open it would leak “everything”.  This was a MemShrink P1 bug.

Anton Kovalyov fixed a leak involving scratchpad.  This bug was also found by Jesse Ruderman.

Randell Jesup fixed some WebRTC leaks.

John Schoenick fixed a leak involving plugins.

Josh Aas fixed a leak in some networking code.

DMD

I wrote a more detailed blog post about DMD.  Here is the take-away message.

about:memory is MemShrink’s not-so-secret weapon when it comes to understanding Firefox’s memory consumption… and DMD is how we make about:memory better.

Lots of under-the-hood improvements have been made to DMD since I wrote that.  Users on Mac, Linux and B2G who aren’t afraid of doing their own builds should try it out.  Also, Ehsan Akhgari got it to build on Windows, though it’s not yet clear how well it works on that platform.  If anyone wants to try it out, please let me know how it goes.

Memory Reporters

Ben Turner made the workers memory reporter be able to handle workers that use ctypes.  This was important, especially on B2G, because each process can have one or two or more such workers — this is for Firefox chrome stuff, not web content — and we weren’t measuring them at all, and they can take multiple MiB each.

I fixed the orphan DOM node memory reporter.  The introduction of WebIDL had changed the layout of some paired JS/DOM objects, and such objects weren’t being reported.  (DMD discovered the unreported memory, and Boris Zbarsky helped me interpret what it meant.)  I see this accounting for multiple MiB of orphan nodes when using Gmail.

I added a memory reporter for the event listenener manager’s hash table.  It starts off small, but I’ve seen it go as high as 1.5 MiB after lots of browsing.  (DMD helped me identify this too.)

Kartikaya Gupta added a memory reporter for graphics textures on Android.

I added a memory reporter for any ctypes data that is hanging off JS objects.  I added it because one DMD profile on B2G showed non-trivial amounts of ctypes data, but that seems to have been a fluke and it rarely shows much memory now.  Oh well.

Miscellaneous

Andrew McCreight improved CC shutdown logging, which will make it easier to identify shutdown leaks.

Rail Aliiev and Kartikaya Gupta enabled a new NDK for Fennec builds on releng machines.  This might result in smaller binary sizes, which saves memory.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-2/+0)
  • P2: 116 (-10/+0)
  • P3: 102 (-4/+0)
  • Unprioritized: 18 (-0/+18)

The number of unprioritized bugs is high because we didn’t have a MemShrink meeting this week.  This was because Justin Lebar and Kyle Huey are in Berlin for the B2G work week.  We’ll have our next meeting two weeks from today.

DMD

I recently landed a new version of DMD on mozilla-central.  DMD is a tool helps us understand and thus reduce Firefox’s memory consumption.  But in order to understand DMD, you first have to understand about:memory.

about:memory

The MemShrink project started about 18 months ago, and it has been very successful in reducing Firefox’s memory consumption.  about:memory is MemShrink’s not-so-secret weapon when it comes to understanding Firefox’s memory consumption.

about:memory screenshot

about:memory has some wonderful characteristics:  it provides literally thousands of measurements;  it’s available in ordinary release builds;  and it’s trivial to run (just type “about:memory” in the address bar).  This means that non-expert users can easily provide developers with detailed measurements if they are having problems.

However, about:memory has two shortcomings.  First, it simply visualizes the data provided by the memory reporter infrastructure.  The coverage provided by this infrastructure is good, but there are still some gaps.  These gaps manifest primarily in the “heap-unclassified” number in about:memory, which represents all the heap allocations that the memory reporters didn’t cover.  Unfortunately, about:memory can provide zero insight into what is within “heap-unclassified”.

Second, it’s hard to verify.  There’s no obvious way to tell if the numbers it gives are accurate (with a few exceptions;  e.g. negative numbers are obviously wrong).  We have established good practices for writing memory reporters (measure sizes, don’t compute then;  traverse data structures rather than maintaining size counters) that prevent most errors, but it’s still quite easy to accidentally count a block of memory twice.

This is where DMD comes in.  It helps with both the heap-unclassified and double-counting problems.

DMD version 1

I wrote the first version of DMD over a year ago.  “DMD” is short for “dark matter detector”, because “heap-unclassified” memory is sometimes jokingly called “dark matter”.

DMD works by intercepting all calls to malloc/free/etc.  This lets DMD track extra information about every heap block, such as where it was allocated.  Furthermore, DMD has hooks into the memory reporters (via functions created with the NS_MEMORY_REPORTER_MALLOC_SIZEOF_FUN macro, for those who are interested) so it knows when any heap block is measured by a memory reporter.

With that in place, DMD knows how many times each heap block has been reported.  So, after running the memory reporters, we can up each block into one of the following three groups.

  • Blocks that haven’t been reported indicate gaps in existing memory reporters.  They contribute to “heap-unclassified”.
  • Blocks that have been reported once are good.
  • Blocks that have been reported twice or more indicate defects in existing memory reporters.  They cause some measurements in about:memory to be too high, and “heap-unclassified” to be too low.

DMD presents its results in a sorted fashion that lists the unreported and twice-reported cases that are more important first.

This first version of DMD was implemented as a Valgrind tool, and what’s more, it required a special, patched build of Valgrind to run.  This meant it was difficult to set up, ran very slowly, and could only be used on Linux and Mac.  Despite these shortcomings, it has proved itself extremely useful, helping us get “heap-unclassified” down greatly (on my Linux desktop machine it’s typically between 8 and 12%) and identifying several cases of double-counting.

DMD Version 2

Some time after I wrote the first version of DMD, I realized that all it needed to work was the ability to intercept malloc/free/etc. and to obtain stack traces.  Valgrind was overkill for these purposes — it’s capable of supporting much more invasive tools — and, furthermore, there were existing tools (e.g. trace-malloc) in the Mozilla codebase that did exactly these things.  In other words, it would be possible to build a new version of DMD that integrated directly into the browser.

Work on this new version stalled for a while, because the old one was working well enough.  But then B2G’s Operation Slim Fast started, and it quickly became obvious that “heap-unclassified” on B2G was typically much higher than it was on desktop.  This is primarily because B2G has lots of small processes, and so various unmeasured things that weren’t a big deal on desktop became much more significant on B2G.  And DMD version 1 doesn’t work on B2G devices.

So that provided the impetus to complete the new version.  Mike Hommey finished his new replace-malloc infrastructure recently, which helped greatly, and the new version of DMD landed on mozilla-central last week.  The old version will soon be retired.

DMD now works on Linux, Android, Mac, B2G, and is just shy of working on Windows (where Ehsan Akhgari has been making gradual progress).  It’s also much faster, partly because it avoids the overhead of Valgrind, and partly because it now uses sampling.  The sampling causes blocks smaller than a certain size (4 KiB by default, though you can change that) to be sampled, while blocks larger than that are measured accurately;  this sacrifices a moderate amount of accuracy for a large amount of speed.

Try it

about:memory is wonderful, and DMD is how we make about:memory better.  There are now detailed instructions on how to build DMD, run it, and interpret its output.  It’s quite easy now, requiring only minor changes to the usual build and run steps, and several people have already done it successfully.  Please try it out, and help us further improve about:memory.

MemShrink progress, week 73–74

B2G!  Fennec!  Social API!  They’re all happening, and memory consumption is a big deal for all of them.  Time for a MemShrink report.

B2G

Lots of B2G work is happening, unsurprisingly.  All of these changes have been backported to the Aurora channel.

Kyle Huey landed an important patch that merges system compartments together.  This avoids wasted space caused by having 100s of small compartments.  As Chris Jones said: “It’s a completely different phone with these patches.”  Unfortunately, the preference controlling this behaviour is currently turned off on B2G, because it’s causing some Marionette test failures.  Hopefully they’ll be dealt with soon.  Note that this preference won’t be turned on in desktop builds, and there’s a medium-term plan to avoid this wasted space in a less hacky fashion.

Marco Bonardo disabled Places in B2G, saving about 300 KiB in the main process.

Justin Lebar added code to trigger memory pressure events when lowmemkiller notifications occur.

Justin also tweaked things so that the garbage collector is more likely to run when screenshots are taken.

Jeff Muizelaar fixed a graphics bug that I don’t understand but apparently avoids allocating lots of memory on the B2G unlock screen.

Chris Jones ensured that remote content drop their buffers when they’re hidden.

I tweaked the size of the arena chunks used by XPT, which saved about 120 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I also shrunk the initial size of the SPS hash table, which saves 48 KiB per process on 64-bit builds and 24 KiB on 32-bit builds.

Memory Reporting

Lots of work happened on the memory reporting front.  This was inspired by  “heap-unclassified” typically being much higher on B2G than on desktop.  Desktop Firefox uses a single process and its memory consumption is usually measured in the 100s of MiBs or more.  In contrast, B2G uses one “main” process and then one process per running app, and the smaller ones are typically around 10 MiB.  As a result, the sundry small per-process things that don’t matter much for desktop loom larger on B2G.  (For the same reason, changes that reduce per-process memory by smallish amounts have outsize value on B2G.)  Most of the following changes have also been backported to the Aurora channel.

Nick Hurley added a “explicit/network/disk-cache” memory reporter.  It typically measures 100s of KiBs in desktop Firefox.  (It doesn’t get used on B2G.)

I added memory reporters “explicit/xpcom/component-manager” and “explicit/xpcom/category-manager”.  Together they measure about 280 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/script-namespace-manager”.  It measures just over 150 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/xpcom/effective-TLD-service”.  It measures about 128 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added some detail to the JS object memory reporters.  This gives slightly more insight into where this memory is going, but the extra memory measured as a result is usually pretty small.

I fixed the “explicit/atom-tables” memory reporter, which was erroneously always reporting 0 bytes.  It now measures anywhere from a few 100 KiBs to several MiBs of memory.  This was a frustrating bug to find.  DMD exists to check that memory reporters are measuring memory properly, and the reporter was doing its measurements just fine.  But DMD cannot check that the measured amounts are added correctly, and that’s what was going wrong here — a line that should have been this:

return n;

instead was this:

return 0;

In other words, the code I had written was doing the difficult part correctly, but botched the trivial part.  How annoying.

Social API

Felipe Gomes fixed a document leak that occurred when the social sidebar was hidden, and disappeared only when it was unhidden.

Felipe also added code to clear the previous profile when Social is toggled off, which prevents a leak.

The social API still has some significant unresolved memory consumption issues, which are a concern.

Fennec

Kartikaya Gupta (a.k.a. Kats) turned on tab expiration for Fennec.  What this means is that Fennec will now unload the contents of background tabs in certain circumstances — when memory is low, and in some cases when a tab hasn’t been viewed for over an hour — and then reload them when they are brought back into the foreground.  Read Kats’ blog post for more details.

(This change was prompted by Project 256meg, which aims to make Fennec usable on phones with only 256 MiB of RAM.  (Fennec currently requires 512 MiB, according to the official specs.)  Kats has just got to the point where Fennec can actually start on 256 MiB devices.  Although Fennec only uses a single process, there is obviously significant overlap between this goal and B2G’s memory reduction goals.)

An obvious follow-up idea is to add tab expiration to desktop Firefox.  But the bug tracking that idea has seen more heated discussion.  The reason is that tab expiration might cause data loss in some cases.  This is less of a problem on Fennec because (as far as I can tell) there is less of an expectation that in-flight data will be saved on mobile devices.  For example, having processes killed due to memory constraints on mobile is much more common than on desktop.  Still, I expect plenty more arguing before this issue is resolved one way or the other.  One option is to allow it on desktop but have it disabled by default.

Add-ons

Michel Gutierrez fixed a zombie compartment in Video DownloadHelper, which is the 2nd most popular add-on at AMO.   Version 4.9.11 has the fix.

We hit a notable milestone on the add-on front this fortnight:  on November 1st hit bug for tracking known leaks in add-ons — filed 16 months ago — had zero blockers.  In other words, we reached the point where no add-ons were known to leak!  Unfortunately this didn’t last long, and at the time of writing the tracking bug has three blocking bugs.  Still, it’s confirmation that what used to be our biggest memory consumption problem is well under control.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+5)
  • P2: 110 (-10/+7)
  • P3: 102 (-3/+8)
  • Unprioritized: 4 (-2/+4)

As always, the current bug lists can be found by following the links on the MemShrink wiki page.

MemShrink Progress, week 69–70

It’s been a bumper two weeks for MemShrink.

B2G

Gregor Wagner tweaked the JS garbage collection heuristics on B2G so that collections occur more frequently.  This prevents too much garbage from accruing, which is important on low-memory B2G devices.

Justin Lebar disabled sync on B2G, because it isn’t used.

Justin also implemented the ability to dump GC/CC logs when a particular signal is received, which is particularly useful for B2G.

B2G’s Operation Slim Fast is in full swing.  Please see the tracking bug for details about what is happening.  While there are a number of important fixes in the pipeline, memory consumption is still a critical issue for B2G simply because the devices don’t have much memory in the first place.  Any additional help from Mozilla developers will be greatly appreciated.  For example, if anyone can help identify unnecessary compartments, that would be helpful.

Memory Reporting

Boris Zbarsky improved the reporting coverage of style sheets.  We’ve seen this improved coverage account for over 1% of explicit on desktop and over 2% of explicit on B2G.

I fixed some double-counting of style sheets, and slightly improved the reporting coverage of style sheets and JS compartments.

Nathan Froyd added memory reporting for telemetry.

Justin Lebar fixed some over-reporting of the JS stack.  Justin also added to about:memory the ability to read JSON memory report data from the clipboard.

I renamed DMD as DMDV, in preparation for its in-progress, non-Valgrind replacement to take over the original name.

Testing With Valgrind

Gary Kwong has done some heroic work recently getting Valgrind tests passing on Tinderbox.  (Strictly speaking, the tests run Memcheck, which is the default Valgrind tool and is synonymous with Valgrind)  His blog post about it is well worth reading.

This is important for MemShrink because Memcheck can detect an important class of memory leaks — ones where all references to a block of memory are lost and so the block leaks forever.  But it’s also important for Firefox as a whole because Memcheck can detect a slew of other memory-related errors, such as writing unaddressable memory (e.g. buffer overruns) and using undefined values in dangerous ways.  These are the kinds of defects that cause crashes, security vulnerabilities, and other major problems.

The tests are currently hidden on TBPL because they only run on mozilla-central once a day, which is generally not enough for a test suite to be reliable enough to be worth unhiding.  Having said that, the ability to write suppressions for errors found with Memcheck complicates considerations;  see bug 800435 for further discussion.

Add-ons

Gregg Lind fixed things so that our Aurora/Beta users are certain to get an up-to-date, non-leaky version of the Test Pilot add-on.  (Version 1.1.2 was the leaky version;  version 1.2.0 or later has the fix.)  This was a longstanding MemShrink:P1 bug.

Erik Vold fixed a zombie compartment in Scriptish.  Version 0.1.8 has the fix.

Josep del Rio fixed a zombie compartment in Speed Dial.  Version 0.9.6.10 has the fix.

Miscellaneous

Boris Zbarsky fixed a bad zombie compartment leak relating to expandos.  This was a recent regression, and the fix is present on Aurora and Beta.  The problem was first reported in the comments of my MemShrink report from four weeks ago — many thanks to Alex for reporting it and preventing it from reaching an official Firefox release!

Bug Counts

Here are the current bug counts.

  • P1: 19 (-2/+5)
  • P2: 101 (-2/+6)
  • P3: 91 (-6/+0)
  • Unprioritized: 13 (-1/+11)

The unprioritized count is high because we spent the first part of today’s MemShrink meeting talking with Gavin Sharp, Jared Wein and Felipe Gomes about the upcoming social API, and then spent some additional time discussing B2G with David Clarke.  As a result we didn’t have time to triage the 23 new MemShrink bugs.  We’ve scheduled an out-of-band MemShrink meeting for next week to get on top of the bug list.

Visiting Mountain View

I’ll be in Mountain View all next week for the JS team work week.  I’ll be happy to meet up with anyone who wants to talk about memory consumption!

MemShrink progress, week 55-56

Memory Reporting

I changed things so that JavaScript memory consumption for web content is reported on a per-tab basis, as the following example shows.

│  ├────4,472,568 B (00.50%) -- top(http://www.mozilla.org/en-US/firefox/fx/, id=236)/active
│  │    ├──4,192,640 B (00.47%) -- window(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,979,112 B (00.22%) ++ js/compartment(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,607,216 B (00.18%) ++ layout
│  │    │  ├────352,520 B (00.04%) ── style-sheets
│  │    │  ├────253,312 B (00.03%) ++ dom
│  │    │  └────────480 B (00.00%) ── property-tables

All the major components of each tab’s memory consumption — JS, layout, style sheets, and DOM — are now reported on a per-tab basis.  This is great, because it’s something that people have been requesting for  years.  However, the presentation is still intimidating.  Consider the following excerpt from about:memory.

├───18,288,696 B (16.43%) -- window-objects
│   ├───6,263,208 B (05.62%) ++ top(about:memory?verbose, id=6)/active
│   ├───5,090,352 B (04.57%) ++ top(chrome://browser/content/browser.xul, id=1)/active
│   ├───3,422,280 B (03.07%) ++ top(http://www.mozilla.org/en-US/firefox/fx/, id=17)/active/window(http://www.mozilla.org/en-US/firefox/fx/)
│   ├───3,178,432 B (02.85%) ++ top(chrome://global/content/console.xul, id=21)/active
│   └─────334,424 B (00.30%) ++ top(resource://gre-resources/hiddenWindow.html, id=3)/active/window(resource://gre-resources/hiddenWindow.html)

Among those top(...) entries are two browser tabs (one for www.mozilla.org, one for about:memory), two browser chrome windows (browser.xul is the main browser window, and console.xul is the error console window), and the hidden window which is always present.  Firefox developers can easily understand this, and technically-inclined users may be able to guess what’s going on, but the average user will struggle.

From a user’s point of view, the visible things they have control over (i.e. windows and tabs) should be better distinguished from everything else, and be shown with nicer names than “browser.xul”.  But the architecture of Firefox’s internals doesn’t make this easy.  I’ve made one not-very-successful attempt to improve this, and I’d love to hear suggestions from other Firefox devs on how to present this information better.

(The step after that one is to present some kind of “tab manager” — a massively stripped-down variant of about:memory that ordinary users can understand.  This would also require localization for all the languages that Firefox supports.  One step at a time!)

Finally, another memory reporting improvement happened:  Andrew McCreight and I added measurement of orphan DOM nodes.  These are DOM nodes that have been discarded by a page but are still accessible from JavaScript objects.   They were the biggest single remaining contributor to “heap-unclassified”.  For example, when I hit the “update” button on about:memory I get 5MB of them (don’t worry, they go away when the garbage collector runs).  Also, badly coded sites can create large numbers of them.  So it’s a good thing to be able to measure, for both users and web developers.

Miscellaneous

Mihai Sucan fixed a bad leak involving pages that use window.console.  Multiple people reported that this caused Firefox’s memory consumption to balloon to multiple GB.  This leak was introduced during the Firefox 15 development cycle, and the fix has been backported to the Aurora channel, which is where Firefox 15 currently resides.  One notable thing about this fix is that the problem was quite easy to diagnose because about:memory clearly indicated that the ConsoleAPI.js compartment was responsible for most of the memory consumption.  Without the fine-grained memory reporting that was enabled by compartment-per-global, that memory would have been reported in a generic “System Principal” compartment which would have made diagnosis much harder.

Wladimir Palant fixed an issue in AdBlock Plus that causes its memory consumption to slowly grow when viewing sites that repeatedly reset the src property of images, such as Mibbit.  (The fine-grained memory reporting enabled by compartment-per-global also made this diagnosis much easier.)  This fix is in Adblock Plus 2.1.

Alexandre Poirot fixed a minor leak in the Add-on SDK.

I wrote about the difficulties of cross-browser memory comparisons.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+0)
  • P2: 90 (-3/+6)
  • P3: 102 (-6/+2)
  • Unprioritized: 2 (-0/+0)

A net reduction of seven bugs!

Update on The big ticket Items

In January, I described six “big ticket items” that needed tackling to improve Firefox’s memory consumption.  Let’s look at how they’ve progressed since then.

#6: Better Script Handling

This had two parts.  The first part was to only generate bytecode for a function once it is run.  I’ve made a good chunk of progress towards and have it working on very simple examples.  This has required a great deal of refactoring of the SpiderMonkey front-end.  (The front-end is a hairy piece of code, so this is virtuous in and of itself.)  There are still a couple of other bugs blocking it from further progress.  I intend to return to continue on this once they are done.

The second part was to share immutable parts of scripts between web pages.  No progress has been made on this.

So, overall, this item is about 25% done, and progress is still ticking along slowly.

#5: Better Memory Reporting

This one also had two parts.  The first part was to reduce about:memory’s “heap-unclassified” number (a.k.a “dark matter”) to typically 10%.  People such as Nathan Froyd and I have made good progress on this.  It’s now occasionally below 10% (my current session shows 7.5%) but often is around 15%.  That’s much better than the 20–25% we typically were getting at the start of the year, and it’s low enough that people rarely complain about it now.  And we’re very much into the long tail now, so it’ll be hard to improve things much further.  For the most part I don’t think we need to.

The second part was to report memory in a per-tab fashion.  As mentioned above, I just completed this, although the presentation is not as user-friendly as it could be.

The per-tab reporting was enabled by compartment-per-global.  Compartment-per-global also gives us much more insight into the memory consumption of Firefox’s own JavaScript code, and even some idea about add-on memory consumption, which was (to me) an unexpected bonus.  Indeed, its usefulness was demonstrated by the two leak fixes mentioned above.

So, overall, this item is about 80% done.  Although there is still some work to be done, it has progressed enough that I will remove it from the “big ticket items” list.

#4: Better Memory Consumption Tracking

We can cross this item off the list, thanks to areweslimyet.com.  In January I also mentioned the idea of using telemetry for this, but we’ve found that telemetry results are so noisy that little (with rare exceptions) can be gleaned from it, so there’s not really anything to be done on that front.

#3: Compacting Generational GC

This is one of the two key SpiderMonkey performance features under development (the other being IonMonkey), and so people such as Terrence Cole, Steve Fink, Brian Hackett, and Bill McCloskey and actively working on it.  Unfortunately, it’s a huge undertaking that requires rewriting many of SpiderMonkey’s internal and external APIs.  If I had to guess I’d say it’s 25% complete, but that really is just a wild guess.  I’ll be pleased if it is finished before the end of the year.

#2: Better Foreground Tab Image Handling

This is the big ticket item for which the least progress has been made. There’s a single bug that is blocking all the follow-on work.  Both Timothy Nikkel and Jet Villegas have spent time working on it, but it has been a difficult nut to crack.  Justin Lebar had a recent suggestion for narrowing its scope slightly.  I hope it’s a good suggestion, because I’m out of ideas on this one.

#1: Better Detection and Notification of Leaky Add-ons

This is the most satisfying item of the lot.  I expected to make only grudging, piecemeal progress, but Kyle Huey’s fix to prevent zombie compartments prevents the vast majority of add-on leaks, enough to declare victory and cross this item off the list.  I’ll be writing more about this next week.

Summary

Three items (#1, #4, #5) have progressed enough that they can be removed from the list.  Only one of these directly improved Firefox’s memory consumption;  the other two were about better measuring and monitoring of memory consumption.

The remaining items are about reducing memory consumption.  Two of them (#3, #6) have a long way to go, but progress has been made and is ongoing.  One item (#2) has made little progress and has stalled.

The New big ticket Items

Here’s my updated list of the most important things.

#5: Better Script Handling

As mentioned above, this has two parts: lazy bytecode generation, and the  sharing of immutable parts of scripts.  I intend to continue working on the first part.  Once that’s done, it may turn out that the second part isn’t necessary;  we’ll see.

#4: Regain compartment-per-global losses

The measurements on areweslimyet.com have been creeping up for the past two months.  The major cause is compartment-per-global, which has lots of benefits (as mentioned above) but also increases memory consumption by a small-to-moderate amount across the board.  This is because there are many more compartments than there used to be, and there’s a certain amount of overhead and waste in each compartment.

Generational GC will help, but there are some other things that can be done more quickly that will help.  See the tracking bug for one example.

#3: Boot2Gecko

Boot2Gecko (now officially known as “Firefox OS”) is going to be running on low-end phones without much physical memory, so that’ll require care.  The first step is to get about:memory working on B2G.  The next step is to find a way to copy the data from about:memory somewhere else, because B2G doesn’t have cut and paste!  Beyond that, it’s unclear;  but once about:memory is working it’s very likely it will identify some sub-optimal B2G-specific behaviour.

#2: Compacting Generational GC

Let me quote myself to explain the memory consumption benefits of compacting, generational GC.

Virtual memory consumption drops in two ways.  First, the compaction minimizes waste due to fragmentation.  Second, the heap grows more slowly.  Physical memory consumption drops for the same two reasons.

The performance benefits are also significant.

Performance improves for three reasons.  First, paging is reduced because of the generational behaviour:  much of the JS engine activity occurs in the nursery, which is small;  in other words, the memory activity is concentrated within a smaller part of the working set.  Second, paging is further reduced because of the compaction: this reduces fragmentation within pages in the tenured heap, reducing the total working set size.  Third, the tenured heap grows more slowly because of the generational behaviour… which means that structure traversals done by the garbage collector (during full-heap collections) and cycle collector are faster.

Important stuff.

#1: Better Foreground Tab Image Handling

On image-heavy pages, Firefox uses much more memory than other browsers.  This includes things like Facebook image slideshows.  One commenter said the following.

On behalf of Facebook Inc, let me just say that we are prepared to give over one thousand free “pokes” to anyone who fixes this bug or alternatively https://bugzilla.mozilla.org/show_bug.cgi?id=683290. I will also gladly show you Mark Zuckerberg’s desk and give you free dinner and cookies as part of a complementary tour of Facebook HQ in Menlo Park.

I have no idea if this person is really from Facebook, but it’s certainly suggestive.  Also, I’ve heard from B2G people that they are having to work around this problem in their Gallery app.

To repeat myself from week 32:  there are three MemShrink:P1 bugs relating to this:  one about not decoding all images immediately, one about discarding non-visible decoded images after some time, and one about some infrastructure work that is required for the first two.  (See this discussion on the dev-platform mailing list for more details about this topic.)

Summary

There are three repeat items, and two new items, but there are fewer items than last time;  this reflects the fact that we are in a better position than we were in January.  If you think I’ve overestimated or underestimated the importance of any of these, or omitted anything, I’d be interested to hear.

One final thing:  last time, several people suggested “better reporting of add-on memory consumption”.  If I knew of any ideas on how to do this better than about:memory currently does, I’d gladly put it on the list.  But I don’t, and there doesn’t seem to be much point to put an item on the list that we don’t know how to implement.

Vacation

I’m taking the next two weeks as vacation, so there won’t be a MemShrink report on July 24.  I’ll be back with the next MemShrink round-up on August 8th.  See you then.

MemShrink progress, week 53-54

Week 53-54?  We’ve been going for over a year!  To celebrate MemShrink’s 1st birthday, I wrote about the major achievements we’ve made in the past year.

Memory Reporting

I overhauled the cross-cutting “Other Measurements” shown for JavaScript in about:memory.  It’s now in two parts.  The first part is a functional breakdown of the memory consumption of the main JavaScript runtime, i.e. all JavaScript memory usage outside of web worker threads.  Here’s an example.

194,179,064 B (100.0%) -- js-main-runtime
├──159,064,360 B (81.92%) -- compartments
│  ├───73,850,880 B (38.03%) -- gc-heap
│  │   ├──22,014,080 B (11.34%) -- objects
│  │   │  ├──14,119,616 B (07.27%) ── non-function
│  │   │  └───7,894,464 B (04.07%) ── function
│  │   ├──19,553,168 B (10.07%) ── unused-gc-things
│  │   ├──14,014,208 B (07.22%) -- shapes
│  │   │  ├───6,747,240 B (03.47%) ── tree
│  │   │  ├───4,926,000 B (02.54%) ── dict
│  │   │  └───2,340,968 B (01.21%) ── base
│  │   ├──10,204,960 B (05.26%) ── strings
│  │   ├───5,338,816 B (02.75%) ── scripts
│  │   ├───1,493,472 B (00.77%) ── type-objects
│  │   ├───1,229,456 B (00.63%) ── arena-admin
│  │   └───────2,720 B (00.00%) ── sundries
│  ├───29,459,680 B (15.17%) ── analysis-temporary
│  ├───14,484,704 B (07.46%) ── script-data
│  ├───13,062,744 B (06.73%) ── string-chars
│  ├────9,300,704 B (04.79%) -- shapes-extra
│  │    ├──3,325,952 B (01.71%) ── compartment-tables
│  │    ├──2,845,984 B (01.47%) ── tree-tables
│  │    ├──2,125,056 B (01.09%) ── dict-tables
│  │    └──1,003,712 B (00.52%) ── tree-shape-kids
│  ├────7,688,912 B (03.96%) -- type-inference
│  │    ├──5,750,576 B (02.96%) ── script-main
│  │    ├──1,368,560 B (00.70%) ── object-main
│  │    └────569,776 B (00.29%) ── tables
│  ├────6,232,272 B (03.21%) -- objects
│  │    ├──4,406,720 B (02.27%) ── slots
│  │    ├──1,726,640 B (00.89%) ── elements
│  │    └─────98,912 B (00.05%) ── misc
│  ├────4,256,768 B (02.19%) ── cross-compartment-wrappers
│  └──────727,696 B (00.37%) ── mjit-data
├───19,472,384 B (10.03%) -- gc-heap
│   ├──18,006,016 B (09.27%) ── unused-arenas
│   ├───1,458,176 B (00.75%) ── chunk-admin
│   ├───────8,192 B (00.00%) ── decommitted-arenas
│   └───────────0 B (00.00%) ── unused-chunks
└───15,642,320 B (08.06%) ── runtime

The contents of the “js-main-runtime/compartments” sub-tree is the sum of all the individual compartment entries in the “explicit/js” sub-tree, and the “js-main-runtime” total is the same as the “explicit/js” total.  This new view will become more important when per-tab memory reporting is implemented which will cause each tab’s JavaScript memory consumption to be reported separately.

The second part is designed to show the utilization of the garbage-collected heap.  The new measurements and presentation are more detailed and easier to understand than the old version.  Here’s an example.

93,315,072 B (100.0%) -- js-main-runtime-gc-heap-committed
├──55,755,888 B (59.75%) -- used
│  ├──53,068,256 B (56.87%) ── gc-things
│  ├───1,458,176 B (01.56%) ── chunk-admin
│  └───1,229,456 B (01.32%) ── arena-admin
└──37,559,184 B (40.25%) -- unused
   ├──19,553,168 B (20.95%) ── gc-things
   ├──18,006,016 B (19.30%) ── arenas
   └───────────0 B (00.00%) ── chunks

One thing this makes clear is that our “unused” fraction is often very high.  We’ll need a compacting generation garbage collector to really fix that.  This is being actively worked on, but it’s a large task.

Nathan Froyd had a productive couple of weeks.

These four reporters don’t cover a great deal of memory, but that’s inevitable these days;  all the low-handing “heap-unclassified” fruit is gone, and the remaining improvements will necessarily be small.  We’re gradually closing in on the target of 10%, though I’m not sure we’ll ever get there.

Nils Maier added some more identifying information to JavaScript sandboxes, which will help identify them in about:memory.

Bas Schouten added a memory reporter for VRAM used by Azure.

Fixed Leaks

Robert O’Callahan fixed a leak involving canvas, which I won’t pretend to understand.  This also fixed bug  757749.

Cervantes Yu fixed a B2G memory leak that would leak about 1MB of memory per day.  This was found by Valgrind, under the operation of Julian Seward.

Quote of the Week

A Hacker News thread contained the following comment.

Mozilla needs to focus on making its browser consume less memory and CPU when idle. These are the things that are killing its marketshare and leading users to switch to alternatives. Every time I see initiatives like the password account manager, iOS apps that use web view, a browser based editor and what not I just scratch my head because Firefox marketshare is plummeting and the old refrain “just wait until the next version” was tired since 3.0. Instead the general response is to get defensive and list new features. Yeah and are those new features stopping the user drain? Nope. Firefox is a sluggish memory sucking heap of software that drains resources even when it isn’t being used.

I love and cherish Firefox as a product and a brand, because you have achieved a lot in your past. But now you are failing and it is depressing to see this happen.

That’s not the quote of the week, BTW;  I’ve heard variations of that many times before, and while it had some truth to it 18 months ago, that piece of conventional wisdom is now well past its use-by date.  Pleasingly, there was quite a bit of push-back in response.  My favourite comment, which is the quote of the week, was the following.

Firefox is to Ford what Chrome is to Toyota. Even if Firefox is as reliable and efficient as Chrome these days, some people still see it as being the gas-guzzling unreliable Ford vehicle of yesteryear.

A bad reputation is a difficult thing to lose.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-2/+0)
  • P2: 87 (-10/+9)
  • P3: 106 (-3/+3)
  • Unprioritized: 2 (-2/+2)

A net change of -2 P1 bugs, and -1 P2 bugs.  Not bad!

MemShrink’s 1st Birthday

A year ago today, the first meeting of the MemShrink project took place.  The project had been announced almost three months earlier, but until that first meeting it was basically just me working on it and there wasn’t much momentum.  So I now think of those three months as something of a gestation period, and the first meeting as the true birth of the project.

So, happy birthday, MemShrink!  I’ll take this opportunity to go over some of the accomplishments we’ve made in the past year.

Big Win #1: Better Javascript Heap Management

There have been two huge MemShrink improvements that stand out in my mind.  The first one happened very early on, and it consisted of two related changes, both implemented by Gregor Wagner.

In Firefox 4, the implementation of the JavaScript heap was totally overhauled.  In particular, it was segregated into compartments, each of which held (roughly speaking) the memory for a single domain.  System compartments, i.e. those for JavaScript code that implements parts of Firefox (such as the UI), tend to be long-lived.  In contrast, user compartments, i.e. those for JavaScript code in web pages, tend to be short-lived.  But memory from both system and user compartments was being co-located in 1MB chunks.  What tended to happen is that if you browsed for a long time and then closed many tabs, you’d end up with lots of 1MB chunks that were kept alive by a small amount of memory from system compartments.  This is a form of fragmentation, and it prevented Firefox from releasing lots of memory back to the operating system that it should have been able to.

Fortunately, Gregor came up with a simple fix:  don’t allow system and user compartments to share the same 1MB chunks.  To see just how effective this was, consider this visualization of the JavaScript heap after closing many tabs, without Gregor’s change.

badly fragmented JavaScript heap

Each horizontal line is a 1MB chunk, and each square is a 4KB arena.  White squares are unused arenas, and coloured squares are in use.  Arenas belong to a particular compartment all have the same colour;  there are three compartments alive, all of which are system compartments.  You can see that many 1MB chunks are kept alive by a small number of arenas.

After Gregor’s change, in this scenario the heap ended up looking like this.

non-fragmented JavaScript heap

Much better.

Gregor’s second change related to garbage collection scheduling.  In Firefox 4 the garbage collection heuristics were poorly tuned, and the garbage collector simply didn’t run frequently enough.  Many people who left Firefox open for a while came back to find that JavaScript memory consumption had ballooned, and the machine was unusable due to paging, for seconds or minutes, until the garbage collector kicked in and paging finished.  The fix was simple:  run the garbage collector occasionally, using a timer.

These two fixes of Gregor were the main reason why the first item in the Firefox 7 release notes was “drastically improved memory handling for certain use cases”.  More specifically, Firefox 7 used less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.

Big Win #2: Fewer add-on Leaks

The second huge MemShrink improvement is much more recent.  In fact it hasn’t even made it into a released version of Firefox yet.  And the path towards it was much more circuitous.

I first revamped about:memory in May of last year, and those changes made it into Firefox 6. This started giving us insight into Firefox’s memory consumption, but it wasn’t until I broke up the reporting of JS memory consumption on a per-compartment basis that it really started to make a difference.  (This change made it into Firefox 7, another reason why that release was so good from a memory consumption point of view.)

Per-compartment reporters made some blatant inefficiencies obvious, which were quickly fixed.  More importantly, people quickly noticed that sometimes compartments were present for sites that had been closed long ago.  This is a kind of memory leak that became known as a zombie compartment, and hunting them became a popular enough sport that  more than 100 zombie compartment bug reports have been filed.

Some of the zombie compartments were due to defects in Firefox itself, and these were generally fixed fairly quickly.  However, it soon became clear that the majority of them are due to add-ons.  It’s quite easy to unintentionally create zombie compartments in add-ons.  In the worst case, add-ons could leak the compartment of every single site visited.

This led to some lively discussion about how best to handle these leaks, because they are defects in third-party code that is largely out of Mozilla’s control, and yet they make Firefox look bad.  Fortunately, this discussion became mostly moot when Kyle Huey took advantage of an 8,000 mile trip from home to implement a crazy idea he’d had six months earlier –  when a tab is closed, simply cut the references from other compartments that would keep the compartment of the tab’s page alive.  This turns out to work tremendously well and prevents most of the cases where add-ons can create zombie compartments.  This change is scheduled to ship in Firefox 15, which is due for release in late August, and should more or less fix what I six months ago ranked as the single worst problem affecting Firefox’s memory consumption.

The Key Tool: Memory Reporters and About:memory

I mentioned about:memory above.  It’s worth discussing in more detail, because it’s the single most important tool we’ve created, and has driven a lot of the MemShrink improvements.

Let’s look at about:memory as it was in Firefox 6, which was released about 10 months ago.about:memory screenshot from Firefox 6

Even these early versions had several key properties.

  • about:memory doesn’t need a special build to run;  it works even in released versions.
  • It is trivially easy to use — you just type “about:memory” in the address bar.
  • Because it’s all text you can cut and paste the output, and unlike many web pages it is carefully constructed so that it reproduces beautifully when you do so.  No screenshots are necessary.

These properties massively expand the number of people who are able to detect and report problems relating to memory consumption.  Any Firefox user can potentially do it;  if the user doesn’t already know how, they can be taught in 10 seconds.

Having said that, this old version is almost laughably primitive compared to what we have now.  First, let’s consider the state of the memory reporting infrastructure, which about:memory completely relies on, as it was at that time.

  • There were a fixed number of measurements, which meant there were no individual measurements for variable-numbered entities such as compartments, window objects, or SQLite connections.  Such measurements were only possible once I added support for memory multi-reporters.  We now have dozens of measurements per variable-numbered entity, and thousands of measurements in total.
  • Even once per-compartment reporters were added, we still didn’t have much insight into the memory consumption of the JavaScript code used by Firefox itself (and its add-ons).  But compartment-per-global landed recently and gave us much finer-grained measurements, which (a) gives us some idea of how much memory is being used by JavaScript-based add-ons, and (b) gets us very close to being able to report memory consumption for each tab.  Users have been requesting both of these things for years.
  • Early memory reporters had lots of defects that caused them to report incorrect numbers, but these were obvious only when the numbers were ridiculously large or negative (both of which happened on occasion).  The first of two major improvements in correctness came about as we learned how to structure memory reporters to avoid common defects.  In particular, it’s best to write reporters that are traversal-based rather than counter-based, and to measure heap blocks with malloc_usable_size rather than computing the sizes.  Reporters that use malloc_usable_size also measure slop bytes, which are caused by the heap allocator rounding up allocation requests and can account for 5–10% of the heap.  Early reporters didn’t have these characteristics, but most of them have been converted now.
  • Reporters with these characteristics are doubly-virtuous, because they also integrate well with DMD, a tool I wrote that dynamically checks memory reports.  DMD provided the second major improvement in memory reporter correctness, because it identifies heap blocks that are counted twice, which is a mistake that’s easy to make.
  • DMD also identifies unreported heap blocks, and was crucial in getting the “heap-unclassified” number down.  In the example above it was 68.82%(!), whereas now it’s typically around 15%, which is low enough that people have mostly stopped complaining about it.  Prior to DMD, identifying this “dark matter” was much harder and less systematic.
  • The only tree shown in the above example is “explicit”.  On Linux, Justin Lebar added code to summarize the information in the /proc/<pid>/smaps file in “size”, “rss”, “pss” and “swap” trees.
  • The reporters back then could only report byte measurements.  They can now report unitless counts and percentages, both of which are useful.

Now consider the about:memory page itself, as it was back then.

  • Insignificant measurements were simply omitted, and you had to reload in verbose mode to see them (and the measurements would be re-taken when that happened, potentially losing any interesting measurements).  Now it is possible to expand and collapse arbitrary parts of the trees on the fly.
  • The “Other Measurements” section contained a flat list of measurements.  Trees are now possible, which makes that section more flexible and easier to understand.
  • about:memory’s has been carefully streamlined to consume as little memory as possible, so that it doesn’t perturb what it’s measuring.  This is really important now that we report so many measurements.
  • about:memory now has a companion page, about:compartments, that just lists compartments (and ghost windows).  about:compartments shares a lot of code with about:memory, and it is better than about:memory for certain tasks such as spotting zombie compartments.

All this stuff is important because you make what you measure.  Just about every time we’ve started measuring some new thing, we’ve found and fixed problems as a result.

Smaller Things

The following smaller accomplishments were also made in the past year.

  • John Schoenick used the MozMill endurance tests to build areweslimyet.com, which helps identify regressions in memory consumption.
  • I removed a lot of unnecessary memory waste caused by “clownshoes” allocations.
  • Olli Pettay wrote about:cc and Jan Honza Odvarko wrote about:ccdump.  These closely related tools have been used to find various cycle collector leaks.
  • I wrote 41 progress reports.  I had/have two goals with these:  to keep interested people informed of what it happening, and to provide evidence that Mozilla actually does care about memory consumption in Firefox.

This is only the tip of the iceberg, though.  As I am writing this, there have been 314 MemShrink-tagged bugs marked FIXED in Bugzilla.  In comparison, there are only 221 MemShrink-tagged bugs currently open.  Breaking them down by priority we get the following.

  • P1: 52 fixed, 25 open
  • P2: 135 fixed, 88 open
  • P3: 53 fixed, 106 open
  • Unprioritized: 74 fixed, 2 open

Over 75 people were responsible for these fixes.  I give many thanks to all those people, plus everybody who contributed in any other way to fixes, such as bug reporters and patch reviewers.  And I’d like to give extra thanks to the MemShrink regulars:  Justin Lebar, Kyle Huey, Andrew McCreight, Johnny Stenback, and Jet Villegas.

What’s Next?

There’s no real secret to MemShrink.  So far it’s basically been a long, steady grind, gradually improving tools, fixing leaks, slimming down data structures, and responding to user’s problems.  There are no plans to change that, and we’ll continue our recently-lowered pace of fortnightly meetings for the foreseeable future.  And if we’re lucky, I might be writing a report on MemShrink’s second birthday a year from now!