Category Archives: about:memory

MemShrink progress, week 109–112

There’s been a lot of focus on B2G memory consumption in the past four weeks.  Indeed, of the 38 MemShrink bugs fixed in that time, a clear majority of them relate in some way to B2G.

In particular, Justin Lebar, Kyle Huey and Andrew McCreight have done a ton of important work tracking down leaks in both Gecko and Gaia.  Many of these have been reported by B2G partner companies doing stress testing such as opening and closing apps 100s or 1000s of times over long period.  Some examples (including three MemShrink P1s) are here, here, here, here, here, here, here and here.  There are still some P1s remaining (e.g. here, here, here).  This work is painstaking and requires lots of futzing around with low-level tools such as the GC/CC logs, unfortunately.

Relatedly, Justin modified the JS memory reporter to report “notable” strings, which includes smallish strings that are duplicated many times, a case that has occurred on B2G a couple of times.  Justin also moved some of the “heap-*” reports that previously lived in about:memory’s “Other measurements” section into the “explicit” tree.  This makes “explicit” closer to “resident” a lot of the time, which is a useful property.

Finally, Luke Wagner greatly reduced the peak memory usage seen during parsing large asm.js examples.  For the Unreal demo, this reduced the peak from 881MB to 6MB, and reduced start-up time by 1.5 seconds!  Luke also slightly reduced the size of JSScript, which is one of the very common structures on the JS GC heap, thus reducing pressure on the GC heap, which is always a good thing.

 

MemShrink progress, week 105–108

This is the first of the every-four-weeks MemShrink reports that I’m now doing.  The 21 bugs fixed in the past four weeks include 11 leak fixes, which is great, but I won’t bother describing them individually.  Especially when I have several other particularly impressive fixes to describe…

Image Handling

Back in March I described how Timothy Nikkel had greatly improved Firefox’s handling of image-heavy pages.  Unfortunately, the fix had to be disabled in Firefox 22 and Firefox 23 because it caused jerky scrolling on pages with lots of small images, such as Pinterest.

Happily, Timothy has now fixed those problems, and so his previous change has been re-enabled in Firefox 24.  This takes a big chunk out of the #1 item on the MemShrink big ticket items list.  Fantastic news!

Lazy Bytecode Generation

Brian Hackett finished implementing lazy bytecode generation.  This change means that JavaScript functions don’t have bytecode generated for them until they run.  Because lots of websites use libraries like jQuery, in practice a lot of JS functions are never run, and we’ve found this can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.

Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

The completion of this finished off item #5 on the old Memshrink big ticket items list.  Great stuff.  This will be in Firefox 24.

Add-on Memory Reporting

Nils Maier implemented add-on memory reporting in about:memory.  Here’s some example output from my current session.

├───33,345,136 B (05.08%) -- add-ons
│   ├──18,818,336 B (02.87%) ++ {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
│   ├──11,830,424 B (01.80%) ++ {59c81df5-4b7a-477b-912d-4e0fdf64e5f2}
│   └───2,696,376 B (00.41%) ++ treestyletab@piro.sakura.ne.jp/js-non-window/zones/zone(0x7fbd7bf53800)

It’s obvious that Tree Style Tabs is taking up 2.7 MB.  What about the other two entries?  It’s not immediately obvious, but if I look in about:support at the “extensions” section I can see that they are AdBlock Plus and ChatZilla.

If you’re wondering why those add-ons are reported as hex strings, it’s due to a combination of the packaging of each individual add-on, and the fact that the memory reporting code is C++ and the add-on identification code is JS and there aren’t yet good APIs to communicate between the two.  (Yes, it’s not ideal and should be improved, but it’s a good start.)  Also, not all add-on memory is reported, just that in JS compartments;  old-style XUL add-ons in particular can have their memory consumption under-reported.

Despite the shortcomings, this is a big deal.  Users have been asking for this information for years, and we’ve finally got it.  (Admittedly, the fact that we’ve tamed add-on leaks makes it less important than it used to be, but it’s still cool.)  This will also be in Firefox 24.

b2g

Gregor Wagner has landed a nice collection of patches to help the Twitter and Notes+ apps on B2G.

While on the topic of B2G, in today’s MemShrink meeting we discussed the ongoing problem of slow memory leaks in the main B2G process.  Such leaks can cause the phone to crash or become flaky after its been running for hours or days or weeks, and they’re really painful to reproduce and diagnose.  Our partners are finding these leaks when doing multi-hour stress tests as part of their QA processes.  In contrast, Mozilla doesn’t really have any such testing, and as a result we are reacting, flat-footed, to external reports, rather than catching them early ourselves.  This is a big problem because users will rightly expect to have their phones run for weeks (or even months) without rebooting.

Those of us present at the meeting weren’t quite sure how we can improve our QA situation to look for these leaks.  I’d be interested to hear any suggestions.  Thanks!

Recent about:memory improvements

The landing of patches from two recent bugs has substantially changed the look of about:memory.  When you load the page all see now is the following.

about:memory screenshot

Measurements aren’t taken away;  you have to click on the “Measure” button for that to happen.  Also, adding ?verbose to the URL no longer has an effect.  If you want verbose output, you need to click on the “verbose” checkbox.

The motivation for this change was that about:memory’s functionality has expanded greatly over the past two years, and cramming more functionality into the existing UI was a bad idea.  There are numerous advantages to the new UI.

  • No need to control behaviour via the URL, which is admittedly an odd way to do it.
  • You can switch between verbose and non-verbose modes without having to reload the page.
  • All the buttons are at the top of the page instead of the bottom, so you don’t have to scroll down.
  • You can trigger a GC/CC/minimize memory usage event without having do a memory measurement.
  • When you save reports to file, it’s clearer that a new measurement will be taken, rather than saving any measurements that are currently displayed on the screen.
  • The buttons are grouped by function, which makes them easier to understand.

There’s also the “Load and diff…” button, which lets you easily find the difference between two saved memory report files.  Here’s some example output taken after closing a tab containing a Google search result.

about:memory diff output

You can see that all the measurements are negative.  The [-] markers on leaf nodes in the tree indicate that they are present in the first file but not in the second file.  Corresponding [+] markers are used for measurements present in the second file but not the first.

MemShrink progress, week 95–96

B2G Fixes

Kyle Huey made image.src='' discard the image immediately even if the image is not in the document.  This provides a way for image memory to be discarded immediately, which is important for some B2G apps such as the Gallery app.  This was a MemShrink:P1 bug.

Justin Lebar fixed a bad leak in AudioChannelAgent.

Mike Habicher fixed an image-related leak that was manifesting on B2G.

Other Fixes

I exposed the existing JSON memory report dumping functionality in about:memory.  As a result, gzipped JSON is now the preferred format for attaching memory report data to bugs.  This was a MemShrink:P1 bug.

Honza Bambas overhauled the DOM storage code.  Apparently this might reduce memory consumption, but I fully admit to not knowing the details.

Nicolas Silva fixed a leak in layers code relating to OMTC.

I removed the MEMORY_EXPLICIT telemetry measurement.  I didn’t want to do this, but it’s been causing hangs for some users, and those hangs are hard to avoid due to the complexity of reporting memory consumption from web workers.  Furthermore, in practice the measurement was sufficiently noisy that we’ve never gotten anything useful from it.

Help Needed

Mozilla code uses a mixture of fallible and infallible allocations.  In theory, any allocation that could allocate a large amount of memory (e.g. a few hundred KiB or more) should use a fallible allocator.  We’ve seen recently some cases where large allocations were being made infallibly, which led to OOM crashes.  Bug 862592 proposes adding an assertion to the infallible allocators that the request size isn’t too large.  This is an easy bug to get started with, if anyone is interested.  Details are in the bug.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-4/+4)
  • P2: 145 (-0/+7)
  • P3: 131 (-2/+4)
  • Unprioritized: 5 (-1/+5)

Interregnum

I will be on vacation during May.  As a result, there will be no MemShrink reports for the next three fortnights.  (Although I’ll be away for just over four weeks, that period covers three MemShrink meetings.)  I’ll be back with the next report on June 11.  See you then!

Gzipped JSON is now the preferred format for attaching memory report data to bugs

Old: Text

about:memory is our main memory profiling tool.  You can copy and paste the memory reports shown by about:memory and they will reproduce nicely — so long as the displaying text is a fixed-width font — as the following example shows.

Main Process

Explicit Allocations
536.22 MB (100.0%) -- explicit
├──217.45 MB (40.55%) -- window-objects
│  ├───91.55 MB (17.07%) -- top(https://mail.google.com/mail/u/0/#label/A-moz%2Fbugz/13d41da77907a707, id=16)/active
│  │   ├──61.18 MB (11.41%) -- window(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  ├──56.29 MB (10.50%) -- js
│  │   │  │  ├──56.11 MB (10.46%) -- compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1)
│  │   │  │  │  ├──35.23 MB (06.57%) -- gc-heap
│  │   │  │  │  │  ├──18.17 MB (03.39%) -- objects
│  │   │  │  │  │  │  ├───9.88 MB (01.84%) ── dense-array
│  │   │  │  │  │  │  ├───6.08 MB (01.13%) ── ordinary
│  │   │  │  │  │  │  └───2.21 MB (00.41%) ++ (2 tiny)
│  │   │  │  │  │  ├──10.71 MB (02.00%) ++ (6 tiny)
│  │   │  │  │  │  └───6.35 MB (01.18%) ── unused-gc-things
│  │   │  │  │  ├──12.01 MB (02.24%) ++ (7 tiny)
│  │   │  │  │  └───8.86 MB (01.65%) ++ objects-extra
│  │   │  │  └───0.19 MB (00.04%) ++ compartment(https://mail.google.com/_/mail-static/_/js/main/m_i,t/rt=h/ver=sobKNxkH_hk.en./sv=1/am=!rsy2VLKefbT1RPF-0_JbQoI4nDfwRxdk-CbEiLu_TGxDGzw05OfUOUODZR_O2flp0CcdPg/d=1, about:blank)
│  │   │  └───4.89 MB (00.91%) ++ (4 tiny)

Accordingly, until now the preferred way of reporting about:memory’s data in bug reports has been to include a copy of the entire contents of about:memory?verbose.

This worked well when about:memory contained a few dozen lines.  But the level of detail in the underlying memory reporters has increased greatly since then.  This is a good thing — more data is helpful — but it means that a full about:memory?verbose is now typically thousands of lines.  When viewing the data directly in about:memory?verbose this isn’t a problem, because you can easily collapse and expand sub-trees to focus on the interesting parts.  However, when reading pasted data as text, it’s overwhelming, even for experts.

New: Gzipped JSON

With the landing of bug 848560, about:memory can now import and export memory reports as gzipped JSON.  Use the “Write reports to a file” and “Read reports from a file” buttons at the bottom of about:memory.  (Note: the UI will likely change soon, but not substantively.)

(The JSON schema is documented in a comment near the top of xpcom/base/nsIMemoryInfoDumper.idl, for those who are interested.)

This is now the preferred format for attaching memory report data to bug reports.  It has two important advantages over text.

  • When you load one of these files into about:memory, you can do the aforementioned collapsing and expanding of sub-trees.
  • If you have two of these files, you can diff them using toolkit/components/aboutmemory/tools/diff-memory-reports.js.  (For the moment you’ll have to ungzip both files first, which is stupid.  Hopefully this’ll be fixed soon.)

The ability to compute memory report diffs will be extremely useful for diagnosing cases where Firefox’s memory consumption increases unexpectedly.  (There are already two bugs that are waiting on this feature for a better diagnosis.)

There is a bug open to implement a “Load reports from URL” feature in about:memory, which will make the viewing memory reports attached to bugs even easier.  It’s currently unassigned;  please email me or comment in the bug if you are interested in implementing it.

Finally, although the gzipped JSON format is now the preferred format for transferring the full memory report data, copy and paste is still useful for communicating snippets of about:memory’s contents.  Support for it won’t be disappearing.

MemShrink progress, week 93–94

After lots of activity in the previous month, the past two weeks have been fairly quiet for MemShrink.

AWSY

areweslimyet.com has been proving its worth.

Jonathan Kew reduced the amount of memory taken by fonts on Fennec at start-up, which was detected by AWSY/mobile.  Jonathan also reverted a change that AWSY detected as increasing Fennec memory consumption, and filed a follow-up to investigate further.

Joe Drew fixed a bad regression on AWSY relating to image decoding.  It’s not clear to me if this was a genuine regression that users would have seen, or if it was an artifact of the way AWSY does its measurements.  Either way, it’s good that it was fixed, and props to Joe for doing it so quickly.

Finally, we closed bug 833518, which was for an AWSY regression caused by the new DOM bindings.  This was previously improved by an Aurora-only hack, but enough cases have been translated to the new bindings that we’re naturally down almost to where we were.

Miscellaneous

The mobile team abandoned their goal of making Fennec work on phones with only 256 MiB of memory.  The rationale is that Android phones with only 256 MiB of RAM are uncommon, whereas low-end phones that meet the current minimum of 384 MiB are much more common.  The mobile team will of course continue to look for ways to improve memory consumption in order to make life for users with 384 MiB phones.

I modified the JS engine so that it doesn’t emit bytecode for asm.js functions in the normal case.  This reduced the memory consumption of the Unreal 3 demo used at GDC by about 100 MiB.  I also added a memory reporter for array buffers used by asm.js, which are often quite large and weren’t being measured on 64-bit platforms.

Alexandre Poirot fixed a leak relating to dev tools.

Randell Jesup fixed a small leak in WebRTC.

Help Needed

I’m working on adding a button to about:memory trigger the dumping of memory reporter data to file.  I have a patch awaiting review, but I’m getting a test failure on Windows.  The test saves gzipped memory reports to file, and then immediately loads that saved file (and uncompresses it) and checks the data looks as expected.  This works fine on Mac and Linux, but on Windows I’m sometimes getting incomplete data in the load step.  The file is quite short (just 253 bytes compressed, and 620 bytes uncompressed) and the truncation point varies between runs;  in the most severe occurrence only 9 bytes of uncompressed data were loaded, though the cut-off point seems to vary randomly.

I suspect there’s a file synchronization problem between the save and the load, even though gzclose() is called on the save file before the loading occurs.  If anyone has ideas about what the problem might be, I’d love to hear them.

Update: Nils Maier and an anonymous commenter pointed out the problem — I was using “r” instead of “rb” for the file mode.  On Windows, this causes mangling of EOL chars.

Also, we’re seeing some strange behaviour on Mac OS X where memory managed by jemalloc doesn’t appear to be released back to the OS as it should.  This is both alarming and hard to understand, which is not a good combination.

Good First Bugs

I have two easy bugs assigned to me that I probably won’t get around to for some time.  Both of them would be good first (or second, or third…) bugs.

  • Bug 857382 is about making about:memory handle memory report diffs more elegantly.
  • Bug 798914 is just a minor code clean-up.  Nothing too exciting, but first bugs often aren’t!

Please email or comment in one of the bugs if you are interested in helping.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-0/+2)
  • P2: 138 (-4/+8)
  • P3: 129 (-2/+7)
  • Unprioritized: 1 (-3/+0)

MemShrink progress, week 83–84

Fixed

Justin Lebar made it so that memory reports could be collected on production B2G phones (i.e. not just developer phones with root access).  This was a MemShrink:P1, because getting these reports is crucial.

Gregor Wagner tuned the GC heuristics used by workers.  This is important for B2G, which uses workers extensively.

Andrew McCreight fixed a leak involving audio contexts.

I added a memory reporter for event targets, which includes XHRs.  This can measure multiple MiB of memory when running Gmail.

I added a memory reporter for data held by the JS engine’s regexp JIT compiler.  It usually measures insignificant amounts.

I fixed an inaccuracy in the “resident” memory report tree, which is visible in about:memory when running on Linux, which was caused by a change in recent kernels.

AWSY

The recent results on AWSY have been ugly.  There were two bad regressions in December, as the following graph makes clear.

areweslimyet.com, december 2012

John Schoenick did some work to improve AWSY to make regression hunting easier, and as a result we finally know which changes caused these regressions.

  • A refactoring of images code caused the bigger regression, on December 18.  Seth Fowler is looking into this.
  • Two changes relating to the new DOM bindings caused the smaller regression on December 11/12.  This is largely because many more JS getter/setter functions are present.  It’s not clear yet how to win back this memory, though it should be possible to turn these changes off in the short-term.

These regressions have made it to the Aurora branch, which means there is some urgency now to either fix them or back out/disable them soon.  We don’t want them to reach Beta.

Bug Counts

Here are the current bug counts.

  • P1: 21 (-1/+7)
  • P2: 125 (-2/+11)
  • P3: 104 (-1/+3)
  • Unprioritized: 4 (-17/+3)

The changes are larger than usual because we had a big log of untriaged bugs to go through, due to the six week break since the last MemShrink meeting.

MemShrink progress, week 79–82

I skipped the last MemShrink report due to Christmas, so we have four weeks’ worth of bug fixes today.

LEAKS

Joe Walker fixed a bad leak found by Jesse Ruderman:  if you closed a browser window with the developer toolbar open it would leak “everything”.  This was a MemShrink P1 bug.

Anton Kovalyov fixed a leak involving scratchpad.  This bug was also found by Jesse Ruderman.

Randell Jesup fixed some WebRTC leaks.

John Schoenick fixed a leak involving plugins.

Josh Aas fixed a leak in some networking code.

DMD

I wrote a more detailed blog post about DMD.  Here is the take-away message.

about:memory is MemShrink’s not-so-secret weapon when it comes to understanding Firefox’s memory consumption… and DMD is how we make about:memory better.

Lots of under-the-hood improvements have been made to DMD since I wrote that.  Users on Mac, Linux and B2G who aren’t afraid of doing their own builds should try it out.  Also, Ehsan Akhgari got it to build on Windows, though it’s not yet clear how well it works on that platform.  If anyone wants to try it out, please let me know how it goes.

Memory Reporters

Ben Turner made the workers memory reporter be able to handle workers that use ctypes.  This was important, especially on B2G, because each process can have one or two or more such workers — this is for Firefox chrome stuff, not web content — and we weren’t measuring them at all, and they can take multiple MiB each.

I fixed the orphan DOM node memory reporter.  The introduction of WebIDL had changed the layout of some paired JS/DOM objects, and such objects weren’t being reported.  (DMD discovered the unreported memory, and Boris Zbarsky helped me interpret what it meant.)  I see this accounting for multiple MiB of orphan nodes when using Gmail.

I added a memory reporter for the event listenener manager’s hash table.  It starts off small, but I’ve seen it go as high as 1.5 MiB after lots of browsing.  (DMD helped me identify this too.)

Kartikaya Gupta added a memory reporter for graphics textures on Android.

I added a memory reporter for any ctypes data that is hanging off JS objects.  I added it because one DMD profile on B2G showed non-trivial amounts of ctypes data, but that seems to have been a fluke and it rarely shows much memory now.  Oh well.

Miscellaneous

Andrew McCreight improved CC shutdown logging, which will make it easier to identify shutdown leaks.

Rail Aliiev and Kartikaya Gupta enabled a new NDK for Fennec builds on releng machines.  This might result in smaller binary sizes, which saves memory.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-2/+0)
  • P2: 116 (-10/+0)
  • P3: 102 (-4/+0)
  • Unprioritized: 18 (-0/+18)

The number of unprioritized bugs is high because we didn’t have a MemShrink meeting this week.  This was because Justin Lebar and Kyle Huey are in Berlin for the B2G work week.  We’ll have our next meeting two weeks from today.

DMD

I recently landed a new version of DMD on mozilla-central.  DMD is a tool helps us understand and thus reduce Firefox’s memory consumption.  But in order to understand DMD, you first have to understand about:memory.

about:memory

The MemShrink project started about 18 months ago, and it has been very successful in reducing Firefox’s memory consumption.  about:memory is MemShrink’s not-so-secret weapon when it comes to understanding Firefox’s memory consumption.

about:memory screenshot

about:memory has some wonderful characteristics:  it provides literally thousands of measurements;  it’s available in ordinary release builds;  and it’s trivial to run (just type “about:memory” in the address bar).  This means that non-expert users can easily provide developers with detailed measurements if they are having problems.

However, about:memory has two shortcomings.  First, it simply visualizes the data provided by the memory reporter infrastructure.  The coverage provided by this infrastructure is good, but there are still some gaps.  These gaps manifest primarily in the “heap-unclassified” number in about:memory, which represents all the heap allocations that the memory reporters didn’t cover.  Unfortunately, about:memory can provide zero insight into what is within “heap-unclassified”.

Second, it’s hard to verify.  There’s no obvious way to tell if the numbers it gives are accurate (with a few exceptions;  e.g. negative numbers are obviously wrong).  We have established good practices for writing memory reporters (measure sizes, don’t compute then;  traverse data structures rather than maintaining size counters) that prevent most errors, but it’s still quite easy to accidentally count a block of memory twice.

This is where DMD comes in.  It helps with both the heap-unclassified and double-counting problems.

DMD version 1

I wrote the first version of DMD over a year ago.  “DMD” is short for “dark matter detector”, because “heap-unclassified” memory is sometimes jokingly called “dark matter”.

DMD works by intercepting all calls to malloc/free/etc.  This lets DMD track extra information about every heap block, such as where it was allocated.  Furthermore, DMD has hooks into the memory reporters (via functions created with the NS_MEMORY_REPORTER_MALLOC_SIZEOF_FUN macro, for those who are interested) so it knows when any heap block is measured by a memory reporter.

With that in place, DMD knows how many times each heap block has been reported.  So, after running the memory reporters, we can up each block into one of the following three groups.

  • Blocks that haven’t been reported indicate gaps in existing memory reporters.  They contribute to “heap-unclassified”.
  • Blocks that have been reported once are good.
  • Blocks that have been reported twice or more indicate defects in existing memory reporters.  They cause some measurements in about:memory to be too high, and “heap-unclassified” to be too low.

DMD presents its results in a sorted fashion that lists the unreported and twice-reported cases that are more important first.

This first version of DMD was implemented as a Valgrind tool, and what’s more, it required a special, patched build of Valgrind to run.  This meant it was difficult to set up, ran very slowly, and could only be used on Linux and Mac.  Despite these shortcomings, it has proved itself extremely useful, helping us get “heap-unclassified” down greatly (on my Linux desktop machine it’s typically between 8 and 12%) and identifying several cases of double-counting.

DMD Version 2

Some time after I wrote the first version of DMD, I realized that all it needed to work was the ability to intercept malloc/free/etc. and to obtain stack traces.  Valgrind was overkill for these purposes — it’s capable of supporting much more invasive tools — and, furthermore, there were existing tools (e.g. trace-malloc) in the Mozilla codebase that did exactly these things.  In other words, it would be possible to build a new version of DMD that integrated directly into the browser.

Work on this new version stalled for a while, because the old one was working well enough.  But then B2G’s Operation Slim Fast started, and it quickly became obvious that “heap-unclassified” on B2G was typically much higher than it was on desktop.  This is primarily because B2G has lots of small processes, and so various unmeasured things that weren’t a big deal on desktop became much more significant on B2G.  And DMD version 1 doesn’t work on B2G devices.

So that provided the impetus to complete the new version.  Mike Hommey finished his new replace-malloc infrastructure recently, which helped greatly, and the new version of DMD landed on mozilla-central last week.  The old version will soon be retired.

DMD now works on Linux, Android, Mac, B2G, and is just shy of working on Windows (where Ehsan Akhgari has been making gradual progress).  It’s also much faster, partly because it avoids the overhead of Valgrind, and partly because it now uses sampling.  The sampling causes blocks smaller than a certain size (4 KiB by default, though you can change that) to be sampled, while blocks larger than that are measured accurately;  this sacrifices a moderate amount of accuracy for a large amount of speed.

Try it

about:memory is wonderful, and DMD is how we make about:memory better.  There are now detailed instructions on how to build DMD, run it, and interpret its output.  It’s quite easy now, requiring only minor changes to the usual build and run steps, and several people have already done it successfully.  Please try it out, and help us further improve about:memory.

MemShrink progress, week 73–74

B2G!  Fennec!  Social API!  They’re all happening, and memory consumption is a big deal for all of them.  Time for a MemShrink report.

B2G

Lots of B2G work is happening, unsurprisingly.  All of these changes have been backported to the Aurora channel.

Kyle Huey landed an important patch that merges system compartments together.  This avoids wasted space caused by having 100s of small compartments.  As Chris Jones said: “It’s a completely different phone with these patches.”  Unfortunately, the preference controlling this behaviour is currently turned off on B2G, because it’s causing some Marionette test failures.  Hopefully they’ll be dealt with soon.  Note that this preference won’t be turned on in desktop builds, and there’s a medium-term plan to avoid this wasted space in a less hacky fashion.

Marco Bonardo disabled Places in B2G, saving about 300 KiB in the main process.

Justin Lebar added code to trigger memory pressure events when lowmemkiller notifications occur.

Justin also tweaked things so that the garbage collector is more likely to run when screenshots are taken.

Jeff Muizelaar fixed a graphics bug that I don’t understand but apparently avoids allocating lots of memory on the B2G unlock screen.

Chris Jones ensured that remote content drop their buffers when they’re hidden.

I tweaked the size of the arena chunks used by XPT, which saved about 120 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I also shrunk the initial size of the SPS hash table, which saves 48 KiB per process on 64-bit builds and 24 KiB on 32-bit builds.

Memory Reporting

Lots of work happened on the memory reporting front.  This was inspired by  “heap-unclassified” typically being much higher on B2G than on desktop.  Desktop Firefox uses a single process and its memory consumption is usually measured in the 100s of MiBs or more.  In contrast, B2G uses one “main” process and then one process per running app, and the smaller ones are typically around 10 MiB.  As a result, the sundry small per-process things that don’t matter much for desktop loom larger on B2G.  (For the same reason, changes that reduce per-process memory by smallish amounts have outsize value on B2G.)  Most of the following changes have also been backported to the Aurora channel.

Nick Hurley added a “explicit/network/disk-cache” memory reporter.  It typically measures 100s of KiBs in desktop Firefox.  (It doesn’t get used on B2G.)

I added memory reporters “explicit/xpcom/component-manager” and “explicit/xpcom/category-manager”.  Together they measure about 280 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/script-namespace-manager”.  It measures just over 150 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/xpcom/effective-TLD-service”.  It measures about 128 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added some detail to the JS object memory reporters.  This gives slightly more insight into where this memory is going, but the extra memory measured as a result is usually pretty small.

I fixed the “explicit/atom-tables” memory reporter, which was erroneously always reporting 0 bytes.  It now measures anywhere from a few 100 KiBs to several MiBs of memory.  This was a frustrating bug to find.  DMD exists to check that memory reporters are measuring memory properly, and the reporter was doing its measurements just fine.  But DMD cannot check that the measured amounts are added correctly, and that’s what was going wrong here — a line that should have been this:

return n;

instead was this:

return 0;

In other words, the code I had written was doing the difficult part correctly, but botched the trivial part.  How annoying.

Social API

Felipe Gomes fixed a document leak that occurred when the social sidebar was hidden, and disappeared only when it was unhidden.

Felipe also added code to clear the previous profile when Social is toggled off, which prevents a leak.

The social API still has some significant unresolved memory consumption issues, which are a concern.

Fennec

Kartikaya Gupta (a.k.a. Kats) turned on tab expiration for Fennec.  What this means is that Fennec will now unload the contents of background tabs in certain circumstances — when memory is low, and in some cases when a tab hasn’t been viewed for over an hour — and then reload them when they are brought back into the foreground.  Read Kats’ blog post for more details.

(This change was prompted by Project 256meg, which aims to make Fennec usable on phones with only 256 MiB of RAM.  (Fennec currently requires 512 MiB, according to the official specs.)  Kats has just got to the point where Fennec can actually start on 256 MiB devices.  Although Fennec only uses a single process, there is obviously significant overlap between this goal and B2G’s memory reduction goals.)

An obvious follow-up idea is to add tab expiration to desktop Firefox.  But the bug tracking that idea has seen more heated discussion.  The reason is that tab expiration might cause data loss in some cases.  This is less of a problem on Fennec because (as far as I can tell) there is less of an expectation that in-flight data will be saved on mobile devices.  For example, having processes killed due to memory constraints on mobile is much more common than on desktop.  Still, I expect plenty more arguing before this issue is resolved one way or the other.  One option is to allow it on desktop but have it disabled by default.

Add-ons

Michel Gutierrez fixed a zombie compartment in Video DownloadHelper, which is the 2nd most popular add-on at AMO.   Version 4.9.11 has the fix.

We hit a notable milestone on the add-on front this fortnight:  on November 1st hit bug for tracking known leaks in add-ons — filed 16 months ago — had zero blockers.  In other words, we reached the point where no add-ons were known to leak!  Unfortunately this didn’t last long, and at the time of writing the tracking bug has three blocking bugs.  Still, it’s confirmation that what used to be our biggest memory consumption problem is well under control.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+5)
  • P2: 110 (-10/+7)
  • P3: 102 (-3/+8)
  • Unprioritized: 4 (-2/+4)

As always, the current bug lists can be found by following the links on the MemShrink wiki page.