Category Archives: compartments

MemShrink’s 1st Birthday

A year ago today, the first meeting of the MemShrink project took place.  The project had been announced almost three months earlier, but until that first meeting it was basically just me working on it and there wasn’t much momentum.  So I now think of those three months as something of a gestation period, and the first meeting as the true birth of the project.

So, happy birthday, MemShrink!  I’ll take this opportunity to go over some of the accomplishments we’ve made in the past year.

Big Win #1: Better Javascript Heap Management

There have been two huge MemShrink improvements that stand out in my mind.  The first one happened very early on, and it consisted of two related changes, both implemented by Gregor Wagner.

In Firefox 4, the implementation of the JavaScript heap was totally overhauled.  In particular, it was segregated into compartments, each of which held (roughly speaking) the memory for a single domain.  System compartments, i.e. those for JavaScript code that implements parts of Firefox (such as the UI), tend to be long-lived.  In contrast, user compartments, i.e. those for JavaScript code in web pages, tend to be short-lived.  But memory from both system and user compartments was being co-located in 1MB chunks.  What tended to happen is that if you browsed for a long time and then closed many tabs, you’d end up with lots of 1MB chunks that were kept alive by a small amount of memory from system compartments.  This is a form of fragmentation, and it prevented Firefox from releasing lots of memory back to the operating system that it should have been able to.

Fortunately, Gregor came up with a simple fix:  don’t allow system and user compartments to share the same 1MB chunks.  To see just how effective this was, consider this visualization of the JavaScript heap after closing many tabs, without Gregor’s change.

badly fragmented JavaScript heap

Each horizontal line is a 1MB chunk, and each square is a 4KB arena.  White squares are unused arenas, and coloured squares are in use.  Arenas belong to a particular compartment all have the same colour;  there are three compartments alive, all of which are system compartments.  You can see that many 1MB chunks are kept alive by a small number of arenas.

After Gregor’s change, in this scenario the heap ended up looking like this.

non-fragmented JavaScript heap

Much better.

Gregor’s second change related to garbage collection scheduling.  In Firefox 4 the garbage collection heuristics were poorly tuned, and the garbage collector simply didn’t run frequently enough.  Many people who left Firefox open for a while came back to find that JavaScript memory consumption had ballooned, and the machine was unusable due to paging, for seconds or minutes, until the garbage collector kicked in and paging finished.  The fix was simple:  run the garbage collector occasionally, using a timer.

These two fixes of Gregor were the main reason why the first item in the Firefox 7 release notes was “drastically improved memory handling for certain use cases”.  More specifically, Firefox 7 used less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.

Big Win #2: Fewer add-on Leaks

The second huge MemShrink improvement is much more recent.  In fact it hasn’t even made it into a released version of Firefox yet.  And the path towards it was much more circuitous.

I first revamped about:memory in May of last year, and those changes made it into Firefox 6. This started giving us insight into Firefox’s memory consumption, but it wasn’t until I broke up the reporting of JS memory consumption on a per-compartment basis that it really started to make a difference.  (This change made it into Firefox 7, another reason why that release was so good from a memory consumption point of view.)

Per-compartment reporters made some blatant inefficiencies obvious, which were quickly fixed.  More importantly, people quickly noticed that sometimes compartments were present for sites that had been closed long ago.  This is a kind of memory leak that became known as a zombie compartment, and hunting them became a popular enough sport that  more than 100 zombie compartment bug reports have been filed.

Some of the zombie compartments were due to defects in Firefox itself, and these were generally fixed fairly quickly.  However, it soon became clear that the majority of them are due to add-ons.  It’s quite easy to unintentionally create zombie compartments in add-ons.  In the worst case, add-ons could leak the compartment of every single site visited.

This led to some lively discussion about how best to handle these leaks, because they are defects in third-party code that is largely out of Mozilla’s control, and yet they make Firefox look bad.  Fortunately, this discussion became mostly moot when Kyle Huey took advantage of an 8,000 mile trip from home to implement a crazy idea he’d had six months earlier –  when a tab is closed, simply cut the references from other compartments that would keep the compartment of the tab’s page alive.  This turns out to work tremendously well and prevents most of the cases where add-ons can create zombie compartments.  This change is scheduled to ship in Firefox 15, which is due for release in late August, and should more or less fix what I six months ago ranked as the single worst problem affecting Firefox’s memory consumption.

The Key Tool: Memory Reporters and About:memory

I mentioned about:memory above.  It’s worth discussing in more detail, because it’s the single most important tool we’ve created, and has driven a lot of the MemShrink improvements.

Let’s look at about:memory as it was in Firefox 6, which was released about 10 months ago.about:memory screenshot from Firefox 6

Even these early versions had several key properties.

  • about:memory doesn’t need a special build to run;  it works even in released versions.
  • It is trivially easy to use — you just type “about:memory” in the address bar.
  • Because it’s all text you can cut and paste the output, and unlike many web pages it is carefully constructed so that it reproduces beautifully when you do so.  No screenshots are necessary.

These properties massively expand the number of people who are able to detect and report problems relating to memory consumption.  Any Firefox user can potentially do it;  if the user doesn’t already know how, they can be taught in 10 seconds.

Having said that, this old version is almost laughably primitive compared to what we have now.  First, let’s consider the state of the memory reporting infrastructure, which about:memory completely relies on, as it was at that time.

  • There were a fixed number of measurements, which meant there were no individual measurements for variable-numbered entities such as compartments, window objects, or SQLite connections.  Such measurements were only possible once I added support for memory multi-reporters.  We now have dozens of measurements per variable-numbered entity, and thousands of measurements in total.
  • Even once per-compartment reporters were added, we still didn’t have much insight into the memory consumption of the JavaScript code used by Firefox itself (and its add-ons).  But compartment-per-global landed recently and gave us much finer-grained measurements, which (a) gives us some idea of how much memory is being used by JavaScript-based add-ons, and (b) gets us very close to being able to report memory consumption for each tab.  Users have been requesting both of these things for years.
  • Early memory reporters had lots of defects that caused them to report incorrect numbers, but these were obvious only when the numbers were ridiculously large or negative (both of which happened on occasion).  The first of two major improvements in correctness came about as we learned how to structure memory reporters to avoid common defects.  In particular, it’s best to write reporters that are traversal-based rather than counter-based, and to measure heap blocks with malloc_usable_size rather than computing the sizes.  Reporters that use malloc_usable_size also measure slop bytes, which are caused by the heap allocator rounding up allocation requests and can account for 5–10% of the heap.  Early reporters didn’t have these characteristics, but most of them have been converted now.
  • Reporters with these characteristics are doubly-virtuous, because they also integrate well with DMD, a tool I wrote that dynamically checks memory reports.  DMD provided the second major improvement in memory reporter correctness, because it identifies heap blocks that are counted twice, which is a mistake that’s easy to make.
  • DMD also identifies unreported heap blocks, and was crucial in getting the “heap-unclassified” number down.  In the example above it was 68.82%(!), whereas now it’s typically around 15%, which is low enough that people have mostly stopped complaining about it.  Prior to DMD, identifying this “dark matter” was much harder and less systematic.
  • The only tree shown in the above example is “explicit”.  On Linux, Justin Lebar added code to summarize the information in the /proc/<pid>/smaps file in “size”, “rss”, “pss” and “swap” trees.
  • The reporters back then could only report byte measurements.  They can now report unitless counts and percentages, both of which are useful.

Now consider the about:memory page itself, as it was back then.

  • Insignificant measurements were simply omitted, and you had to reload in verbose mode to see them (and the measurements would be re-taken when that happened, potentially losing any interesting measurements).  Now it is possible to expand and collapse arbitrary parts of the trees on the fly.
  • The “Other Measurements” section contained a flat list of measurements.  Trees are now possible, which makes that section more flexible and easier to understand.
  • about:memory’s has been carefully streamlined to consume as little memory as possible, so that it doesn’t perturb what it’s measuring.  This is really important now that we report so many measurements.
  • about:memory now has a companion page, about:compartments, that just lists compartments (and ghost windows).  about:compartments shares a lot of code with about:memory, and it is better than about:memory for certain tasks such as spotting zombie compartments.

All this stuff is important because you make what you measure.  Just about every time we’ve started measuring some new thing, we’ve found and fixed problems as a result.

Smaller Things

The following smaller accomplishments were also made in the past year.

  • John Schoenick used the MozMill endurance tests to build areweslimyet.com, which helps identify regressions in memory consumption.
  • I removed a lot of unnecessary memory waste caused by “clownshoes” allocations.
  • Olli Pettay wrote about:cc and Jan Honza Odvarko wrote about:ccdump.  These closely related tools have been used to find various cycle collector leaks.
  • I wrote 41 progress reports.  I had/have two goals with these:  to keep interested people informed of what it happening, and to provide evidence that Mozilla actually does care about memory consumption in Firefox.

This is only the tip of the iceberg, though.  As I am writing this, there have been 314 MemShrink-tagged bugs marked FIXED in Bugzilla.  In comparison, there are only 221 MemShrink-tagged bugs currently open.  Breaking them down by priority we get the following.

  • P1: 52 fixed, 25 open
  • P2: 135 fixed, 88 open
  • P3: 53 fixed, 106 open
  • Unprioritized: 74 fixed, 2 open

Over 75 people were responsible for these fixes.  I give many thanks to all those people, plus everybody who contributed in any other way to fixes, such as bug reporters and patch reviewers.  And I’d like to give extra thanks to the MemShrink regulars:  Justin Lebar, Kyle Huey, Andrew McCreight, Johnny Stenback, and Jet Villegas.

What’s Next?

There’s no real secret to MemShrink.  So far it’s basically been a long, steady grind, gradually improving tools, fixing leaks, slimming down data structures, and responding to user’s problems.  There are no plans to change that, and we’ll continue our recently-lowered pace of fortnightly meetings for the foreseeable future.  And if we’re lucky, I might be writing a report on MemShrink’s second birthday a year from now!

MemShrink progress, week 49-50

System Compartment Reporting

With the recent landing of compartment-per-global, Firefox now regularly has 200+ system compartments at start-up.  However, most of these compartments didn’t have names, which meant that they were merged into a single “[System Principal]” entry in about:memory and about:compartments.

Until last week, that is, when Nils Maier added identifying information to the vast majority of these.  Here’s a small selection of interesting ones from about:compartments on my own machine.

about:blank
about:compartments
about:memory?verbose

chrome://adblockplus-modules/content/FilterClasses.jsm

chrome://browser/content/bookmarks/bookmarksPanel.xul
chrome://browser/content/browser.xul
chrome://browser/content/places/menu.xml
chrome://browser/content/search/search.xml

chrome://chatzilla/content/browserOverlay.xul

chrome://global/content/bindings/button.xml
chrome://global/content/globalOverlay.xul

chrome://treestyletab/content/treestyletab.xul

file:///home/njn/moz/mi0/o64/dist/bin/components/TelemetryPing.js

jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/bootstrap.js
jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/components/PdfStreamConverter.js

resource:///modules/TelemetryTimestamps.jsm
resource:///modules/sessionstore/DocumentUtils.jsm

resource://gre-resources/hiddenWindow.html

resource://gre/modules/AddonManager.jsm

resource://services-common/preferences.js
resource://services-crypto/WeaveCrypto.js
resource://services-sync/constants.js
resource://services-sync/engines/bookmarks.js

resource://treestyletab-modules/browser.js
resource://treestyletab-modules/lib/animationManager.js

Just from this, it’s obvious that I had about:compartments and about:memory?verbose open at the time.  It’s also obvious that I had the following add-ons installed:  AdBlock Plus, Chatzilla, Tree Style Tab, and pdf.js.  And about:memory now gives at least a partial measurement of how much memory these add-ons are using.  This will help identify add-ons that are using excessive amounts of memory.  (Having said that, I identified the add-on compartments simply by their names.  It’d be great if there was a way to systematically identify them within the code, but I don’t know if that’s possible.)

I also hope people will scrutinize Firefox’s own compartment use closely, and start to file bug reports saying things like “hey, that .jsm module shouldn’t be present, there must be a leak”.  If you want to see what the full list of 200+ looks like, try out a recent Nightly build!

In related news, I also added some new compartment-specific reports, including ones for cross-compartment wrappers.

One consequence of all these change is that the number of entries in about:memory jumped tremendously.  As a result, I aggregated the small entries within each compartment, which reduces the number of entries by a factor of roughly four while still reporting full information for large compartments.  Nils and I also made about:memory more efficient, so the amount of memory required to generate each line dropped by about 20%.  about:memory still takes up memory itself, but it does so at a level that I’m fairly happy with.

Add-ons

For a change, the biggest MemShrink-related news in this report wasn’t related to add-ons!  But there was still some interesting movement there.

Justin Lebar uncovered some evidence that the Hueyfix is having a real, positive effect among users.  Telemetry data from Nightly users shows that the number of ghost windows — a concept for which we don’t have good documentation, but they correlate with zombie compartments — has dropped dramatically, as the following graph shows.

ghost windows telemetry data graph

Telemetry data tends to be extremely noisy, so it’s nice to see a clear signal — Kyle’s change made it into Nightly builds on May 5th [Update: that's incorrect, see below] and immediately caused the mean number of ghost windows to drop from roughly three to roughly one.  The variance also dropped dramatically.

Update: Justin just wrote a blog post that explains very nicely what ghost windows are.  That post also explains better the circumstances behind the drop in ghost window numbers;  my explanation above was too simple and got the timing wrong.  Thanks, Justin!

In other add-on news, the following add-ons had leaks fixed: Readability, ProxTube, Youtube MP3 Podcaster.

Firefox vs The New York Times

Robert O’Callahan fixed a leak relating to mouse events that triggered when he visited nytimes.com.  He wrote a great blog post explaining the heroic debugging — searching through full memory dumps! — that was required.  It’s great that Robert found and fixed this, though it’s a shame it took such expertise.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-1/+2)
  • P2: 85 (-2/+4)
  • P3: 102 (-5/+2)
  • Unprioritized: 2 (-3/+2)

Not a great deal of movement.  We only had to triage twelve bugs in today’s MemShrink meeting, which is the fewest we’ve had since we switched to fortnightly meetings.

MemShrink progress, week 42

 Are we slim yet?

John’s Schoenick’s areweslimyet.com (AWSY) has had its password removed and is now open to the public!

areweslimyet.com screenshot

This is a major milestone for the MemShrink project.  It shows the progress we have made (MemShrink started in earnest in June 2011) and will let us identify regressions more easily.

John’s done a wonderful job implementing the site.  It has lots of functionality:  there are multiple graphs, you can zoom in on parts of graphs to see more detail, and you can see revisions, dates and about:memory snapshots for individual runs.

John has also put in a great deal of work refining the methodology to the point where we believe it provides a reasonable facsimile of real-world browsing;  please read the FAQ to understand exactly what is being measured.  Many thanks also to Dave Hunt and the QA team for their work on the Mozmill Endurance Tests, which are at the core of AWSY’s testing.

Update: Hacker News has reported on this.

Ghost windows

Frequent readers of this blog will be familiar with zombie compartments, which are JavaScript compartments that have leaked, due to defects in Firefox or add-ons.  Windows (i.e. window objects) can also be leaked, and often defects that cause compartments leaks will cause window leaks as well.

Justin Lebar has introduced the notion of “ghost windows”.  A ghost window is one that meets the following criteria.

  1. Shows up in about:memory under “window-objects/top(none)”.
  2. Does not share a domain name with any window under “window-objects/active.
  3. Has met criteria (1) and (2) for a moderate amount of time (e.g. two minutes).

The basic idea is that a ghost window has a high chance of representing a genuine leak, and this automated identification of suspicious windows will make leak detection simpler.  Justin has added ghost window tracking to about:memory, about:compartments, and telemetry.  (These three bugs were all marked as MemShrink:P1.)  Ghost window tracking is mostly untested right now, but hopefully it will become another powerful tool for finding memory leaks.

Add-ons

We’ve been tracking leaky add-ons in Bugzilla for a while now, but we’ve never had a good product/component to put them in.  David Lawrence, Byron Jones, Stormy Peters and I have together created a new “Add-ons” component under the “Tech Evangelism” product.  The rationale for putting it under “Tech Evangelism” is that it nicely matches the existing meaning of that phrase — it’s a case where a non-Mozilla entity is writing defective code that interacts with Firefox and hurts users’ experiences with and perceptions of Firefox, and Mozilla can only inform, educate and encourage fixes in that defective code.  This component is only intended for certain classes of common defects (such as leaks) that Mozilla contributors are tracking.  It is not intended for vanilla add-on bugs;  as now, they should be reported through whatever bug-reporting mechanism each add-on uses.  I’ve updated the existing open bugs that track leaky add-ons to use this new component.

Leaks in the following add-ons were fixed:  Video DownloadHelper (the 2nd most popular add-on on AMO!), Scrapbook Plus, Amazon Price Tracker.

Bug counts

This week’s bug counts:

  • P1: 21 (-5/+0)
  • P2: 137 (-1/+7)
  • P3: 90 (-1/+3)
  • Unprioritized: 1 (-1/+1)

Good progress on the P1 bugs!

A new reporting schedule

Many of the weekly MemShrink reports lately have been brief.  From now on I plan to write a report every two weeks.  This will make things easier for me and will also ensure each report is packed full of interesting things.  See you again in two weeks!

MemShrink progress, week 36

Lots of activity this week.

Leaky add-ons

I created a new devmo wiki page documenting common causes of memory leaks in add-ons.  I based this on a lovely selection of examples provided by Nils Maier.  Unfortunately, I know almost nothing about writing add-ons and so I’m not happy with the current state of the documentation.  I’ve taken it as far as I can, but it very likely has errors, leaves out important cases, conflates distinct concepts, and generally is not as good as it should be.  This documentation is really important — we have good tools for identifying when add-on leaks occur but we don’t yet have good tools for identifying their causes.  Until we have those tools, documentation is the only way we can help add-on authors fix leaks.  If someone who knows more about add-ons is willing to help please contact me!

Alexandre Poirot fixed a leak in the Add-on SDK that was causing zombie compartments when certain add-ons were disabled.  This also fixed bug 725603 and probably fixes some zombie compartments reported in some other add-ons (e.g. MemChaser).

Jan fixed several leaks in the Galaxytoolbar add-on.  The new version is 2.6.12.

A small number of add-ons have been found that disable Firefox’s XUL cache.  This is a recipe for disastrous performance, and so Matt Basta updated the AMO add-on validator to detect this.

Tools

I landed support for a new page called about:compartments. The exposure of zombie compartments in about:memory has become an extremely powerful leak detection tool.  But using about:memory to find zombies is non-trivial — there are multiple steps and ways to get it wrong.  The motivation for about:compartments is to make the detection of zombie compartments as simple as possible.  Here’s a screenshot:

about:compartments screenshot

Some things to note:

  • It only shows compartments, and all of them.
  • User compartments from web content (usually the interesting ones) are listed separately from system compartments.
  • The garbage and cycle collectors are automatically run when the page is loaded, ensuring that dead compartments aren’t listed.
  • The “More verbose” link at the bottom just causes the truncated URLs to be shown in full.
  • about:compartments shares a lot of code with about:memory.
  • I updated the documentation on zombie compartments accordingly.

Olli Pettay created about:cc, an add-on that can detect various kinds of document leaks relating to cycle collection.  Olli has already found several leaks with this tool.  Jan Honza Odvarko has written about:ccdump, a similar but prettier add-on.  I’m not certain but I think Olli and Jan are now co-ordinating their efforts.

I mentioned Mozilla QA’s MemChaser add-on a few weeks ago.  It lets you track memory usage and GC/CC activity easily via the add-on bar, and also allows logging of memory-related activities.  It’s now available on AMO, which guarantees you’ll receive updates as they’re released.

Jesse Ruderman tweaked his DOM fuzzer and found several new small leaks relating to nsITimers.  Jesse also modified about:memory so that it produces assertions that his fuzzer will catch.  This means that his fuzzer will detect if any memory reporters produce bogus values.

Miscellaneous

Kyle Huey fixed a zombie compartment that occurred when searching within pages with onclick handlers, which is a pretty common operation.

Jeff Muizelaar fixed a huge Mac-only leak relating to text rendering.  This was a recent regression that isn’t present in any released version of Firefox.

Josh Aas fixed a bug in cookie clearing — prior to Josh’s fix, if you cleared all cookies, Firefox would launch an instance of the plugin-container process for every plug-in installed, which could cause freezes and memory spikes.

Andrew Quartey updated the WebGL memory reporters to the new style.

Bug counts

This week’s bug counts:

  • P1: 28 (-1/+3)
  • P2: 134 (-6/+9)
  • P3: 79 (-8/+11)
  • Unprioritized: 2 (-1/+2)

Lots of movement there:  plenty of bugs fixed, but even more new ones.  Roughly 20 of the new ones fell into the following three categories.

  • Zombie compartments found in AMO add-ons by Kris Maglione and Andreas Wagner.
  • Leaks found by Olli Pettay’s new about:cc tool, mentioned above.
  • Leaks found by Jesse Ruderman’s tweaked DOM fuzzer, mentioned above.

This is a good thing!  It shows that new policies and tools are exposing existing problems.  I expect this higher level of new bug filing will continue for a couple of weeks.

Notes on Reducing Firefox’s Memory Consumption

I gave a talk yesterday at the linux.conf.au Browser MiniConf, held in Ballarat, Australia.  Its title was “Notes On Reducing Firefox’s Memory Consumption”.

Below are the slides and notes in a SlideShare embedding. If you find that embedding problematic (some people do) you may prefer to download the PDF version directly.

MemShrink progress, week 30

Add-ons

This was the week of add-ons in MemShrink-land.

Jared Wein fixed a problem in Firefox that was causing zombie compartments if you viewed a native video with any add-on installed that implements the nsIContentPolicy interface.  Examples of such add-ons are Adblock Plus, GreaseMonkey, and NoScript, which are respectively the #1, #3 and #9 most popular add-ons on AMO!  Welcome to the MemShrink club, Jared.

Speaking of GreaseMonkey, Arantius fixed a bug in it that was causing zombie compartments on some GM scripts when opening a background tabs.

I can’t tell who was responsible for the next fix, because the Add-on SDK folks use Github in a way I don’t understand.   But Myk Melez and/or Gabor Krizsanits greatly reduced the number of compartments used in JetPack-style add-ons.  In one example, the number of compartments dropped from 156 to 8, saving about 20MB of memory.  This was a MemShrink:P1 bug.

Also, as far as I can tell, the same patch also fixed bug 680821 which means that all compartments that hold sandboxes created by JetPack-style add-ons will be marked as belonging to that add-on in about:memory.

Jordan Miner reported that the Delicious Bookmarks add-on is leaking excessive numbers of connections to the Places database.  I tried to reproduce the problem and failed, but I don’t have a Delicious account.  If anyone else who does have a Delicious account is able to reproduce, please let me know or comment in the bug.

Finally, Jorge Villalobos and Andrew Williamson from the add-ons team came to this week’s MemShrink meeting.  We had a very fruitful discussion about how to help add-on authors detect and avoid memory leaks, and how to help add-on users understand which add-ons have high memory consumption.  Stay tuned for more details in the coming weeks!

Three Cheers for New Contributors

Krzysztof Kotlenga, a new contributor, removed an OpenGL cache that was wasting 1GB+ of memory on Linux when hardware acceleration was enabled (it’s currently not enabled by default, but will be at some point in the future).   Great work, Krzysztof!  [Update: I originally wrote WebGL instead of OpenGL, which was incorrect.]

Bug Counts

Here are the current bug counts.

  • P1: 26 (-2/+0)
  • P2: 132 (-14/+3)
  • P3: 67 (-2/+5)
  • Unprioritized: 4 (-0/+4)

That’s a net reduction of six bugs.  The main factor here was that I went through some of our P2s and closed ones that were stale and/or covered by other bugs, and downgraded to P3 a few more that are now known to be less important than we first thought.

Before finishing, I’d like to highlight bug 703427.  Richard Hipp, one of the SQLite developers, has a tiny patch that drastically reduces the amount of memory used by SQLite in Firefox — I ran with it for a while and my SQLite memory consumption dropped from ~15MB to less than 3MB.  The patch also causes a moderate speed drop, but it’s unclear if that speed drop is noticeable to Firefox.  We need someone who understands Firefox’s SQLite usage well to evaluate this patch so that Richard knows if it’s something that should go into a released version of SQLite as an option.  The bug is assigned to Marco Bonardo but he’s currently very busy.  Is there anyway else who knows enough about SQLite to take on this bug?

MemShrink progress, week 5

There were four main areas of MemShrink progress this week.

Killing zombie compartments

A zombie compartment is one that outlives its tab.  Kyle Huey fixed a very common class of short-lived zombie compartments caused by XmlHttpRequests.  These zombie compartments only lived for a couple of minutes, but this is an important fix because it makes makes memory usage follow tab usage better, and also makes detecting longer-lived zombie compartments much easier.

Alon Zakai has also been doing heroic work tracking down a zombie compartment related to web workers exhibited at coffeekup.org.  I’m sure he’d appreciate additional help from anyone who might be able to provide it.

Improving about:memory

Lots of progress here.

  • I changed the per-compartment memory reporters so that multiple system compartments with the same name are reported separately, instead of being merged.  This made it obvious that JetPack-based add-ons can have dozens of system compartments.  I don’t know if anybody realized this previously, and it’s something of a concern.  The compartments are currently distinguished in about:memory only by their address, as the following example shows.  system principal including its address
    It would be great to add some identifying information that indicates what part of the code is responsible for creating each system compartment.  That might even give us some much-needed per-add-on accounting.
  • Andrew McCreight added a memory reporter for the cycle collector.
  • I added a memory reporter for the Necko (network) memory cache.
  • Justin Lebar fixed a problem with the image memory reporters, but it bounced due to a possible Tp5 RSS/PrivateBytes regression on Mac.  This was surprising, maybe it was just noise?
  • I changed about:memory so that if there are multiple memory reporters with the same name, you can tell they were merged into a single entry.  For example, in the following screenshot you can easily tell that there were four separate connections to the places database.
    places entry indicating duplication

Acting on memory pressure

Justin Lebar made some great progress on triggering memory pressure events when physical or virtual memory is low on Windows.  See comments 28 and 29 on that bug for some nice graphs showing how this kept Firefox responsive when browsing some very image-intensive sites.  This kind of adaptive behaviour is really important for getting Firefox to behave well on the full range of supported devices, from smartphones all the way up to desktop machines with lots of RAM.  Go Justin!

Tweaking MemShrink processes

The MemShrink wiki page used to be vague about the project’s goals.  So this week I made it more precise.

[The] goal is to get the number of MemShrink P1 bugs down to zero. That will mean that all the bad leaks will have been fixed, and also the important auxiliary work (e.g. infrastructure to detect regressions) will be in place.

As a result, Jeff Muizelaar changed areweslimyet.com to point to the list of open MemShrink P1 bugs.  It previously pointed to a Tp5 Talos graph.

On a related noted, we have six open meta-bugs for tracking leaks against Firefox releases, one for tracking non-leak memory use reductions, and one for improving memory-related tools.  (These are listed on the wiki.)  I created these bugs before MemShrink meetings started.  But now that we are using the MemShrink whiteboard annotations assiduously, these tracking bugs don’t seem necessary — having two tracking mechanisms is overkill.  In particular, I think their dependencies aren’t being updated consistently.  So I propose to stop using them and close them.  If you have any objections, please let me know and I’ll reconsider.  If I do close them, I’ll make sure that all bugs blocking them have a MemShrink annotation so they won’t fall through the cracks.

And that segues nicely into the MemShrink bug count for this week:

  • P1: 26 (+2)
  • P2: 49 (+0)
  • P3: 33 (+4)
  • Unprioritized: 6 (+4)

Like last week, this increase mostly reflects the fact that people are coming up with new ideas for improvements.

Finally, thanks to Jesse Ruderman for taking minutes at this week’s meeting (and the previous two).

Zombie compartments! Recognize and report them. Stop the screaming.

Update (November 30, 2011): I wrote a wiki page about zombie compartments.  It’s much clearer than this post, you should read it instead.

Update (July 31, 2011): This blog post has been linked to from mozilla.org‘s front page.  Although any help in improving Firefox’s memory usage is very welcome, please note that this post was aimed at Firefox developers and other technically-inclined users, and I wasn’t expecting its existence to be publicized so widely.   Furthermore, the per-compartment reporters that help identify zombie compartments were only added to Firefox 7 (currently in the Aurora channel), and several existing bugs that cause zombie compartments have been subsequently fixed in the Firefox 8 development code.  This means the hunting of zombie compartments is a sport best left to those who either are using Nightly builds or their own development builds of Firefox.

Firefox’s JavaScript memory is segregated into compartments.  Roughly speaking, all memory used by JavaScript code that is from a particular origin (i.e. website) goes into its own compartment.  Firefox’s own JavaScript code also gets one or more compartments.  Compartments improve security and memory locality.

Per-compartment memory reporters allow you to look at about:memory to see what compartments are present.  Once you close a tab containing a web page, all the compartments associated with that web page should disappear.  (But note that they won’t necessarily disappear immediately;  garbage collection and/or cycle collection has to run first.)

Sometimes this doesn’t happen and you end up with a Zombie Compartment.  This shouldn’t happen, and it indicates a bug.  It also makes children and 1950s B-movie actresses scream.

 1950s B-movie woman screaming

If you notice Zombie Compartments while browsing, please report them to The Authorities.  Here are some steps you can follow when reporting one that will increase the chance it’ll be hunted down.

  • First, you should use about:memory?verbose for diagnosis.  You want the “?verbose” suffix (which you can also get to by clicking the “More verbose” link at the bottom of about:memory) otherwise small compartments might be omitted.
  • Second, beware that many sites utilize scripts from other origins.  Scripts from Google, Facebook and Twitter are particularly common. This means that the most reliable diagnosis of a Zombie Compartment occurs if you do the following: start Firefox anew, open about:memory?verbose and one other tab, then close that other tab, then hit “minimize memory usage” at the bottom of about:memory?verbose several times to force multiple garbage and cycle collections.  (Sometimes hitting it once isn’t enough, I’m not sure why.)  If the compartment remains, it’s very likely a Zombie Compartment.
  • After that, try waiting a while, say 10 or 20 minutes, then try the “minimize memory usage” button again.  Some Zombie Compartments stick around for a limited time before disappearing;  others are immortal, and it’s useful to know which is which.
  • Some Zombie Compartments are caused by add-ons.  So if you have add-ons enabled, please try to reproduce in safe mode, which disables them.  If you can identify, by disabling them one at a time, a single add-on that is responsible, that is extremely helpful.  Zombie compartments that are caused by add-ons are definitely interesting, but their importance depends on the popularity of the add-on.
  • Finally, please file a bug that includes all the information you’ve gathered, add “[MemShrink]” to its whiteboard, and mark it as blocking bug 668871.  Attaching the full contents of about:memory?verbose is very helpful.  See bug 669545 for an example.

Please, stop the screaming.  Report Zombie Compartments to The Authorities.