about:memory AdBlock Plus add-ons Firefox Garbage Collection Memory consumption MemShrink

MemShrink progress, week 32

There wasn’t much MemShrink activity this week in terms of bugs fixed, just bug 718100 and bug 720359.  So I’m going to take the opportunity this week to talk about the bigger picture.

Bug Counts

As a prelude, here are this week’s bug counts.

  • P1: 20 (-4/+0)
  • P2: 131 (-3/+3)
  • P3: 74 (-2/+7)
  • Unprioritized: 4 (-3/+4)

The drop in P1s was just due to bug re-classification;  in particular, three bugs relating to long cycle collector pauses were un-MemShrink’d because they are more about responsiveness, and they are being tracked by Project Snappy.

The Big Ticket Items

David Mandelin asked me today what where the big ticket items for MemShrink.  I’d been looking a lot at the MemShrink:P1 list recently (which is why some were re-classified) and so I was able to break it down into six main areas that cover most of the P1s and various P2s.  I’ll list these from what I think is least important to most important.

#6: Better Script Handling

Internally, a JSScript represents (more or less) the code of a JS function, including things like the internal bytecode that SpiderMonkey generates for it.  The memory used by JSScripts is measured by the “gc-heap/scripts” and “script-data” entries in about:memory.

Luke Wagner did some measurements that showed that most (70–80%) JSScripts created in the browser are never run.  In hindsight, this isn’t so surprising — many websites load libraries like jQuery but only use a fraction of the functions in those libraries.  If SpiderMonkey could be changed to generate bytecode for scripts lazily, it could reduce “script-data” memory usage by 60–70%.  This would also allow the decompiler to be removed, which would be great.

Luke also proposed sharing immutable parts of scripts between web pages.  This would avoid a lot of duplication in the case where you have many tabs open with pages from a single site.

Both of these changes potentially will make the browser faster as well, because SpiderMonkey will spend less time compiling JavaScript source code to bytecode.

No-one is assigned to work on these bugs.  The lazy script creation can be done entirely within the JS engine;  the script sharing requires assistance from Necko.  Luke is currently busy with some other righteous refactorings, but I’m quietly hoping once they’re done he might find time for one or both of these bugs.

#5: Better Memory Reporting

Before you can reduce memory consumption you have to measure it.  about:memory is the critical tool that has facilitated much of MemShrink’s work.  (For example, we never would have known about zombie compartments without it.)  It’s in pretty good shape now but there are two major improvements that can be made.

First, the “heap-unclassified” number (a.k.a “dark matter”) is still typically around 20–25%.  My goal is to reduce that to 10%. This won’t require any great new insights, we already have the tools and data required.  Rather, it’s just a matter of grinding through the list of memory reporters that need to be added and improved.

Second, the resources used by each browser tab are reported in an unwieldy fashion:  JS memory on a per-compartment basis;  layout memory on a per-docshell basis;  DOM memory on a per-window basis.  Only a few internal architectural changes stand in the way of uniting these to provide the oft-requested feature of per-tab memory reporting.  This will be great for users, because if Firefox is using more memory than they’d like, it tells them which tabs they should close in order to free up memory.

I am actively working on both these improvements, and I’m hoping that within a couple of months they’ll be mostly done.

#4: Better Memory Consumption Tracking

One thing we haven’t done well in MemShrink is to improve the state of tracking Firefox’s memory consumption.  We have plenty of anecdotes but not much hard data about the improvements we’ve made, and we don’t have good ways to detect any regressions.  A couple of ideas haven’t gone very far, but some good news is that John Schoenick is making great progress on a proper implementation.  John has demonstrated preliminary versions of the site at two MemShrink meetings and it’s looking very promising.  It uses the endurance test framework to make the measurements, and opens lots of pages from the Talos tp5 pageset.

We also hope to use telemetry data to analyze how the memory consumption of each released version of Firefox stacks up.  That analysis would come with a significant delay — weeks or months after each release — but it would be much more comprehensive than any oft-run benchmark, coming from the real-world usage patterns of thousands of users.

#3: Compacting Generational GC

If you look in about:memory, JavaScript memory usage usually dominates.  In particular, the “js-gc-heap” is usually large.  There’s also the “js-gc-heap-unused-fraction” number, often 30% or higher, which tells you how much of that space is unused because of fragmentation.  That percentage overstates things somewhat, because often a good proportion of that unused space (see “js-gc-heap-decommitted”) is decommitted, which means that it’s costing nothing but address space… but that is cold comfort if you’re suffering out-of-memory aborts on Windows due to virtual memory exhaustion.

A compacting garbage collector is one that can move objects around the heap, filling up all those little gaps that constitute fragmentation.  The JS team (especially Bill McCloskey and Terrence Cole) is implementing a compacting generational garbage collector, which is a particular kind that tends to have good performance.  In particular, many objects die young and generational collectors find these quickly, which means that the heap will grow at a significantly slower rate than it currently does.  I could be wrong, but I’m convinced this will be a big win for both memory consumption and speed.

#2: Better Foreground Tab Image Handling

Images are stored in a compressed format (e.g. JPEG, PNG, GIF) on disk.  In order to display them, a browser must decompress (a.k.a decode) the compressed form into a raw pixel form that can easily be ten times larger.  This decoded form can be discarded and regenerated as necessary, and there are trade-offs to be made — for example, if you are too aggressive in discarding decoded images, you might have to decode them again, which will take CPU cycles and the user might see flickering if the decoding occurs in the visible part of the page.

However, Firefox goes way too far in the other direction.  If you open a page in the foreground tab, every single image in that page will be immediately decoded, and none of the decoded data will be discarded unless you switch away to another tab.  For pages that contain many images, this is a recipe for horrific memory consumption, and Firefox does much worse than all the other browsers.  So this is a problem that doesn’t rear its head for all users, but it’s terrible for those that are affected.

There are three MemShrink:P1 bugs relating to this:  one about not decoding all images immediately, one about discarding non-visible decoded images after some time, and one about some infrastructure work that is required for the first two.  As far as I know, no progress has been made on these three bugs, and although two of them are assigned they are not being actively worked on.

(See this discussion on the dev-platform mailing list for more details about this topic.)

#1: Better Detection and Notification of Leaky Add-ons

It’s been the case for several months that when a user complains about Firefox consuming an excessive amount of memory, it’s usually because of one or more add-ons, and the “can you try that again in safe mode?” / “oh yeah, that fixes it” dance is getting tiresome.

Many add-ons leak.  Even popular, well-written ones:  in the past few months leaks have been found in Adblock Plus, Video DownloadHelper, GreaseMonkey and Firebug.  That’s four of the top five add-ons on AMO!  We’re now getting several reports about leaky add-ons a week;  in this week’s MemShrink meeting there were four:  TorButton, NoSquint, Customize Your Web, and 1Password.  I strongly suspect the leaks we know about are just the tip of the iceberg.

Although leaks in add-ons are not Mozilla’s fault, they are Mozilla’s problem:  Firefox gets blamed for the sins of its add-ons.  And it’s not just memory consumption;  the story is the same for performance in general.  Here’s the quote of the week, from a user of 1Password:

I only use a handful of extensions and honestly never suspected 1P, however after disabling it I noticed my FireFox performance increased very noticibly. I’ve been running for 48 hours now without the 1P extension in Firefox and wow what a difference. Browsing is faster, switching is faster, memory usage is way down.

I’ve lost count of the number of stories like this that I’ve heard.  How many users have we lost to Chrome because of these issues, I wonder?

(And it’s not just leaks.  See this analysis of 16 add-ons and their effect on memory consumption when Firefox starts.)

One small step towards improving this situation was made this week:  Jorge Villalobos and Andrew Williamson added a “check for memory leaks” item to the AMO review checklist (under “Memory leaks from content”).  And Kris Maglione added some support for this checking in his Extension Test add-on.  This means that add-ons with obvious memory leaks (and many of them are obvious if you are actively looking for them) will not be accepted by AMO.

So that will prevents leaks in some new add-ons and new versions of established add-ons.  What about existing add-ons?  One idea is that AMO could also have a flag that indicates add-ons that have known memory problems (and other performance problems).  (This flag wouldn’t be an automatic thing, it would only be set once a leak has been confirmed, and after giving the author notification and some time to fix the problem.)  So that would also improve things a bit.

But lots of add-ons aren’t hosted on AMO.  Another idea is to have a stronger mechanism, one that informs the user if they have any add-ons installed that are known to cause high memory consumption (or other bad performance problems).  There is an existing mechanism for blocking add-ons that are known to be malware or exceptionally crashy, so hopefully the warnings could piggy-back on top of that.

Then, we need a better way to detect leaky add-ons.  Currently this is entirely done manually — and a couple of excellent contributors have found leaks on multiple add-ons — but I’m hoping that it’ll be possible to do a much more thorough job by analyzing telemetry data to find out which add-ons are correlated with high memory consumption.  That information could be used to trigger manual checking.

Finally, once you know an add-on leaks, it’s not always easy to work out why.  Tools could help a lot here, if they can be made to work well.


I listed six big areas for improvement. If we fixed all of these I think we’d be in a fantastic position.

Three of them (#5 better memory reporting, #4 better memory consumption tracking, #3 compacting generational GC) have people working on them and are in a good state.

Three of them (#6 better script handling, #2 better foreground image tab handling, #1 better detection and notification of leaky add-ons) don’t have people working on them, as far as I know.  If you are willing and have the skills to contribute to any of these areas, please contact me!

And if you think I’ve overestimated or underestimated the importance of any issue, I’d love to hear about it.  Thanks!

about:memory AdBlock Plus add-ons compartments DMD Firefox Garbage Collection JägerMonkey Massif Memory allocation Memory consumption MemShrink Performance SQLite Talks Valgrind

Notes on Reducing Firefox’s Memory Consumption

I gave a talk yesterday at the Browser MiniConf, held in Ballarat, Australia.  Its title was “Notes On Reducing Firefox’s Memory Consumption”.

Below are the slides and notes in a SlideShare embedding. If you find that embedding problematic (some people do) you may prefer to download the PDF version directly.

about:memory AdBlock Plus add-ons compartments Firefox Memory consumption MemShrink

MemShrink progress, week 30


This was the week of add-ons in MemShrink-land.

Jared Wein fixed a problem in Firefox that was causing zombie compartments if you viewed a native video with any add-on installed that implements the nsIContentPolicy interface.  Examples of such add-ons are Adblock Plus, GreaseMonkey, and NoScript, which are respectively the #1, #3 and #9 most popular add-ons on AMO!  Welcome to the MemShrink club, Jared.

Speaking of GreaseMonkey, Arantius fixed a bug in it that was causing zombie compartments on some GM scripts when opening a background tabs.

I can’t tell who was responsible for the next fix, because the Add-on SDK folks use Github in a way I don’t understand.   But Myk Melez and/or Gabor Krizsanits greatly reduced the number of compartments used in JetPack-style add-ons.  In one example, the number of compartments dropped from 156 to 8, saving about 20MB of memory.  This was a MemShrink:P1 bug.

Also, as far as I can tell, the same patch also fixed bug 680821 which means that all compartments that hold sandboxes created by JetPack-style add-ons will be marked as belonging to that add-on in about:memory.

Jordan Miner reported that the Delicious Bookmarks add-on is leaking excessive numbers of connections to the Places database.  I tried to reproduce the problem and failed, but I don’t have a Delicious account.  If anyone else who does have a Delicious account is able to reproduce, please let me know or comment in the bug.

Finally, Jorge Villalobos and Andrew Williamson from the add-ons team came to this week’s MemShrink meeting.  We had a very fruitful discussion about how to help add-on authors detect and avoid memory leaks, and how to help add-on users understand which add-ons have high memory consumption.  Stay tuned for more details in the coming weeks!

Three Cheers for New Contributors

Krzysztof Kotlenga, a new contributor, removed an OpenGL cache that was wasting 1GB+ of memory on Linux when hardware acceleration was enabled (it’s currently not enabled by default, but will be at some point in the future).   Great work, Krzysztof!  [Update: I originally wrote WebGL instead of OpenGL, which was incorrect.]

Bug Counts

Here are the current bug counts.

  • P1: 26 (-2/+0)
  • P2: 132 (-14/+3)
  • P3: 67 (-2/+5)
  • Unprioritized: 4 (-0/+4)

That’s a net reduction of six bugs.  The main factor here was that I went through some of our P2s and closed ones that were stale and/or covered by other bugs, and downgraded to P3 a few more that are now known to be less important than we first thought.

Before finishing, I’d like to highlight bug 703427.  Richard Hipp, one of the SQLite developers, has a tiny patch that drastically reduces the amount of memory used by SQLite in Firefox — I ran with it for a while and my SQLite memory consumption dropped from ~15MB to less than 3MB.  The patch also causes a moderate speed drop, but it’s unclear if that speed drop is noticeable to Firefox.  We need someone who understands Firefox’s SQLite usage well to evaluate this patch so that Richard knows if it’s something that should go into a released version of SQLite as an option.  The bug is assigned to Marco Bonardo but he’s currently very busy.  Is there anyway else who knows enough about SQLite to take on this bug?

add-ons Firefox MemShrink

MemShrink progress, weeks 28–29

It’s been a quiet couple of weeks, due to Christmas, New Year, and all that.

Two leaks in add-ons were fixed.

  • cyberscorpio fixed a zombie compartment in the Super Start add-on.  The problem was that the add-on was failing to remove an event observer when a page is unloaded.  The fix will be present in v3.6.2.
  • Jesse Hakanen fixed a zombie compartment in the AdBlock Plus Pop-up Addon add-on (note that this is add-on extends the blocking functionality of AdBlock Plus, it’s not AdBlock Plus itself).  The problem was that the add-on was failing to catch the page unload event, and so some references to closed pages were left in memory.  The fix is now available in v0.3.

Because both leaks had similar causes, I updated the documentation on avoiding zombie compartments in add-ons accordingly.

Patches for two other MemShrink bugs landed.

I saw two noteworthy mentions of Firefox’s recent reductions in memory consumption. The first was in’s review of Firefox 9.

Alongside the faster JavaScript processing Firefox 9 continues to show improvements from Mozilla’s MemShrink project, an ongoing effort to reduce memory usage in the browser. Indeed, for the first time in a very long time my testing showed Firefox 9 using less memory than Opera (which has long been the least RAM-hungry browser I test). Opening the same dozen tabs in both Firefox and Opera used only 367MB of RAM in Firefox compared to 378MB in Opera 11.60 [Update: Note that the memory test was performed with the following Firefox add-ons running: AdBlock, Ghostery, BetterPrivacy and HTTPS-Everywhere.] There’s no longer much difference between the two, which is a testament to Firefox’s dramatic improvement over the last six months of MemShrink efforts.

Cross-browser memory comparisons are fraught with difficulties, both in the measuring and the interpreting, so I don’t give the comparison with Opera much weight.  But the fact that this author has seen Firefox’s own memory consumption drop over the last few versions is meaningful.

The second was a comment on the previous MemShrink progress report from AC about Firefox 10.

You may remember that I posted a few weeks ago saying that Firefox 9 beta was great and that I could definitely see more memory and performance improvements over Fire 7 and 8. Well I am pleased to say that after a few days of using Firefox 10 beta 1, I can clearly see more memory improvements again. There is an obvious difference between Firefox 9 and Firefox 10 on the beta channel.

I have no hard evidence as such other than saying that nothing has changed on my laptop in the last few weeks but I am using the same heavy tab load and the memory usage displayed by Windows task manager is lower than it was by about 20 MB and the responsiveness and speed feels sharp.

Whatever you guys are doing with the Memshrink project, keep doing it because the results are paying dividends.

I’ve read literally 1000s of complaints about Firefox’s memory consumption online, so it’s very encouraging to hear compliments.

Here are the current bug counts.

  • P1: 28 (-0/+2)
  • P2: 143 (-1/+5)
  • P3: 64 (-1/+1)
  • Unprioritized: 0 (-0/+0)

No big changes, unsurprisingly.  [Update: the P2 numbers were incorrect and have been fixed.]