Category Archives: add-ons

MemShrink progress, week 105–108

This is the first of the every-four-weeks MemShrink reports that I’m now doing.  The 21 bugs fixed in the past four weeks include 11 leak fixes, which is great, but I won’t bother describing them individually.  Especially when I have several other particularly impressive fixes to describe…

Image Handling

Back in March I described how Timothy Nikkel had greatly improved Firefox’s handling of image-heavy pages.  Unfortunately, the fix had to be disabled in Firefox 22 and Firefox 23 because it caused jerky scrolling on pages with lots of small images, such as Pinterest.

Happily, Timothy has now fixed those problems, and so his previous change has been re-enabled in Firefox 24.  This takes a big chunk out of the #1 item on the MemShrink big ticket items list.  Fantastic news!

Lazy Bytecode Generation

Brian Hackett finished implementing lazy bytecode generation.  This change means that JavaScript functions don’t have bytecode generated for them until they run.  Because lots of websites use libraries like jQuery, in practice a lot of JS functions are never run, and we’ve found this can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.

Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

The completion of this finished off item #5 on the old Memshrink big ticket items list.  Great stuff.  This will be in Firefox 24.

Add-on Memory Reporting

Nils Maier implemented add-on memory reporting in about:memory.  Here’s some example output from my current session.

├───33,345,136 B (05.08%) -- add-ons
│   ├──18,818,336 B (02.87%) ++ {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
│   ├──11,830,424 B (01.80%) ++ {59c81df5-4b7a-477b-912d-4e0fdf64e5f2}
│   └───2,696,376 B (00.41%) ++ treestyletab@piro.sakura.ne.jp/js-non-window/zones/zone(0x7fbd7bf53800)

It’s obvious that Tree Style Tabs is taking up 2.7 MB.  What about the other two entries?  It’s not immediately obvious, but if I look in about:support at the “extensions” section I can see that they are AdBlock Plus and ChatZilla.

If you’re wondering why those add-ons are reported as hex strings, it’s due to a combination of the packaging of each individual add-on, and the fact that the memory reporting code is C++ and the add-on identification code is JS and there aren’t yet good APIs to communicate between the two.  (Yes, it’s not ideal and should be improved, but it’s a good start.)  Also, not all add-on memory is reported, just that in JS compartments;  old-style XUL add-ons in particular can have their memory consumption under-reported.

Despite the shortcomings, this is a big deal.  Users have been asking for this information for years, and we’ve finally got it.  (Admittedly, the fact that we’ve tamed add-on leaks makes it less important than it used to be, but it’s still cool.)  This will also be in Firefox 24.

b2g

Gregor Wagner has landed a nice collection of patches to help the Twitter and Notes+ apps on B2G.

While on the topic of B2G, in today’s MemShrink meeting we discussed the ongoing problem of slow memory leaks in the main B2G process.  Such leaks can cause the phone to crash or become flaky after its been running for hours or days or weeks, and they’re really painful to reproduce and diagnose.  Our partners are finding these leaks when doing multi-hour stress tests as part of their QA processes.  In contrast, Mozilla doesn’t really have any such testing, and as a result we are reacting, flat-footed, to external reports, rather than catching them early ourselves.  This is a big problem because users will rightly expect to have their phones run for weeks (or even months) without rebooting.

Those of us present at the meeting weren’t quite sure how we can improve our QA situation to look for these leaks.  I’d be interested to hear any suggestions.  Thanks!

MemShrink progress, week 73–74

B2G!  Fennec!  Social API!  They’re all happening, and memory consumption is a big deal for all of them.  Time for a MemShrink report.

B2G

Lots of B2G work is happening, unsurprisingly.  All of these changes have been backported to the Aurora channel.

Kyle Huey landed an important patch that merges system compartments together.  This avoids wasted space caused by having 100s of small compartments.  As Chris Jones said: “It’s a completely different phone with these patches.”  Unfortunately, the preference controlling this behaviour is currently turned off on B2G, because it’s causing some Marionette test failures.  Hopefully they’ll be dealt with soon.  Note that this preference won’t be turned on in desktop builds, and there’s a medium-term plan to avoid this wasted space in a less hacky fashion.

Marco Bonardo disabled Places in B2G, saving about 300 KiB in the main process.

Justin Lebar added code to trigger memory pressure events when lowmemkiller notifications occur.

Justin also tweaked things so that the garbage collector is more likely to run when screenshots are taken.

Jeff Muizelaar fixed a graphics bug that I don’t understand but apparently avoids allocating lots of memory on the B2G unlock screen.

Chris Jones ensured that remote content drop their buffers when they’re hidden.

I tweaked the size of the arena chunks used by XPT, which saved about 120 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I also shrunk the initial size of the SPS hash table, which saves 48 KiB per process on 64-bit builds and 24 KiB on 32-bit builds.

Memory Reporting

Lots of work happened on the memory reporting front.  This was inspired by  “heap-unclassified” typically being much higher on B2G than on desktop.  Desktop Firefox uses a single process and its memory consumption is usually measured in the 100s of MiBs or more.  In contrast, B2G uses one “main” process and then one process per running app, and the smaller ones are typically around 10 MiB.  As a result, the sundry small per-process things that don’t matter much for desktop loom larger on B2G.  (For the same reason, changes that reduce per-process memory by smallish amounts have outsize value on B2G.)  Most of the following changes have also been backported to the Aurora channel.

Nick Hurley added a “explicit/network/disk-cache” memory reporter.  It typically measures 100s of KiBs in desktop Firefox.  (It doesn’t get used on B2G.)

I added memory reporters “explicit/xpcom/component-manager” and “explicit/xpcom/category-manager”.  Together they measure about 280 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/script-namespace-manager”.  It measures just over 150 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/xpcom/effective-TLD-service”.  It measures about 128 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added some detail to the JS object memory reporters.  This gives slightly more insight into where this memory is going, but the extra memory measured as a result is usually pretty small.

I fixed the “explicit/atom-tables” memory reporter, which was erroneously always reporting 0 bytes.  It now measures anywhere from a few 100 KiBs to several MiBs of memory.  This was a frustrating bug to find.  DMD exists to check that memory reporters are measuring memory properly, and the reporter was doing its measurements just fine.  But DMD cannot check that the measured amounts are added correctly, and that’s what was going wrong here — a line that should have been this:

return n;

instead was this:

return 0;

In other words, the code I had written was doing the difficult part correctly, but botched the trivial part.  How annoying.

Social API

Felipe Gomes fixed a document leak that occurred when the social sidebar was hidden, and disappeared only when it was unhidden.

Felipe also added code to clear the previous profile when Social is toggled off, which prevents a leak.

The social API still has some significant unresolved memory consumption issues, which are a concern.

Fennec

Kartikaya Gupta (a.k.a. Kats) turned on tab expiration for Fennec.  What this means is that Fennec will now unload the contents of background tabs in certain circumstances — when memory is low, and in some cases when a tab hasn’t been viewed for over an hour — and then reload them when they are brought back into the foreground.  Read Kats’ blog post for more details.

(This change was prompted by Project 256meg, which aims to make Fennec usable on phones with only 256 MiB of RAM.  (Fennec currently requires 512 MiB, according to the official specs.)  Kats has just got to the point where Fennec can actually start on 256 MiB devices.  Although Fennec only uses a single process, there is obviously significant overlap between this goal and B2G’s memory reduction goals.)

An obvious follow-up idea is to add tab expiration to desktop Firefox.  But the bug tracking that idea has seen more heated discussion.  The reason is that tab expiration might cause data loss in some cases.  This is less of a problem on Fennec because (as far as I can tell) there is less of an expectation that in-flight data will be saved on mobile devices.  For example, having processes killed due to memory constraints on mobile is much more common than on desktop.  Still, I expect plenty more arguing before this issue is resolved one way or the other.  One option is to allow it on desktop but have it disabled by default.

Add-ons

Michel Gutierrez fixed a zombie compartment in Video DownloadHelper, which is the 2nd most popular add-on at AMO.   Version 4.9.11 has the fix.

We hit a notable milestone on the add-on front this fortnight:  on November 1st hit bug for tracking known leaks in add-ons — filed 16 months ago — had zero blockers.  In other words, we reached the point where no add-ons were known to leak!  Unfortunately this didn’t last long, and at the time of writing the tracking bug has three blocking bugs.  Still, it’s confirmation that what used to be our biggest memory consumption problem is well under control.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+5)
  • P2: 110 (-10/+7)
  • P3: 102 (-3/+8)
  • Unprioritized: 4 (-2/+4)

As always, the current bug lists can be found by following the links on the MemShrink wiki page.

MemShrink progress, week 71–72

B2G

Justin Lebar led the B2G charge this fortnight, doing the following.

Thinker Li identified that we were using overly large (128 KiB) arena chunks for type inference data in the JS engine.  Reducing them to 32 KiB saved over 3 MiB of memory on B2G.

Fabrice Desré disabled the add-on manager services, saving about 1 MiB.

These changes have all been backported to Aurora, because that’s the current home of the code that will be used for the V1 release of B2G.

Social API

Jared Wein modified the social sidebar so that it unloads itself 10 seconds after it’s been hidden.  This is good because lots of people thought that closing the sidebar was equivalent to disabling the social functionality, and with this change it now effectively is.

Felipe Gomes fixed a major problem which caused the social API to create and leak many unnecessary ports.

Gecko & SpiderMonkey

Benoit Jacob fixed a leak relating to WebGL and cycle collection.  This fixed 10 seconds pauses that occurred after running the RO.ME demo, and also fixed leaks seen in the WebGL version of Nokia Maps.

Kyle Huey fixed a bug that was causing storage code to leak one thread per connection.  This doesn’t affect most Firefox users much, because most connections live for the full lifetime of the process.  However, it could affect some add-ons, and it helped avoid some frequent OOM oranges in some of our tests.

Benjamin Peterson reduced the amount of memory used to compress JavaScript source code.

Add-ons

Zig fixed a zombie compartment in the Visited add-on.

Bug Counts

Here are the current bug counts.

  • P1: 18 (-4/+3)
  • P2: 113 (-2/+14)
  • P3: 97 (-1/+7)
  • Unprioritized: 2 (-13/+2)

Note:  in all my previous MemShrink reports, I mentioned bug fixes as soon as they landed on mozilla-inbound.  As of this week, I’m changing things so that I (mostly) only mention bug fixes that have landed on mozilla-central and thus been marked as RESOLVED.  This will make my life easier (tracking RESOLVED bugs is easy with Bugzilla search), and it will also mean that the changes in bug counts better match the descriptions of bug fixes.  But it means that some bug fixes will be reported two weeks later than they previously would.

MemShrink Progress, week 69–70

It’s been a bumper two weeks for MemShrink.

B2G

Gregor Wagner tweaked the JS garbage collection heuristics on B2G so that collections occur more frequently.  This prevents too much garbage from accruing, which is important on low-memory B2G devices.

Justin Lebar disabled sync on B2G, because it isn’t used.

Justin also implemented the ability to dump GC/CC logs when a particular signal is received, which is particularly useful for B2G.

B2G’s Operation Slim Fast is in full swing.  Please see the tracking bug for details about what is happening.  While there are a number of important fixes in the pipeline, memory consumption is still a critical issue for B2G simply because the devices don’t have much memory in the first place.  Any additional help from Mozilla developers will be greatly appreciated.  For example, if anyone can help identify unnecessary compartments, that would be helpful.

Memory Reporting

Boris Zbarsky improved the reporting coverage of style sheets.  We’ve seen this improved coverage account for over 1% of explicit on desktop and over 2% of explicit on B2G.

I fixed some double-counting of style sheets, and slightly improved the reporting coverage of style sheets and JS compartments.

Nathan Froyd added memory reporting for telemetry.

Justin Lebar fixed some over-reporting of the JS stack.  Justin also added to about:memory the ability to read JSON memory report data from the clipboard.

I renamed DMD as DMDV, in preparation for its in-progress, non-Valgrind replacement to take over the original name.

Testing With Valgrind

Gary Kwong has done some heroic work recently getting Valgrind tests passing on Tinderbox.  (Strictly speaking, the tests run Memcheck, which is the default Valgrind tool and is synonymous with Valgrind)  His blog post about it is well worth reading.

This is important for MemShrink because Memcheck can detect an important class of memory leaks — ones where all references to a block of memory are lost and so the block leaks forever.  But it’s also important for Firefox as a whole because Memcheck can detect a slew of other memory-related errors, such as writing unaddressable memory (e.g. buffer overruns) and using undefined values in dangerous ways.  These are the kinds of defects that cause crashes, security vulnerabilities, and other major problems.

The tests are currently hidden on TBPL because they only run on mozilla-central once a day, which is generally not enough for a test suite to be reliable enough to be worth unhiding.  Having said that, the ability to write suppressions for errors found with Memcheck complicates considerations;  see bug 800435 for further discussion.

Add-ons

Gregg Lind fixed things so that our Aurora/Beta users are certain to get an up-to-date, non-leaky version of the Test Pilot add-on.  (Version 1.1.2 was the leaky version;  version 1.2.0 or later has the fix.)  This was a longstanding MemShrink:P1 bug.

Erik Vold fixed a zombie compartment in Scriptish.  Version 0.1.8 has the fix.

Josep del Rio fixed a zombie compartment in Speed Dial.  Version 0.9.6.10 has the fix.

Miscellaneous

Boris Zbarsky fixed a bad zombie compartment leak relating to expandos.  This was a recent regression, and the fix is present on Aurora and Beta.  The problem was first reported in the comments of my MemShrink report from four weeks ago — many thanks to Alex for reporting it and preventing it from reaching an official Firefox release!

Bug Counts

Here are the current bug counts.

  • P1: 19 (-2/+5)
  • P2: 101 (-2/+6)
  • P3: 91 (-6/+0)
  • Unprioritized: 13 (-1/+11)

The unprioritized count is high because we spent the first part of today’s MemShrink meeting talking with Gavin Sharp, Jared Wein and Felipe Gomes about the upcoming social API, and then spent some additional time discussing B2G with David Clarke.  As a result we didn’t have time to triage the 23 new MemShrink bugs.  We’ve scheduled an out-of-band MemShrink meeting for next week to get on top of the bug list.

Visiting Mountain View

I’ll be in Mountain View all next week for the JS team work week.  I’ll be happy to meet up with anyone who wants to talk about memory consumption!

MemShrink progress, week 63–64

Add-ons

The big news this week was the release of Firefox 15, which prevents most add-on memory leaks.  As a consequence, we were able to close open bug reports for several add-ons that no longer leak:  Firebug (yay!), Enter Selects, LinkResolve, We-Care Reminder, Restartless Restart + SmartSearch (the leak happened only when both were installed).

Although Firefox 15 prevents most zombie compartment leaks, some remain.

  • Scriptish still leaks.  Scriptish is a GreaseMonkey fork.  Fortunately it has many fewer users than GreaseMonkey.
  • Speed Dial creates a zombie compartment when adding a new page to the speed dial.  This was only just reported, so we don’t yet know if it pre-dated Firefox 15 or not.

I haven’t heard much feedback about the release from actual users. This is probably a good thing, as users are much more likely to notice and report problems than improvements.  One thing that has cropped up is that a small number of add-ons have had their functionality adversely affected by the change, i.e. they were accessing compartments for a page after the page has been closed.

  • This caused certain GreaseMonkey scripts to leak badly.  This has now been fixed.
  • AutoPager’s main functionality is now broken.  On the upside, it no longer leaks.  AutoPager’s author hasn’t been responsive for the past five months, unfortunately.
  • TabMix Plus is popping up some error windows that it shouldn’t.
  • Two users with many add-ons installed have reported increased memory consumption.  The add-on(s) responsible haven’t been identified yet.

If you have any additional add-ons that you think have been adversely affected by Firefox 15, open the error console and use the “filter” search box in the top right-hand corner to look for a message that says “TypeError: can’t access dead object” — that’s the message that indicates an add-on is trying to access memory via a reference that Firefox 15 has pre-emptively cut.  Please report any problems in Bugzilla and put “[MemShrink]” in the whiteboard field.

On a related note, the discussion of Firefox 15′s release on various tech sites prompted me to debunk the common misconception that Mozilla claims “this is the release that fixes the memory leaks” on every release.

Other stuff

Andrew McCreight fixed a problem involving the cycle collector that could cause occasional leaks.

Brian Nicholson roughly halved the amount of memory used in Firefox Mobile’s page readability check.  (This wasn’t actually marked as a MemShrink bug;  I just chanced upon it today when looking at TBPL.)

Nick Cameron fixed a small, obscure leak.

Google Reader

For quite a while now we’ve had various people say that Google Reader creates lots of windows and/or compartments and causes Firefox’s memory consumption to gradually increase when it’s open for a long time.  This may only happen if you are signed into a Google+ account.  It feels like it’s a problem with Google Reader itself rather than Firefox, and a Google Reader engineer is CC’d on the bug report, but no progress has been made, and the reported symptoms are on the vague side.

If you have experienced this, we’d love to hear your feedback in the bug report.  In particular, as always, reliable steps to reproduce would make a big difference.

Bug Counts

Here are the current bug counts.

  • P1: 22 (-3/+1)
  • P2: 90 (-2/+3)
  • P3: 96 (-9/+7)
  • Unprioritized: 2 (-2/+2)

Forward progress!  Good to see.

Update to GreaseMonkey 1.0 to avoid leaks in Firefox 15

Firefox 15 will be released tomorrow, and it features a fantastic change that prevents most memory leaks in add-ons, which have been a big problem for a long time.

However, recently we discovered that the change is causing some GreaseMonkey scripts to leak badly.  The scripts we know that leak are “YousableTubeFix” and “Textarea backup with expiry”, but others might have the same problem.  Fortunately, Anthony Lieuallen and Kyle Huey worked together to fix the problem, and version 1.0 of GreaseMonkey contains the fix.  If you have GreaseMonkey installed you can open the Add-ons Manager to see which version you have.  If it’s older than 1.0 (e.g. 0.9.22), you can update by visiting the GreaseMonkey page and clicking on the big green “Add to Firefox” button.

This is the only add-on we know that is adversely affected by the change in Firefox 15.  Hopefully we won’t find any more!

MemShrink progress, week 61–62

It’s been a quiet fortnight for MemShrink.

The biggest news is that Kyle Huey made a change so that Firefox discards images that have been removed from the DOM.  This might sound boring, but it’s important because (among other things) it can greatly reduce the amount of memory used for photo slideshows on Facebook.  Kyle first started working on this patch nearly a year ago, and had to overcome numerous hurdles along the way.  This is a nice improvement to our foreground tab image handling, which is the #1 remaining MemShrink problem.  But there’s still lots of room to improve on that front.

The following add-ons had memory leaks fixed: 1Password, Web Developer, CoKnown Research & Webpage Clipping Toolbar, aaQQin.  The latter three were zombie compartments, a problem that Firefox 15 (due for release next week) should make impossible.  Still, it was nice of the authors to fix the problem early rather than waiting for it to be fixed for them!

Gian-Carlo Pascutto finished a huge overhaul of the SafeBrowsing implementation.  I won’t pretend to understand the changes, but it was marked as a MemShrink:P2 change and so hopefully reduced memory consumption in some notable fashion :)

Here are the current bug counts.

  • P1: 24 (-0/+1)
  • P2: 89 (-3/+3)
  • P3: 98 (-8/+1)
  • Unprioritized: 2 (-1/+2)

A small reduction.

MemShrink progress, week 57–60

I’ve been on vacation for two weeks, so this report covers the past four weeks.  I still have a mountain of unread emails, bug reports and blog posts to get through, but hopefully I haven’t missed much.

Hueyfix

I wrote about Firefox 15 and how it contains the “Hueyfix” that prevents most memory leaks caused by add-ons.  It got lots of attention.

Relatedly, a regression caused by the Hueyfix was discovered:  it caused some Greasemonkey scripts to leak badly, such as “YousableTubeFix” and “Textarea backup with expiry”.  A change was made to Firefox to allow this to be prevented (which has been backported to Firefox 15 Beta), and Greasemonkey 0.9.22 should fix the problem, although that hasn’t yet been confirmed.  If you have had problems with Firefox 15 and you are using Greasemonkey, this is a likely cause.  I’m not aware of any other regressions caused by the Hueyfix.

Also relatedly, Gabor Krizsanits made a change inspired by the Hueyfix that cuts references to sandboxes once they should no longer be used.  This prevents a whole class of possible leaks in add-ons using the Add-on SDK.

Miscellaneous

Gregor Wagner tweaked the JS engine’s garbage collection heuristics to greatly reduce the peak memory consumption in certain cases.  Gregor previously tweaked the GC heuristics in Firefox 7 with great success.

Tim Taubert fixed a leak caused by opening and closing the bookmarks sidebar.  This was a precursor to a more important change from Tim, which adds automated checking of the cycle collection logs to check for leaks during testing.  This should allow a whole class of memory leaks to be detected automatically, thus preventing regressions.

Benoit Jacob made a change that prevents the number of WebGL contexts growing excessively.  This can greatly improve memory consumption on certain sites that use WebGL.

Nick Cameron fixed a problem that caused huge memory consumption (multiple GB) on pages using canvas in conjunction with certain Intel graphics cards and/or drivers.  I don’t understand the details, but it sounds like some cards and/or drivers might still have problems.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-0/+3)
  • P2: 89 (-9/+8)
  • P3: 105 (-2/+5)
  • Unprioritized: 1 (-2/+1)

Nothing spectacular there.

Firefox 15 plugs the add-on leaks

TL;DR

Firefox 15 prevents most memory leaks caused by add-ons, including Firebug.  For many users with add-ons installed this will significantly reduce Firefox’s memory consumption, without requiring upgrades to those add-ons.

For those users, Firefox 15 is likely to be faster (sometimes drastically so) and less likely to crash, especially if they have multiple add-ons installed and/or keep Firefox running for a long time between restarts.

The Problem

A memory leak occurs when a program allocates some memory but then fails to free it.  There are multiple causes of memory leaks, but they all have the same effect:  an increase in memory consumption, which often grows as the program runs. This can cause poor performance, and even lead to crashes or aborts due to the exhaustion of available memory.

Many Add-ons Leak

I’ve lost count of the number of times I’ve seen (and participated in) variants of the following conversation during the past year.

Alice: Firefox is slow and using 2GB of memory.  WTF?
Bob:   Do you have any add-ons installed?
Alice: Yes.  Why?
Bob:   Try disabling them all and restart Firefox.
Alice: Ok... Oh wow.  That's much better.
Bob:   Now you have to disable them one at a time to work
       out which one(s) are at fault.
Alice: Really?  Hmm, that sucks.
Bob:   Yeah.  Sorry about that.

Although Firefox itself isn’t immune to memory consumption problems, Mozilla’s MemShrink project has fixed enough problems in Firefox itself that in January of this year I declared that the #1 way to improve memory consumption in Firefox would be with “Better Detection and Notification of Leaky Add-ons“.

I said this because many add-ons have memory leaks.

Happily, most of the leaks in the mentioned add-ons have been fixed by their developers.  But by now it should be clear that (a) memory leaks are common, and (b) no add-ons are immune, no matter how simple they are, or how good their developers are.

The other thing to note is that these add-ons each exhibited a particular kind of leak that we call a zombie compartment.  The gory details of zombie compartments are beyond the scope of this blog post;  the important thing is that they are common, accounting for 90% or more of the add-on leaks we’ve seen.

Leaky Add-ons Make Firefox Slow and Crashy

So, leaky add-ons are common.  What effect do they have?

Not all leaks are created equal.  Some add-ons leak a small amount of memory; some add-ons leak an unbounded amount of memory.  Users with multiple add-ons are more likely to suffer problems. Certain browsing patterns may exacerbate leaks.  Overall, it’s very hard to characterize.

Nonetheless, affected users may experience high memory consumption, which has the following visible effects.

  • Long freezes, caused by garbage collection and cycle collection.
  • General sluggishness during basic operations, like scrolling and switching tabs.
  • Occasional drastic slowdowns — particularly on machines with small amounts of RAM such as netbooks — due to the paging of data to and from the hard disk.
  • Out-of-memory crashes or aborts, particularly on Windows and ARM machines.

For example, one user who disabled a leaky version of 1Password experienced a much leaner and faster Firefox as a result.

I only use a handful of extensions and honestly never suspected 1P, however after disabling it I noticed my FireFox performance increased very noticibly. I’ve been running for 48 hours now without the 1P extension in Firefox and wow what a difference. Browsing is faster, switching is faster, memory usage is way down.

(My emphasis.)

We’ve heard this kind of story many times, and this one is typical because the user never suspected that an add-on was the cause of the problem.  Leaks in add-ons are not Mozilla’s fault, but they are Mozilla’s problem — Firefox gets blamed for the sins of its add-ons.  This isn’t surprising, because Firefox hasn’t provided tools that would let users understand when add-ons are causing problems.

What has Mozilla done about this before now?

For a long time, we lacked memory profiling tools that were (a) powerful enough to detect add-on leaks, and (b) simple enough to be used by users who weren’t also Firefox developers.  This meant that we didn’t appreciate how many problems were caused by leaky add-ons.

That changed last year.  The infrastructure with which we detect zombie compartments was implemented in two parts:  the segmentation of the JavaScript heap into compartments (which happened in Firefox 4, released in March 2011), and the addition of per-compartment memory reporting in Firefox’s about:memory page (which was added in Firefox 7, released in September 2011).

Once these tools were in place, the reports of leaky add-ons started rolling in, and it only took a few months to realize the magnitude of the problems they were causing.  We started tracking them, we helped authors fix them, and tightened the AMO review guidelines so that any add-on that was found to cause zombie compartments would have its status downgraded.

As a result, dozens of leaky add-ons were fixed.  But there are literally thousands of Firefox add-ons in existence, many of which aren’t hosted on AMO and so aren’t subject to the AMO review policies.  And although many add-on authors were responsive, some weren’t.  It was a tedious game of whack-a-mole, and something better was needed.

The Fix

Leaky add-ons usually cause zombie compartments by holding references to DOM structures within a web page even after the page has been closed or navigated away from.  This prevents Firefox from garbage-collecting the page’s compartment.

Mozilla engineer Kyle Huey had the radical idea of simply detecting these kinds of references and cutting them.  This would eradicate this entire class of memory leaks for good, without requiring any changes to individual add-ons.

However, it wasn’t at all obvious that this would work.  In particular, it was possible that code within Firefox and/or add-ons might be broken by this change.  But it turned out, once some bumps were cleared, that almost no code was broken and the leaks were prevented exactly as predicted.  This fix is present in Firefox 15.

Confirmation via in-house testing

I did some early testing on a number of add-ons that were known to leak:  Firebug 1.9.1, McAfee SiteAdvisor 3.4.1, PicPasso 1.0.1, LoL 1.5, YouTube Ratings Preview 1.03, Enter Selects 7, HttpFox 0.8.10, and Link Resolve 0.7.  In every case, Firefox 15 prevents the leaks.

We’ve subsequently confirmed that a number of other leaking add-ons have been fixed by this change.  Although add-ons can still cause zombie compartments in obscure cases, we are yet to encounter such a case in practice.

Confirmation from real-world users

Firefox users can enable telemetry, which is a feature that periodically sends anonymous performance-related data to Mozilla.  This allows us to analyze how people use Firefox in the real world.

We don’t track zombie compartments via telemetry, but we do track something called ghost windows.  The gory details of ghost windows are also beyond the scope of this blog post;  the important thing to know is that they correlate strongly with zombie compartments — a high ghost window count very likely indicates the presence of zombie compartment, and thus memory leaks.

The following graph shows the ghost windows telemetry measurements for users of Firefox Nightly builds on Windows during April and May 2012.  The line marked with a 1 shows when Kyle’s patch landed, and the lines marked with 2 and 3 show when some related follow-up patches landed.

Ghost Windows telemetry results

Telemetry data tends to be very noisy, and this graph is no exception, but the effect is clear.  The mean number of ghost windows (shown by the blue dots) dropped from typically between three and four to typically less than one.  (As for why it’s not zero, one reason is that ghost windows can be false positives — i.e. they don’t always correspond to actual leaks.)  Furthermore, the variance has dropped greatly.  Previously, the hardest-hit 5% of measurements would typically have at least 15 or 20 ghost windows, and the next hardest-hit 20% would typically have at least two.  After the fix, the hardest-hit 5% of measurements typically have only two or more ghost windows, and the remaining 95% typically have zero.  This graph is a few weeks old now, but the measurements for Firefox 15 since then have remained steady.

In other words, ghost windows have changed from something that many users experience to something that few users experience.  Furthermore, those users who are still affected experience many fewer than before.  This graph also provides further evidence that add-ons are responsible for a large fraction of the memory leaks encountered by Firefox users.

The Benefits of the Fix

Our testing

I compared pre-release versions of Firefox 14 and Firefox 15 with McAfee SiteAdvisor 3.4.1 installed.  SiteAdvisor is a very popular add-on that rates the safety of web sites, and version 3.4.1 had an unbounded memory leak that leaked significant amounts of memory on every page that Firefox visited.  (That leak has since been fixed and version 3.4.1 is no longer available from McAfee.)

I did the testing on a fast desktop machine with Windows 7 installed.  I ran MemBench, which is a memory test benchmark that opens 150 popular websites, one per tab, and then closes them all.  This test is a reasonable facsimile of several hours of web browsing.

I used about:memory to measure memory consumption (a) with the 150 tabs open (plus one more tab for about:memory itself), and then (b) after closing those 150 tabs.  The metric I measured was resident memory consumption, which is the amount of physical memory used by Firefox.  It is measured by the operating system.  Firefox 14′s two measurements are on the left, and Firefox 15′s two measurements are on the right.

Resident Memory Consumption (MB) on MemBench of Firefox 14 & 15 with SiteAdvisor 3.4.1 add-on

The graph tells a clear story.  The leak in SiteAdvisor 3.4.1 prevents Firefox 14 from releasing most of the memory used by the 150 MemBench tabs, even after those tabs are closed.  In contrast, Firefox 15 prevents the leak.  As a result, after closing 150 tabs, Firefox 15′s resident memory consumption is 4.8x smaller than Firefox 14′s (compare the red bars) without any change to the add-on.

Feedback from users

The feedback from users of the pre-release Nightly and Aurora versions of Firefox has been overwhelmingly positive.  Here are some notable examples.

One user experienced a 4x reduction in memory consumption at the end of the day.

I seem to be seeing really significant improvements from this. Normally Fx would start at 170MB and by the end of the day that would often be creeping up to 800 or 900 MB. Today using latest nightly I again started at 170, but now at the end of the day I’m only at 230MB!

I knew that my addons were the culprit, yet the list I have at the moment is a tenured list of addons I wish to keep. This fix allows me to continue being an addon junkie without the pernicious side effects. Thank you so much!

Another user reported via email that a 1.5GB per hour leak — which caused 10+ second pauses and necessitated frequent restarts — disappeared.

Firefox was leaking about 1.5GB per hour for me. It started with Firefox 3. I tracked it down to Ghostery and NoScript, but even without those addons it leaked about 500MB per hour of browsing.

GC and CC times got up into the 10 second range. Ugly. Really really ugly! And this is on top of the line massively overclocked hardware, too. I had to install a new addon to add a restart button to Firefox, because Firefox froze solid after hitting 2GB of memory usage…

Then your patch comes along and solves it all… totally awesome!

Another user experienced faster painting, scrolling, and input operations.

Certainly, before this fix I would find that Firefox often became sluggish (input lag, slow paint operations, less than silky smooth scroll animations) as the memory usage built up… I haven’t noticed the same symptoms since… the signs certainly seem to be good.

These quotes nicely highlight the potential benefits:  greatly reduced memory consumption, which leads to fewer pauses, faster overall performance, and no need to restart Firefox to restore optimum performance.

YMMV

Based on the evidence above, we are confident that Firefox 15 fixes the vast majority of add-on memory leaks, and that as a result, many users will see drastic improvements in Firefox’s performance and stability.  However, predicting the exact effects of this fix on the hundreds of millions of Firefox users is essentially impossible, due to the huge number of different add-ons that are available, and the great variety of add-on and browsing usage patterns.

Nonetheless, if you are a heavy add-on user and you have noticed Firefox’s performance degrade during long browsing sessions, there is a good chance you will see large improvements.  Firebug users may notice improvements as well.  But, as they say:  your mileage may vary.

CONCLUSION

Over the past year, Mozilla has made great progress in reducing Firefox’s memory consumption.  However, the excessive memory consumption caused by add-ons with memory leaks has remained an ongoing problem.

Firefox 15 fixes that problem.  We have confirmed, via in-house testing and from real-world telemetry data, that it prevents the vast majority of leaks that occur in existing add-ons.

Users who upgrade to Firefox 15 won’t have to upgrade their add-ons to see the benefits.  While it is hard to predict the effect of this improvement on any individual user, many users should experience greatly reduced memory consumption, particularly on long browsing sessions.  This should result in better performance, fewer pauses, and fewer crashes.

Mozilla’s MemShrink efforts are ongoing.  We have various projects in the pipeline that aim to further reduce Firefox’s memory consumption, and help users understand better how Firefox is using memory.  Hopefully these changes will make it into Firefox releases not far from now!

MemShrink progress, week 55-56

Memory Reporting

I changed things so that JavaScript memory consumption for web content is reported on a per-tab basis, as the following example shows.

│  ├────4,472,568 B (00.50%) -- top(http://www.mozilla.org/en-US/firefox/fx/, id=236)/active
│  │    ├──4,192,640 B (00.47%) -- window(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,979,112 B (00.22%) ++ js/compartment(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,607,216 B (00.18%) ++ layout
│  │    │  ├────352,520 B (00.04%) ── style-sheets
│  │    │  ├────253,312 B (00.03%) ++ dom
│  │    │  └────────480 B (00.00%) ── property-tables

All the major components of each tab’s memory consumption — JS, layout, style sheets, and DOM — are now reported on a per-tab basis.  This is great, because it’s something that people have been requesting for  years.  However, the presentation is still intimidating.  Consider the following excerpt from about:memory.

├───18,288,696 B (16.43%) -- window-objects
│   ├───6,263,208 B (05.62%) ++ top(about:memory?verbose, id=6)/active
│   ├───5,090,352 B (04.57%) ++ top(chrome://browser/content/browser.xul, id=1)/active
│   ├───3,422,280 B (03.07%) ++ top(http://www.mozilla.org/en-US/firefox/fx/, id=17)/active/window(http://www.mozilla.org/en-US/firefox/fx/)
│   ├───3,178,432 B (02.85%) ++ top(chrome://global/content/console.xul, id=21)/active
│   └─────334,424 B (00.30%) ++ top(resource://gre-resources/hiddenWindow.html, id=3)/active/window(resource://gre-resources/hiddenWindow.html)

Among those top(...) entries are two browser tabs (one for www.mozilla.org, one for about:memory), two browser chrome windows (browser.xul is the main browser window, and console.xul is the error console window), and the hidden window which is always present.  Firefox developers can easily understand this, and technically-inclined users may be able to guess what’s going on, but the average user will struggle.

From a user’s point of view, the visible things they have control over (i.e. windows and tabs) should be better distinguished from everything else, and be shown with nicer names than “browser.xul”.  But the architecture of Firefox’s internals doesn’t make this easy.  I’ve made one not-very-successful attempt to improve this, and I’d love to hear suggestions from other Firefox devs on how to present this information better.

(The step after that one is to present some kind of “tab manager” — a massively stripped-down variant of about:memory that ordinary users can understand.  This would also require localization for all the languages that Firefox supports.  One step at a time!)

Finally, another memory reporting improvement happened:  Andrew McCreight and I added measurement of orphan DOM nodes.  These are DOM nodes that have been discarded by a page but are still accessible from JavaScript objects.   They were the biggest single remaining contributor to “heap-unclassified”.  For example, when I hit the “update” button on about:memory I get 5MB of them (don’t worry, they go away when the garbage collector runs).  Also, badly coded sites can create large numbers of them.  So it’s a good thing to be able to measure, for both users and web developers.

Miscellaneous

Mihai Sucan fixed a bad leak involving pages that use window.console.  Multiple people reported that this caused Firefox’s memory consumption to balloon to multiple GB.  This leak was introduced during the Firefox 15 development cycle, and the fix has been backported to the Aurora channel, which is where Firefox 15 currently resides.  One notable thing about this fix is that the problem was quite easy to diagnose because about:memory clearly indicated that the ConsoleAPI.js compartment was responsible for most of the memory consumption.  Without the fine-grained memory reporting that was enabled by compartment-per-global, that memory would have been reported in a generic “System Principal” compartment which would have made diagnosis much harder.

Wladimir Palant fixed an issue in AdBlock Plus that causes its memory consumption to slowly grow when viewing sites that repeatedly reset the src property of images, such as Mibbit.  (The fine-grained memory reporting enabled by compartment-per-global also made this diagnosis much easier.)  This fix is in Adblock Plus 2.1.

Alexandre Poirot fixed a minor leak in the Add-on SDK.

I wrote about the difficulties of cross-browser memory comparisons.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+0)
  • P2: 90 (-3/+6)
  • P3: 102 (-6/+2)
  • Unprioritized: 2 (-0/+0)

A net reduction of seven bugs!

Update on The big ticket Items

In January, I described six “big ticket items” that needed tackling to improve Firefox’s memory consumption.  Let’s look at how they’ve progressed since then.

#6: Better Script Handling

This had two parts.  The first part was to only generate bytecode for a function once it is run.  I’ve made a good chunk of progress towards and have it working on very simple examples.  This has required a great deal of refactoring of the SpiderMonkey front-end.  (The front-end is a hairy piece of code, so this is virtuous in and of itself.)  There are still a couple of other bugs blocking it from further progress.  I intend to return to continue on this once they are done.

The second part was to share immutable parts of scripts between web pages.  No progress has been made on this.

So, overall, this item is about 25% done, and progress is still ticking along slowly.

#5: Better Memory Reporting

This one also had two parts.  The first part was to reduce about:memory’s “heap-unclassified” number (a.k.a “dark matter”) to typically 10%.  People such as Nathan Froyd and I have made good progress on this.  It’s now occasionally below 10% (my current session shows 7.5%) but often is around 15%.  That’s much better than the 20–25% we typically were getting at the start of the year, and it’s low enough that people rarely complain about it now.  And we’re very much into the long tail now, so it’ll be hard to improve things much further.  For the most part I don’t think we need to.

The second part was to report memory in a per-tab fashion.  As mentioned above, I just completed this, although the presentation is not as user-friendly as it could be.

The per-tab reporting was enabled by compartment-per-global.  Compartment-per-global also gives us much more insight into the memory consumption of Firefox’s own JavaScript code, and even some idea about add-on memory consumption, which was (to me) an unexpected bonus.  Indeed, its usefulness was demonstrated by the two leak fixes mentioned above.

So, overall, this item is about 80% done.  Although there is still some work to be done, it has progressed enough that I will remove it from the “big ticket items” list.

#4: Better Memory Consumption Tracking

We can cross this item off the list, thanks to areweslimyet.com.  In January I also mentioned the idea of using telemetry for this, but we’ve found that telemetry results are so noisy that little (with rare exceptions) can be gleaned from it, so there’s not really anything to be done on that front.

#3: Compacting Generational GC

This is one of the two key SpiderMonkey performance features under development (the other being IonMonkey), and so people such as Terrence Cole, Steve Fink, Brian Hackett, and Bill McCloskey and actively working on it.  Unfortunately, it’s a huge undertaking that requires rewriting many of SpiderMonkey’s internal and external APIs.  If I had to guess I’d say it’s 25% complete, but that really is just a wild guess.  I’ll be pleased if it is finished before the end of the year.

#2: Better Foreground Tab Image Handling

This is the big ticket item for which the least progress has been made. There’s a single bug that is blocking all the follow-on work.  Both Timothy Nikkel and Jet Villegas have spent time working on it, but it has been a difficult nut to crack.  Justin Lebar had a recent suggestion for narrowing its scope slightly.  I hope it’s a good suggestion, because I’m out of ideas on this one.

#1: Better Detection and Notification of Leaky Add-ons

This is the most satisfying item of the lot.  I expected to make only grudging, piecemeal progress, but Kyle Huey’s fix to prevent zombie compartments prevents the vast majority of add-on leaks, enough to declare victory and cross this item off the list.  I’ll be writing more about this next week.

Summary

Three items (#1, #4, #5) have progressed enough that they can be removed from the list.  Only one of these directly improved Firefox’s memory consumption;  the other two were about better measuring and monitoring of memory consumption.

The remaining items are about reducing memory consumption.  Two of them (#3, #6) have a long way to go, but progress has been made and is ongoing.  One item (#2) has made little progress and has stalled.

The New big ticket Items

Here’s my updated list of the most important things.

#5: Better Script Handling

As mentioned above, this has two parts: lazy bytecode generation, and the  sharing of immutable parts of scripts.  I intend to continue working on the first part.  Once that’s done, it may turn out that the second part isn’t necessary;  we’ll see.

#4: Regain compartment-per-global losses

The measurements on areweslimyet.com have been creeping up for the past two months.  The major cause is compartment-per-global, which has lots of benefits (as mentioned above) but also increases memory consumption by a small-to-moderate amount across the board.  This is because there are many more compartments than there used to be, and there’s a certain amount of overhead and waste in each compartment.

Generational GC will help, but there are some other things that can be done more quickly that will help.  See the tracking bug for one example.

#3: Boot2Gecko

Boot2Gecko (now officially known as “Firefox OS”) is going to be running on low-end phones without much physical memory, so that’ll require care.  The first step is to get about:memory working on B2G.  The next step is to find a way to copy the data from about:memory somewhere else, because B2G doesn’t have cut and paste!  Beyond that, it’s unclear;  but once about:memory is working it’s very likely it will identify some sub-optimal B2G-specific behaviour.

#2: Compacting Generational GC

Let me quote myself to explain the memory consumption benefits of compacting, generational GC.

Virtual memory consumption drops in two ways.  First, the compaction minimizes waste due to fragmentation.  Second, the heap grows more slowly.  Physical memory consumption drops for the same two reasons.

The performance benefits are also significant.

Performance improves for three reasons.  First, paging is reduced because of the generational behaviour:  much of the JS engine activity occurs in the nursery, which is small;  in other words, the memory activity is concentrated within a smaller part of the working set.  Second, paging is further reduced because of the compaction: this reduces fragmentation within pages in the tenured heap, reducing the total working set size.  Third, the tenured heap grows more slowly because of the generational behaviour… which means that structure traversals done by the garbage collector (during full-heap collections) and cycle collector are faster.

Important stuff.

#1: Better Foreground Tab Image Handling

On image-heavy pages, Firefox uses much more memory than other browsers.  This includes things like Facebook image slideshows.  One commenter said the following.

On behalf of Facebook Inc, let me just say that we are prepared to give over one thousand free “pokes” to anyone who fixes this bug or alternatively https://bugzilla.mozilla.org/show_bug.cgi?id=683290. I will also gladly show you Mark Zuckerberg’s desk and give you free dinner and cookies as part of a complementary tour of Facebook HQ in Menlo Park.

I have no idea if this person is really from Facebook, but it’s certainly suggestive.  Also, I’ve heard from B2G people that they are having to work around this problem in their Gallery app.

To repeat myself from week 32:  there are three MemShrink:P1 bugs relating to this:  one about not decoding all images immediately, one about discarding non-visible decoded images after some time, and one about some infrastructure work that is required for the first two.  (See this discussion on the dev-platform mailing list for more details about this topic.)

Summary

There are three repeat items, and two new items, but there are fewer items than last time;  this reflects the fact that we are in a better position than we were in January.  If you think I’ve overestimated or underestimated the importance of any of these, or omitted anything, I’d be interested to hear.

One final thing:  last time, several people suggested “better reporting of add-on memory consumption”.  If I knew of any ideas on how to do this better than about:memory currently does, I’d gladly put it on the list.  But I don’t, and there doesn’t seem to be much point to put an item on the list that we don’t know how to implement.

Vacation

I’m taking the next two weeks as vacation, so there won’t be a MemShrink report on July 24.  I’ll be back with the next MemShrink round-up on August 8th.  See you then.