Firefox 15 plugs the add-on leaks

TL;DR

Firefox 15 prevents most memory leaks caused by add-ons, including Firebug.  For many users with add-ons installed this will significantly reduce Firefox’s memory consumption, without requiring upgrades to those add-ons.

For those users, Firefox 15 is likely to be faster (sometimes drastically so) and less likely to crash, especially if they have multiple add-ons installed and/or keep Firefox running for a long time between restarts.

The Problem

A memory leak occurs when a program allocates some memory but then fails to free it.  There are multiple causes of memory leaks, but they all have the same effect:  an increase in memory consumption, which often grows as the program runs. This can cause poor performance, and even lead to crashes or aborts due to the exhaustion of available memory.

Many Add-ons Leak

I’ve lost count of the number of times I’ve seen (and participated in) variants of the following conversation during the past year.

Alice: Firefox is slow and using 2GB of memory.  WTF?
Bob:   Do you have any add-ons installed?
Alice: Yes.  Why?
Bob:   Try disabling them all and restart Firefox.
Alice: Ok... Oh wow.  That's much better.
Bob:   Now you have to disable them one at a time to work
       out which one(s) are at fault.
Alice: Really?  Hmm, that sucks.
Bob:   Yeah.  Sorry about that.

Although Firefox itself isn’t immune to memory consumption problems, Mozilla’s MemShrink project has fixed enough problems in Firefox itself that in January of this year I declared that the #1 way to improve memory consumption in Firefox would be with “Better Detection and Notification of Leaky Add-ons“.

I said this because many add-ons have memory leaks.

• We have found leaks in popular, well-written add-ons, such as Adblock Plus, Video DownloadHelper, GreaseMonkey and Firebug.  (They are the top four add-ons on addons.mozilla.org!)
• We have found leaks in simple, innocuous add-ons, such as It’s All Text and Image Zoom.
• We have found leaks in add-ons written by large software companies, such as McAfee SiteAdvisor and Yahoo! Toolbar.

Happily, most of the leaks in the mentioned add-ons have been fixed by their developers.  But by now it should be clear that (a) memory leaks are common, and (b) no add-ons are immune, no matter how simple they are, or how good their developers are.

The other thing to note is that these add-ons each exhibited a particular kind of leak that we call a zombie compartment.  The gory details of zombie compartments are beyond the scope of this blog post;  the important thing is that they are common, accounting for 90% or more of the add-on leaks we’ve seen.

Leaky Add-ons Make Firefox Slow and Crashy

So, leaky add-ons are common.  What effect do they have?

Not all leaks are created equal.  Some add-ons leak a small amount of memory; some add-ons leak an unbounded amount of memory.  Users with multiple add-ons are more likely to suffer problems. Certain browsing patterns may exacerbate leaks.  Overall, it’s very hard to characterize.

Nonetheless, affected users may experience high memory consumption, which has the following visible effects.

• Long freezes, caused by garbage collection and cycle collection.
• General sluggishness during basic operations, like scrolling and switching tabs.
• Occasional drastic slowdowns — particularly on machines with small amounts of RAM such as netbooks — due to the paging of data to and from the hard disk.
• Out-of-memory crashes or aborts, particularly on Windows and ARM machines.

For example, one user who disabled a leaky version of 1Password experienced a much leaner and faster Firefox as a result.

I only use a handful of extensions and honestly never suspected 1P, however after disabling it I noticed my FireFox performance increased very noticibly. I’ve been running for 48 hours now without the 1P extension in Firefox and wow what a difference. Browsing is faster, switching is faster, memory usage is way down.

(My emphasis.)

We’ve heard this kind of story many times, and this one is typical because the user never suspected that an add-on was the cause of the problem.  Leaks in add-ons are not Mozilla’s fault, but they are Mozilla’s problem — Firefox gets blamed for the sins of its add-ons.  This isn’t surprising, because Firefox hasn’t provided tools that would let users understand when add-ons are causing problems.

What has Mozilla done about this before now?

For a long time, we lacked memory profiling tools that were (a) powerful enough to detect add-on leaks, and (b) simple enough to be used by users who weren’t also Firefox developers.  This meant that we didn’t appreciate how many problems were caused by leaky add-ons.

That changed last year.  The infrastructure with which we detect zombie compartments was implemented in two parts:  the segmentation of the JavaScript heap into compartments (which happened in Firefox 4, released in March 2011), and the addition of per-compartment memory reporting in Firefox’s about:memory page (which was added in Firefox 7, released in September 2011).

Once these tools were in place, the reports of leaky add-ons started rolling in, and it only took a few months to realize the magnitude of the problems they were causing.  We started tracking them, we helped authors fix them, and tightened the AMO review guidelines so that any add-on that was found to cause zombie compartments would have its status downgraded.

As a result, dozens of leaky add-ons were fixed.  But there are literally thousands of Firefox add-ons in existence, many of which aren’t hosted on AMO and so aren’t subject to the AMO review policies.  And although many add-on authors were responsive, some weren’t.  It was a tedious game of whack-a-mole, and something better was needed.

The Fix

Leaky add-ons usually cause zombie compartments by holding references to DOM structures within a web page even after the page has been closed or navigated away from.  This prevents Firefox from garbage-collecting the page’s compartment.

Mozilla engineer Kyle Huey had the radical idea of simply detecting these kinds of references and cutting them.  This would eradicate this entire class of memory leaks for good, without requiring any changes to individual add-ons.

However, it wasn’t at all obvious that this would work.  In particular, it was possible that code within Firefox and/or add-ons might be broken by this change.  But it turned out, once some bumps were cleared, that almost no code was broken and the leaks were prevented exactly as predicted.  This fix is present in Firefox 15.

Confirmation via in-house testing

I did some early testing on a number of add-ons that were known to leak:  Firebug 1.9.1, McAfee SiteAdvisor 3.4.1, PicPasso 1.0.1, LoL 1.5, YouTube Ratings Preview 1.03, Enter Selects 7, HttpFox 0.8.10, and Link Resolve 0.7.  In every case, Firefox 15 prevents the leaks.

We’ve subsequently confirmed that a number of other leaking add-ons have been fixed by this change.  Although add-ons can still cause zombie compartments in obscure cases, we are yet to encounter such a case in practice.

Confirmation from real-world users

Firefox users can enable telemetry, which is a feature that periodically sends anonymous performance-related data to Mozilla.  This allows us to analyze how people use Firefox in the real world.

We don’t track zombie compartments via telemetry, but we do track something called ghost windows.  The gory details of ghost windows are also beyond the scope of this blog post;  the important thing to know is that they correlate strongly with zombie compartments — a high ghost window count very likely indicates the presence of zombie compartment, and thus memory leaks.

The following graph shows the ghost windows telemetry measurements for users of Firefox Nightly builds on Windows during April and May 2012.  The line marked with a 1 shows when Kyle’s patch landed, and the lines marked with 2 and 3 show when some related follow-up patches landed.

Telemetry data tends to be very noisy, and this graph is no exception, but the effect is clear.  The mean number of ghost windows (shown by the blue dots) dropped from typically between three and four to typically less than one.  (As for why it’s not zero, one reason is that ghost windows can be false positives — i.e. they don’t always correspond to actual leaks.)  Furthermore, the variance has dropped greatly.  Previously, the hardest-hit 5% of measurements would typically have at least 15 or 20 ghost windows, and the next hardest-hit 20% would typically have at least two.  After the fix, the hardest-hit 5% of measurements typically have only two or more ghost windows, and the remaining 95% typically have zero.  This graph is a few weeks old now, but the measurements for Firefox 15 since then have remained steady.

In other words, ghost windows have changed from something that many users experience to something that few users experience.  Furthermore, those users who are still affected experience many fewer than before.  This graph also provides further evidence that add-ons are responsible for a large fraction of the memory leaks encountered by Firefox users.

The Benefits of the Fix

Our testing

I compared pre-release versions of Firefox 14 and Firefox 15 with McAfee SiteAdvisor 3.4.1 installed.  SiteAdvisor is a very popular add-on that rates the safety of web sites, and version 3.4.1 had an unbounded memory leak that leaked significant amounts of memory on every page that Firefox visited.  (That leak has since been fixed and version 3.4.1 is no longer available from McAfee.)

I did the testing on a fast desktop machine with Windows 7 installed.  I ran MemBench, which is a memory test benchmark that opens 150 popular websites, one per tab, and then closes them all.  This test is a reasonable facsimile of several hours of web browsing.

I used about:memory to measure memory consumption (a) with the 150 tabs open (plus one more tab for about:memory itself), and then (b) after closing those 150 tabs.  The metric I measured was resident memory consumption, which is the amount of physical memory used by Firefox.  It is measured by the operating system.  Firefox 14’s two measurements are on the left, and Firefox 15’s two measurements are on the right.

The graph tells a clear story.  The leak in SiteAdvisor 3.4.1 prevents Firefox 14 from releasing most of the memory used by the 150 MemBench tabs, even after those tabs are closed.  In contrast, Firefox 15 prevents the leak.  As a result, after closing 150 tabs, Firefox 15’s resident memory consumption is 4.8x smaller than Firefox 14’s (compare the red bars) without any change to the add-on.

Feedback from users

The feedback from users of the pre-release Nightly and Aurora versions of Firefox has been overwhelmingly positive.  Here are some notable examples.

One user experienced a 4x reduction in memory consumption at the end of the day.

Another user reported via email that a 1.5GB per hour leak — which caused 10+ second pauses and necessitated frequent restarts — disappeared.

Firefox was leaking about 1.5GB per hour for me. It started with Firefox 3. I tracked it down to Ghostery and NoScript, but even without those addons it leaked about 500MB per hour of browsing.

GC and CC times got up into the 10 second range. Ugly. Really really ugly! And this is on top of the line massively overclocked hardware, too. I had to install a new addon to add a restart button to Firefox, because Firefox froze solid after hitting 2GB of memory usage…

Then your patch comes along and solves it all… totally awesome!

Another user experienced faster painting, scrolling, and input operations.

Certainly, before this fix I would find that Firefox often became sluggish (input lag, slow paint operations, less than silky smooth scroll animations) as the memory usage built up… I haven’t noticed the same symptoms since… the signs certainly seem to be good.

These quotes nicely highlight the potential benefits:  greatly reduced memory consumption, which leads to fewer pauses, faster overall performance, and no need to restart Firefox to restore optimum performance.

YMMV

Based on the evidence above, we are confident that Firefox 15 fixes the vast majority of add-on memory leaks, and that as a result, many users will see drastic improvements in Firefox’s performance and stability.  However, predicting the exact effects of this fix on the hundreds of millions of Firefox users is essentially impossible, due to the huge number of different add-ons that are available, and the great variety of add-on and browsing usage patterns.

Nonetheless, if you are a heavy add-on user and you have noticed Firefox’s performance degrade during long browsing sessions, there is a good chance you will see large improvements.  Firebug users may notice improvements as well.  But, as they say:  your mileage may vary.

CONCLUSION

Over the past year, Mozilla has made great progress in reducing Firefox’s memory consumption.  However, the excessive memory consumption caused by add-ons with memory leaks has remained an ongoing problem.

Firefox 15 fixes that problem.  We have confirmed, via in-house testing and from real-world telemetry data, that it prevents the vast majority of leaks that occur in existing add-ons.

Users who upgrade to Firefox 15 won’t have to upgrade their add-ons to see the benefits.  While it is hard to predict the effect of this improvement on any individual user, many users should experience greatly reduced memory consumption, particularly on long browsing sessions.  This should result in better performance, fewer pauses, and fewer crashes.

Mozilla’s MemShrink efforts are ongoing.  We have various projects in the pipeline that aim to further reduce Firefox’s memory consumption, and help users understand better how Firefox is using memory.  Hopefully these changes will make it into Firefox releases not far from now!