Category Archives: Garbage Collection

Generational GC has landed

Big news: late last week, generational garbage collection landed. It was backed out at first due to some test failures, but then re-landed and appears to have stuck.

This helps with performance. There are certain workloads where generational GC makes the code run much faster, and Firefox hasn’t been able to keep up with Chrome on these. For example, it has made Firefox slightly faster on the Octane benchmark, and there is apparently quite a bit of headroom for additional improvements.

Interestingly, its effect on memory usage has been small. I was hoping that the early filtering of many short-lived objects would make the tenured heap grow more slowly and thus reduce memory usage, but the addition of other structures (such as the nursery and store buffers) appears to have balanced that out.

The changes to the graphs at AWSY have been all within the noise, with the exception of the “Fresh start” and “Fresh start [+30s]” measurements in the “explicit” graph, both of which ticked up slightly. This isn’t cause for concern, however, because the corresponding “resident” graph hasn’t increased accordingly, and “resident” is the real metric of interest.

“Compacting Generational GC” is the #1 item on the current MemShrink “Big Ticket Items” list. Hopefully the “compacting” part of that, which still remains to be done, will produce some sizeable memory wins.

A big step towards generational and compacting GC

People frequently ask me for status updates on generational GC, and I usually say I’ll tell them when something notable happens. Well, something notable just happened: exact rooting landed.

What is exact rooting? In order to support generational and/or compacting GC, you need to be able to move GC-allocated things such as objects around. This means you can’t have raw C++ pointers to any objects that might move; instead, you need some kind of indirect pointer that can be updated when necessary.

Unfortunately, both the JS engine and Gecko have a lot of pointers to GC-allocated things. The process of checking and converting them has been the main part of a task called “exact rooting”, and that’s what just finished. This has required an enormous amount of what is essentially very tedious work. Jim Blandy summarized it nicely, as follows.

I’ve never heard of a major project escaping from conservative GC once it had entered that state of sin; nor have I heard of anyone implementing a moving collector after starting with a non-moving collector. So, doing *both* is impressive. I hope it pays off big!

Major kudos to Terrence Cole, Steve Fink, Jon Coppeard, Brian Hackett, and the small army of other helpers who did this. Now that they’ve finished eating this gigantic serving of vegetables, they can move onto dessert, i.e. making the GC generational and compacting.

MemShrink progress, week 85–86

Lots of news today.

Fixed Regressions

I wrote last time about a couple of bad regressions that AWSY identified.

The ongoing DOM bindings work will hopefully fully fix the second regression before the end of this development cycle (February 18).

AWSY

John Schoenick made three big improvements to AWSY.

  • It now measures every push to mozilla-inbound.  Previously it measured mozilla-central once per day.  This will make it easier and faster to identify patches responsible for regressions.
  • It’s now possible to trigger an AWSY run for any try build.  Unfortunately John hasn’t yet written instructions on how to do this;  I hope he will soon…
  • AWSY now measures Fennec as well.  Kartikaya Gupta created the benchmark that is used for this.  He also fixed a 4 MB regression that it identified.

Leaks Fixed

Benoit Jacob fixed a CC leak that he found with his refgraph tool.

Johnny Stenback fixed a leak involving SVG that he found with DMD.  This was a very slow leak that Johnny had seen repeatedly, which manifested as slowly increasing “heap-unclassified” values in about:memory over days or even weeks.  It’s a really nice case because it shows that DMD can be used on long-running sessions with minimal performance impact.

Justin Lebar fixed a B2G leak relating to forms.js.

Randall Jesup fixed a leak relating to WebRTC.

Andrew McCreight fixed a leak relating to HTMLButtonElement.

Erik Vold fixed a leak in the Restartless Restart add-on.

Miscellaneous

Brian Hackett optimized the representation of JS objects that feature both array (indexed) elements and named properties.  Previously, if an object had both elements and named properties, it would use a sparse representation that was very memory-inefficient if many array elements were present.  This performance fault had been known for a long time, and it caused bad memory blow-ups every once in a while, so it’s great to have it fixed.

As a follow-up, Brian also made it possible for objects that use the sparse representation to change back to the dense array representation if enough array elements are subsequently added.  This should also avoid some occasional blow-ups that occur when arrays get filled in in complex ways.

Gregory Szorc reduced the memory consumption of the new Firefox Health Report feature, from ~3 MB to ~1–1.5 MB: here and here and here and here. On a related note, Bill McCloskey is making good progress with reducing compartment overhead, which should be a sizeable win once it lands.

Gregory also reduced the memory consumption of Firefox Sync:  https: here and here.

Jonathan Kew reduced the amount of memory used by textruns when Facebook Messenger is enabled.

The Add-on SDK is now present in mozilla-central, which is a big step towards getting all add-ons that use it to always use the latest version.  This is nice because it will mean that when memory leaks in the SDK are fixed (and there have been many) all add-ons that use it will automatically get the benefit, without having to be repacked.

Generational GC

Generational garbage collection is an ongoing JS engine project that should reap big wins once it’s completed.  I don’t normally write about things that haven’t been finished, but this is a big project and I’ve had various people asking about it recently, so here’s an update.

Generational GC is one the JS teams two major goals for the near-term.  (The other is benchmark and/or game performance, I can’t remember which.)  You can see from the plan that there are eight people working on it (though not all of them are working on it all the time).

Brian Hackett implemented a static analysis that can determine which functions in the JS engine can trigger garbage collection.  On top of that, he then implemented a static analysis that can identify rooting hazards and unnecessary roots.  This may sound esoteric, but it has massively reduced the amount of work required to complete exact rooting, which is the key prerequisite for generational GC.  To give you an idea:  Terrence Cole estimated that it reduced the number of distinct code locations that need to be looked at and possibly modified from ~10,000 to ~200!  Great stuff.

Another good step was taken when I removed support for E4X from the JS engine.  E4X is an old JavaScript language extension that never gained wide support and was only implemented in Firefox.  The code implementing it was complicated, and an ongoing source of many bugs and security flaws.  The removal cut almost 13,000 lines of code and over 16,000 lines of tests.  It’s been destined for the chopping block for a long time, and its presence has been blocking generational GC, so all the JS team members are glad to see it go.

Bug Counts

Here are the current bug counts.

  • P1: 16 (-5/+0)
  • P2: 119 (-6/+0)
  • P3: 104 (-0/+0)
  • Unprioritized: 22 (-0/+18)

Three of the P1 “fixes” weren’t actual fixes, but cases where a bug was WONTFIXed, or downgraded.  The unprioritized number is high because we skipped this week’s MemShrink meeting due to the DOM work week in London, which occupied three of our regular contributors.

MemShrink progress, week 65–66

Gecko

Tim Taubert set the style of a not-yet-restored tab’s <browser> element to display:none, eliminating the need for any layout data to be stored in memory for such a tab.  This was a MemShrink:P1 bug and is a nice improvement for users who typically have many (e.g. 100+) tabs open, because it saves 0.17MB per not-yet-restored tab on 64-bit platforms, and slightly less on 32-bit platforms.

Bobby Holley fixed a defect that was causing some add-ons (NoScript, Roboform, and Google PageSpeed) to create ghost windows.   This fix has been also been ported to the Aurora and Beta channels and so will be present in Firefox 16.

Add-on SDK

Version 1.10 of the Add-on SDK was released.  It has two fixes that prevent various leaks.  Furthermore, Alexandre Poirot fixed two other leaks in the Add-on SDK;  these changes will presumably make it into version 1.11.

The Add-on SDK has had a lot of leaks fixed over the past few months.  Hopefully we’re getting near the end of them!

Generational GC

I have started helping with the ongoing project to convert the JavaScript engine’s mark-and-sweep garbage collector to a compacting, generational collector.  This is the #2 item on the current MemShrink “Big Ticket Items” list because it should significantly reduce JS memory consumption, and it should also help performance in other ways.

It’s a big task because it makes a very fundamental change to the JS engine: garbage-collected things such as JS objects, strings and functions can move.  This is a big deal because these things are represented within the JS engine as C++ objects of various kinds, and it’s not normal for C++ objects to move!  Think about it — if you have a pointer to a C++ object, and the object moves, the pointer is no longer valid.

I won’t go into detail about how we handle this but the important thing to know is that pretty much every place in the JS engine where we have a raw C++ pointer to a GC thing object needs to be manually changed to use a new representation.  This is literally tens of thousands of changes that must be completed before the garbage collector can be made either compacting or generational (the two properties are orthogonal, with both providing benefits).

If you are interested in following the progress of this conversion, take a look at bug 753203.

A reminder

In these MemShrink reports I usually write about Firefox changes just after they first land on mozilla-central.  Such changes will show up in Nightly builds within a day, but won’t reach an official Firefox release for 12–18 weeks, as per Firefox’s release schedule.  If you remember that you’ll avoid any confusion about which changes are in which release.

For example:  except where noted otherwise, the changes I’ve mentioned above will be in Firefox 18 — assuming they don’t cause problems that cause them to be backed out — which is due for final release in early January, 2013. (The current scheduled release date is January 1st, but that will probably be delayed slightly because New Year’s Day is not a good day for a release!)

Bug Counts

Here are the current bug counts.

  • P1: 21 (-1/+0)
  • P2: 89 (-5/+4)
  • P3: 100 (-3/+7)
  • Unprioritized: 3 (-2/+3)

MemShrink progress, week 57–60

I’ve been on vacation for two weeks, so this report covers the past four weeks.  I still have a mountain of unread emails, bug reports and blog posts to get through, but hopefully I haven’t missed much.

Hueyfix

I wrote about Firefox 15 and how it contains the “Hueyfix” that prevents most memory leaks caused by add-ons.  It got lots of attention.

Relatedly, a regression caused by the Hueyfix was discovered:  it caused some Greasemonkey scripts to leak badly, such as “YousableTubeFix” and “Textarea backup with expiry”.  A change was made to Firefox to allow this to be prevented (which has been backported to Firefox 15 Beta), and Greasemonkey 0.9.22 should fix the problem, although that hasn’t yet been confirmed.  If you have had problems with Firefox 15 and you are using Greasemonkey, this is a likely cause.  I’m not aware of any other regressions caused by the Hueyfix.

Also relatedly, Gabor Krizsanits made a change inspired by the Hueyfix that cuts references to sandboxes once they should no longer be used.  This prevents a whole class of possible leaks in add-ons using the Add-on SDK.

Miscellaneous

Gregor Wagner tweaked the JS engine’s garbage collection heuristics to greatly reduce the peak memory consumption in certain cases.  Gregor previously tweaked the GC heuristics in Firefox 7 with great success.

Tim Taubert fixed a leak caused by opening and closing the bookmarks sidebar.  This was a precursor to a more important change from Tim, which adds automated checking of the cycle collection logs to check for leaks during testing.  This should allow a whole class of memory leaks to be detected automatically, thus preventing regressions.

Benoit Jacob made a change that prevents the number of WebGL contexts growing excessively.  This can greatly improve memory consumption on certain sites that use WebGL.

Nick Cameron fixed a problem that caused huge memory consumption (multiple GB) on pages using canvas in conjunction with certain Intel graphics cards and/or drivers.  I don’t understand the details, but it sounds like some cards and/or drivers might still have problems.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-0/+3)
  • P2: 89 (-9/+8)
  • P3: 105 (-2/+5)
  • Unprioritized: 1 (-2/+1)

Nothing spectacular there.

MemShrink progress, week 55-56

Memory Reporting

I changed things so that JavaScript memory consumption for web content is reported on a per-tab basis, as the following example shows.

│  ├────4,472,568 B (00.50%) -- top(http://www.mozilla.org/en-US/firefox/fx/, id=236)/active
│  │    ├──4,192,640 B (00.47%) -- window(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,979,112 B (00.22%) ++ js/compartment(http://www.mozilla.org/en-US/firefox/fx/)
│  │    │  ├──1,607,216 B (00.18%) ++ layout
│  │    │  ├────352,520 B (00.04%) ── style-sheets
│  │    │  ├────253,312 B (00.03%) ++ dom
│  │    │  └────────480 B (00.00%) ── property-tables

All the major components of each tab’s memory consumption — JS, layout, style sheets, and DOM — are now reported on a per-tab basis.  This is great, because it’s something that people have been requesting for  years.  However, the presentation is still intimidating.  Consider the following excerpt from about:memory.

├───18,288,696 B (16.43%) -- window-objects
│   ├───6,263,208 B (05.62%) ++ top(about:memory?verbose, id=6)/active
│   ├───5,090,352 B (04.57%) ++ top(chrome://browser/content/browser.xul, id=1)/active
│   ├───3,422,280 B (03.07%) ++ top(http://www.mozilla.org/en-US/firefox/fx/, id=17)/active/window(http://www.mozilla.org/en-US/firefox/fx/)
│   ├───3,178,432 B (02.85%) ++ top(chrome://global/content/console.xul, id=21)/active
│   └─────334,424 B (00.30%) ++ top(resource://gre-resources/hiddenWindow.html, id=3)/active/window(resource://gre-resources/hiddenWindow.html)

Among those top(...) entries are two browser tabs (one for www.mozilla.org, one for about:memory), two browser chrome windows (browser.xul is the main browser window, and console.xul is the error console window), and the hidden window which is always present.  Firefox developers can easily understand this, and technically-inclined users may be able to guess what’s going on, but the average user will struggle.

From a user’s point of view, the visible things they have control over (i.e. windows and tabs) should be better distinguished from everything else, and be shown with nicer names than “browser.xul”.  But the architecture of Firefox’s internals doesn’t make this easy.  I’ve made one not-very-successful attempt to improve this, and I’d love to hear suggestions from other Firefox devs on how to present this information better.

(The step after that one is to present some kind of “tab manager” — a massively stripped-down variant of about:memory that ordinary users can understand.  This would also require localization for all the languages that Firefox supports.  One step at a time!)

Finally, another memory reporting improvement happened:  Andrew McCreight and I added measurement of orphan DOM nodes.  These are DOM nodes that have been discarded by a page but are still accessible from JavaScript objects.   They were the biggest single remaining contributor to “heap-unclassified”.  For example, when I hit the “update” button on about:memory I get 5MB of them (don’t worry, they go away when the garbage collector runs).  Also, badly coded sites can create large numbers of them.  So it’s a good thing to be able to measure, for both users and web developers.

Miscellaneous

Mihai Sucan fixed a bad leak involving pages that use window.console.  Multiple people reported that this caused Firefox’s memory consumption to balloon to multiple GB.  This leak was introduced during the Firefox 15 development cycle, and the fix has been backported to the Aurora channel, which is where Firefox 15 currently resides.  One notable thing about this fix is that the problem was quite easy to diagnose because about:memory clearly indicated that the ConsoleAPI.js compartment was responsible for most of the memory consumption.  Without the fine-grained memory reporting that was enabled by compartment-per-global, that memory would have been reported in a generic “System Principal” compartment which would have made diagnosis much harder.

Wladimir Palant fixed an issue in AdBlock Plus that causes its memory consumption to slowly grow when viewing sites that repeatedly reset the src property of images, such as Mibbit.  (The fine-grained memory reporting enabled by compartment-per-global also made this diagnosis much easier.)  This fix is in Adblock Plus 2.1.

Alexandre Poirot fixed a minor leak in the Add-on SDK.

I wrote about the difficulties of cross-browser memory comparisons.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+0)
  • P2: 90 (-3/+6)
  • P3: 102 (-6/+2)
  • Unprioritized: 2 (-0/+0)

A net reduction of seven bugs!

Update on The big ticket Items

In January, I described six “big ticket items” that needed tackling to improve Firefox’s memory consumption.  Let’s look at how they’ve progressed since then.

#6: Better Script Handling

This had two parts.  The first part was to only generate bytecode for a function once it is run.  I’ve made a good chunk of progress towards and have it working on very simple examples.  This has required a great deal of refactoring of the SpiderMonkey front-end.  (The front-end is a hairy piece of code, so this is virtuous in and of itself.)  There are still a couple of other bugs blocking it from further progress.  I intend to return to continue on this once they are done.

The second part was to share immutable parts of scripts between web pages.  No progress has been made on this.

So, overall, this item is about 25% done, and progress is still ticking along slowly.

#5: Better Memory Reporting

This one also had two parts.  The first part was to reduce about:memory’s “heap-unclassified” number (a.k.a “dark matter”) to typically 10%.  People such as Nathan Froyd and I have made good progress on this.  It’s now occasionally below 10% (my current session shows 7.5%) but often is around 15%.  That’s much better than the 20–25% we typically were getting at the start of the year, and it’s low enough that people rarely complain about it now.  And we’re very much into the long tail now, so it’ll be hard to improve things much further.  For the most part I don’t think we need to.

The second part was to report memory in a per-tab fashion.  As mentioned above, I just completed this, although the presentation is not as user-friendly as it could be.

The per-tab reporting was enabled by compartment-per-global.  Compartment-per-global also gives us much more insight into the memory consumption of Firefox’s own JavaScript code, and even some idea about add-on memory consumption, which was (to me) an unexpected bonus.  Indeed, its usefulness was demonstrated by the two leak fixes mentioned above.

So, overall, this item is about 80% done.  Although there is still some work to be done, it has progressed enough that I will remove it from the “big ticket items” list.

#4: Better Memory Consumption Tracking

We can cross this item off the list, thanks to areweslimyet.com.  In January I also mentioned the idea of using telemetry for this, but we’ve found that telemetry results are so noisy that little (with rare exceptions) can be gleaned from it, so there’s not really anything to be done on that front.

#3: Compacting Generational GC

This is one of the two key SpiderMonkey performance features under development (the other being IonMonkey), and so people such as Terrence Cole, Steve Fink, Brian Hackett, and Bill McCloskey and actively working on it.  Unfortunately, it’s a huge undertaking that requires rewriting many of SpiderMonkey’s internal and external APIs.  If I had to guess I’d say it’s 25% complete, but that really is just a wild guess.  I’ll be pleased if it is finished before the end of the year.

#2: Better Foreground Tab Image Handling

This is the big ticket item for which the least progress has been made. There’s a single bug that is blocking all the follow-on work.  Both Timothy Nikkel and Jet Villegas have spent time working on it, but it has been a difficult nut to crack.  Justin Lebar had a recent suggestion for narrowing its scope slightly.  I hope it’s a good suggestion, because I’m out of ideas on this one.

#1: Better Detection and Notification of Leaky Add-ons

This is the most satisfying item of the lot.  I expected to make only grudging, piecemeal progress, but Kyle Huey’s fix to prevent zombie compartments prevents the vast majority of add-on leaks, enough to declare victory and cross this item off the list.  I’ll be writing more about this next week.

Summary

Three items (#1, #4, #5) have progressed enough that they can be removed from the list.  Only one of these directly improved Firefox’s memory consumption;  the other two were about better measuring and monitoring of memory consumption.

The remaining items are about reducing memory consumption.  Two of them (#3, #6) have a long way to go, but progress has been made and is ongoing.  One item (#2) has made little progress and has stalled.

The New big ticket Items

Here’s my updated list of the most important things.

#5: Better Script Handling

As mentioned above, this has two parts: lazy bytecode generation, and the  sharing of immutable parts of scripts.  I intend to continue working on the first part.  Once that’s done, it may turn out that the second part isn’t necessary;  we’ll see.

#4: Regain compartment-per-global losses

The measurements on areweslimyet.com have been creeping up for the past two months.  The major cause is compartment-per-global, which has lots of benefits (as mentioned above) but also increases memory consumption by a small-to-moderate amount across the board.  This is because there are many more compartments than there used to be, and there’s a certain amount of overhead and waste in each compartment.

Generational GC will help, but there are some other things that can be done more quickly that will help.  See the tracking bug for one example.

#3: Boot2Gecko

Boot2Gecko (now officially known as “Firefox OS”) is going to be running on low-end phones without much physical memory, so that’ll require care.  The first step is to get about:memory working on B2G.  The next step is to find a way to copy the data from about:memory somewhere else, because B2G doesn’t have cut and paste!  Beyond that, it’s unclear;  but once about:memory is working it’s very likely it will identify some sub-optimal B2G-specific behaviour.

#2: Compacting Generational GC

Let me quote myself to explain the memory consumption benefits of compacting, generational GC.

Virtual memory consumption drops in two ways.  First, the compaction minimizes waste due to fragmentation.  Second, the heap grows more slowly.  Physical memory consumption drops for the same two reasons.

The performance benefits are also significant.

Performance improves for three reasons.  First, paging is reduced because of the generational behaviour:  much of the JS engine activity occurs in the nursery, which is small;  in other words, the memory activity is concentrated within a smaller part of the working set.  Second, paging is further reduced because of the compaction: this reduces fragmentation within pages in the tenured heap, reducing the total working set size.  Third, the tenured heap grows more slowly because of the generational behaviour… which means that structure traversals done by the garbage collector (during full-heap collections) and cycle collector are faster.

Important stuff.

#1: Better Foreground Tab Image Handling

On image-heavy pages, Firefox uses much more memory than other browsers.  This includes things like Facebook image slideshows.  One commenter said the following.

On behalf of Facebook Inc, let me just say that we are prepared to give over one thousand free “pokes” to anyone who fixes this bug or alternatively https://bugzilla.mozilla.org/show_bug.cgi?id=683290. I will also gladly show you Mark Zuckerberg’s desk and give you free dinner and cookies as part of a complementary tour of Facebook HQ in Menlo Park.

I have no idea if this person is really from Facebook, but it’s certainly suggestive.  Also, I’ve heard from B2G people that they are having to work around this problem in their Gallery app.

To repeat myself from week 32:  there are three MemShrink:P1 bugs relating to this:  one about not decoding all images immediately, one about discarding non-visible decoded images after some time, and one about some infrastructure work that is required for the first two.  (See this discussion on the dev-platform mailing list for more details about this topic.)

Summary

There are three repeat items, and two new items, but there are fewer items than last time;  this reflects the fact that we are in a better position than we were in January.  If you think I’ve overestimated or underestimated the importance of any of these, or omitted anything, I’d be interested to hear.

One final thing:  last time, several people suggested “better reporting of add-on memory consumption”.  If I knew of any ideas on how to do this better than about:memory currently does, I’d gladly put it on the list.  But I don’t, and there doesn’t seem to be much point to put an item on the list that we don’t know how to implement.

Vacation

I’m taking the next two weeks as vacation, so there won’t be a MemShrink report on July 24.  I’ll be back with the next MemShrink round-up on August 8th.  See you then.

How to Compare the Memory Efficiency of Web Browsers

TL;DR: Cross-browser comparisons of memory consumption should be avoided.  If you want to evaluate how efficiently browsers use memory, you should do cross-browser comparisons of performance across several machines featuring a range of memory configurations.

Cross-browser Memory Comparisons are Bad

Various tech sites periodically compare the performance of browsers.  These often involve some cross-browser comparisons of memory efficiency.  A typical one would be this:  open a bunch of pages in tabs, measure memory consumption, then close all of them except one and wait two minutes, and then measure memory consumption again.  Users sometimes do similar testing.

I think comparisons of memory consumption like these are (a) very difficult to make correctly, and (b) very difficult to interpret meaningfully.  I have suggestions below for alternative ways to measure memory efficiency of browsers, but first I’ll explain why I think these comparisons are a bad idea.

Cross-browser Memory Comparisons are Difficult to Make

Getting apples-to-apples comparisons are really difficult.

  1. Browser memory measurements aren’t easy.  In particular, all browsers use multiple processes, and accounting for shared memory is difficult.
  2. Browsers are non-deterministic programs, and this can cause wide variation in memory consumption results.  In particular, whether or not the JavaScript garbage collector runs can greatly reduce memory consumption.  If you get unlucky and the garbage collector runs just after you measure, you’ll get an unfairly high number.
  3. Browsers can exhibit adaptive memory behaviour.  If running on a machine with lots of free RAM, a browser may choose to take advantage of it;  if running on a machine with little free RAM, a browser may choose to discard regenerable data more aggressively.

If you are comparing two versions of the same browser, problems (1) and (3) are avoided, and so if you are careful with problem (2) you can get reasonable results.  But comparing different browsers hits all three problems.

Indeed, Tom’s Hardware de-emphasized memory consumption measurements in their latest Web Browser Grand Prix due to problem (3).  Kudos to them!

Cross-browser Memory Comparisons are Difficult to Interpret

Even if you could get the measurements right, memory consumption is still not a good thing to compare.  Before I can explain why, I’ll introduce a couple of terms.

  • A primary metric is one a user can directly perceive.  Metrics that measure performance and crash rate are good examples.
  • A secondary metric is one that a user can only indirectly perceive via some kind of tool.  Memory consumption is one example.  The L2 cache miss rate is another example.

(I made up these terms, I don’t know if there are existing terms for these concepts.)

Primary metrics are obviously important, precisely because user can detect them.  They measure things that users notice:  “this browser is fast/slow”, “this browser crashes all the time”, etc.

Secondary metrics are important because they can affect primary metrics:  memory consumption can affect performance and crash rate;  the L2 cache miss rate can affect performance.

Secondary metrics are also difficult to interpret.  They can certainly be suggestive, but there are lots of secondary metrics that affect each primary metric of interest, so focusing too strongly on any single secondary metric is not a good idea.  For example, if browser A has a higher L2 cache miss rate than browser B, that’s suggestive, but you’d be unwise to draw any strong conclusions from it.

Furthermore, memory consumption is harder to interpret than many other secondary metrics.  If all else is equal, a higher L2 cache miss rate is worse than a lower one.  But that’s not true for memory consumption.  There are all sorts of time/space trade-offs that can be made, and there are many cases where using more memory can make browsers faster;  JavaScript JITs are a great example.

And I haven’t even discussed which memory consumption metric you should use.  Physical memory consumption is an obvious choice, but I’ll discuss this more below.

A Better Methodology

So, I’ve explained why I think you shouldn’t do cross-browser memory comparisons.  That doesn’t mean that efficient usage of memory isn’t important! However, instead of directly measuring memory consumption — a secondary metric — it’s far better to measure the effect of memory consumption on primary metrics such as performance.

In particular, I think people often use memory consumption measurements as a proxy for performance on machines that don’t have much RAM.  If you care about performance on machines that don’t have much RAM, you should measure performance on a machine that doesn’t have much RAM instead of trying to infer it from another measurement.

Experimental Setup

I did exactly this by doing something I call memory sensitivity testing, which involves measuring browser performance across a range of memory configurations.  My test machine had the following characteristics.

  • CPU: Intel i7-2600 3.4GHz (quad core with hyperthreading)
  • RAM: 16GB DDR3
  • OS: Ubuntu 11.10, Linux kernel version 3.0.0.

I used a Linux machine because Linux has a feature called cgroups that allows you to restrict the machine resources available to one or more processes.  I followed Justin Lebar’s instructions to create the following configurations that limited the amount of physical memory available: 1024MiB, 768MiB, 512MiB, 448MiB, 384MiB, 320MiB, 256MiB, 192MiB, 160MiB, 128MiB, 96MiB, 64MiB, 48MiB, 32MiB.

(The more obvious way to do this is to use ulimit, but as far as I can tell it doesn’t work on recent versions of Linux or on Mac.  And I don’t know of any way to do this on Windows.  So my experiments had to be on Linux.)

I used the following browsers.

  • Firefox 12 Nightly, from 2012-01-10 (64-bit)
  • Firefox 9.0.1 (64-bit)
  • Chrome 16.0.912.75 (64-bit)
  • Opera 11.60 (64-bit)

IE and Safari aren’t represented because they don’t run on Linux.  Firefox is over-represented because that’s the browser I work on and care about the most :)  The versions are a bit old because I did this testing about six months ago.

I used the following benchmark suites:  Sunspider v0.9.1, V8 v6, Kraken v1.1.  These are all JavaScript benchmarks and are all awful for gauging a browser’s memory efficiency;  but they have the key advantage that they run quite quickly.  I thought about using Dromaeo and Peacekeeper to benchmark other aspects of browser performance, but they take several minutes each to run and I didn’t have the patience to run them a large number of times.  This isn’t ideal, but I did this exercise to test-drive a benchmarking methodology, not make a definitive statement about each browser’s memory efficiency, so please forgive me.

Experimental Results

The following graph shows the Sunspider results.  (Click on it to get a larger version.)

sunspider results graph

As the lines move from right to left, the amount of physical memory available drops.  Firefox was clearly the fastest in most configurations, with only minor differences between Firefox 9 and Firefox 12pre, but it slowed down drastically below 160MiB;  this is exactly the kind of curve I was expecting.  Opera was next fastest in most configurations, and then Chrome, and both of them didn’t show any noticeable degradation at any memory size, which was surprising and impressive.

All the browsers crashed/aborted if memory was reduced enough.  The point at which the graphs stop on the left-hand side indicate the lowest size that each browser successfully handled.  None of the browsers ran Sunspider with 48MiB available, and FF12pre failed to run it with 64MiB available.

The next graph shows the V8 results.

v8 results graph

The curves go the opposite way because V8 produces a score rather than a time, and bigger is better.  Chrome easily got the best scores.  Both Firefox versions degraded significantly.  Chrome and Opera degraded somewhat, and only at lower sizes.  Oddly enough, FF9 was the only browser that managed to run V8 with 128MiB available;  the other three only ran it with 160MiB or more available.

I don’t particularly like V8 as a benchmark.  I’ve always found that it doesn’t give consistent results if you run it multiple times, and these results concur with that observation.  Furthermore, I don’t like that it gives a score rather than a time or inverse-time (such as runs per second), because it’s unclear how different scores relate.

The final graph shows the Kraken results.

kraken results graph

As with Sunspider, Chrome barely degraded and both Firefoxes degraded significantly.  Opera was easily the slowest to begin with and degraded massively;  nonetheless, it managed to run with 128MiB available (as did Chrome), which neither Firefox managed.

Experimental Conclusions

Overall, Chrome did well, and Opera and the two Firefoxes had mixed results. But I did this experiment to test a methodology, not to crown a winner.  (And don’t forget that these experiments were done with browser versions that are now over six months old.)  My main conclusion is that Sunspider, V8 and Kraken are not good benchmarks when it comes to gauging how efficiently browsers use memory.  For example, none of the browsers slowed down on Sunspider until memory was restricted to 128MiB, which is a ridiculously small amount of memory for a desktop or laptop machine;  it’s small even for a smartphone.  V8 is clearly stresses memory consumption more, but it’s still not great.

What would a better benchmark look like?  I’m not completely sure, but it would certainly involve opening multiple tabs and simulate real-world browsing. Something like Membench (see here and here) might be a reasonable starting point.  To test the impact of memory consumption on performance, a clear performance measure would be required, because Membench lacks one currently.  To test the impact of memory consumption on crash rate, Membench could be modified to just keep opening pages until the browser crashes.  (The trouble with that is that you’d lose your count when the browser crashed!  You’d need to log the current count to a file or something like that.)

BTW, if you are thinking “you’ve just measured the working set size“, you’re exactly right! I think working set size is probably the best metric to use when evaluating memory consumption of a browser.  Unfortunately it’s hard to measure (as we’ve seen) and it is best measured via a  curve rather than a single number.

 A Simpler Methodology

I think memory sensitivity testing is an excellent way to gauge the memory efficiency of different browsers.  (In fact, the same methodology can be used for any kind of program, not just browsers.)

But the above experiment wasn’t easy:  it required a Linux machine, some non-trivial configuration of that machine that took me a while to get working, and at least 13 runs of each benchmark suite for each browser.  I understand that tech sites would be reluctant to do this kind of testing, especially when longer-running benchmark suites such as Dromaeo and Peacekeeper are involved.

A simpler alternative that would still be quite good would be to perform all the performance tests on several machines with different memory configurations.  For example, a good experimental setup might involve the following machines.

  • A fast desktop with 8GB or 16GB of RAM.
  • A mid-range laptop with 4GB of RAM.
  • A low-end netbook with 1GB or even 512MB of RAM.

This wouldn’t require nearly as many runs as full-scale memory sensitivity testing would.  It would avoid all the problems of cross-browser memory consumption comparisons:  difficult measurements, non-determinism, and adaptive behaviour.  It would avoid secondary metrics in favour of primary metrics.  And it would give results that are easy for anyone to understand.

(In some ways it’s even better than memory sensitivity testing because it involves real machines — a machine with a 3.4GHz i7-2600 CPU and only 128MiB of RAM isn’t a realistic configuration!)

I’d love it if tech sites started doing this.

MemShrink progress, week 51-52

Memory Reporting

Nathan Froyd added much more detail to the DOM and layout memory reporters, as the following example shows.

├──3,268,944 B (03.76%) -- window(http://www.reddit.com/)
│  ├──1,419,280 B (01.63%) -- layout
│  │  ├────403,904 B (00.46%) ── style-sets
│  │  ├────285,856 B (00.33%) -- frames
│  │  │    ├───90,000 B (00.10%) ── nsInlineFrame
│  │  │    ├───72,656 B (00.08%) ── nsBlockFrame
│  │  │    ├───62,496 B (00.07%) ── nsTextFrame
│  │  │    ├───42,672 B (00.05%) ── nsHTMLScrollFrame
│  │  │    └───18,032 B (00.02%) ── sundries
│  │  ├────259,392 B (00.30%) ── pres-shell
│  │  ├────168,480 B (00.19%) ── style-contexts
│  │  ├────160,176 B (00.18%) ── pres-contexts
│  │  ├─────55,424 B (00.06%) ── text-runs
│  │  ├─────45,792 B (00.05%) ── rule-nodes
│  │  └─────40,256 B (00.05%) ── line-boxes
│  ├────986,240 B (01.13%) -- dom
│  │    ├──780,056 B (00.90%) ── element-nodes
│  │    ├──156,184 B (00.18%) ── text-nodes
│  │    ├───39,936 B (00.05%) ── other [2]
│  │    └───10,064 B (00.01%) ── comment-nodes
│  └────863,424 B (00.99%) ── style-sheets

Although these changes slightly improved the coverage of the reporters (i.e. they reduce “heap-unclassified” a bit), the more important effect is that they give greater insight into DOM and layout memory consumption.  [Update:  Nathan blogged about these changes.]

I modified the memory reporter infrastructure and about:memory to allow trees of measurements to be shown in the “Other Measurements” section of about:memory.  The following excerpt shows two sets of measurements that were previously shown in a flat list.

108 (100.0%) -- js-compartments
├──102 (94.44%) ── system
└────6 (05.56%) ── user

3,900,832 B (100.0%) -- window-objects
├──2,047,712 B (52.49%) -- layout
│  ├──1,436,464 B (36.82%) ── style-sets
│  ├────402,056 B (10.31%) ── pres-shell
│  ├────100,440 B (02.57%) ── rule-nodes
│  ├─────62,400 B (01.60%) ── style-contexts
│  ├─────30,464 B (00.78%) ── pres-contexts
│  ├─────10,400 B (00.27%) ── frames
│  ├──────2,816 B (00.07%) ── line-boxes
│  └──────2,672 B (00.07%) ── text-runs
├──1,399,904 B (35.89%) ── style-sheets
└────453,216 B (11.62%) -- dom
     ├──319,648 B (08.19%) ── element-nodes
     ├──123,088 B (03.16%) ── other
     ├────8,880 B (00.23%) ── text-nodes
     ├────1,600 B (00.04%) ── comment-nodes
     └────────0 B (00.00%) ── cdata-nodes

This change improves the presentation of measurements that cross-cut those in the “explicit” tree.  I intend to modify a number of the JS engine memory reports to take advantage of this change.

One interesting bug was this one, where various people were seeing multiple compartments for the same site in about:compartments, as the following excerpt shows.

https://bugzilla.mozilla.org/
https://bugzilla.mozilla.org/, about:blank

It turns out this is not a bug.  The sites in question contain an empty iframe that doesn’t specify a URL, and so the memory reporters are doing exactly the right thing.  Still, a useful case to know about when reading about:compartments.

Add-ons

Justin Lebar fixed the FUEL API, which is used by numerous add-ons including Test Pilot and PDF.js, so that it doesn’t leak most of the objects it creates.  This leak caused 10+ minute shut-down times(!) for one user, so it’s a good one to have fixed.

Kyle Huey briefly broke his own Hueyfix, and then fixed it.  You had us worried for a bit there, Kyle!

asgerklasker fixed a leak in the HttpFox add-on that caused zombie compartments.

Miscellaneous

I restricted the amount of context that is shown in JavaScript error messages.  This fixed a longstanding bug where if you had enabled the javascript.options.strict option in about:config, and you had the error console open or Firebug installed, memory consumption would spike dramatically when viewing pages that contain JavaScript code that triggered many strict warnings.  This bug manifested rarely, but would bring Firefox to its knees when it did.

Asaf Romano fixed a leak that caused a zombie compartment if you used the “Highlight All” checkbox when doing a text search in a page.

Mike Hommey finished importing the new version of jemalloc into the Mozilla codebase, and then wrote about it.  The new version is currently disabled, but we hope to turn it on soon.  Preliminary experiments indicate that the new version is unlikely to affect performance or memory consumption much.  However, it will be good to be on a version that is in sync with the upstream version, instead of having a highly hacked Mozilla-specific version.

Justin Lebar wrote a nice blog post explaining what “ghost windows” are, and how we’ve used them to gauge some recent leak fixes.

Till Schneidereit fixed a garbage collection defect that could cause memory usage to spike in certain cases involving Firefox Sync.

LifeHacker published a new browser performance comparison which looked at Firefox 13, Chrome 19, IE9, and Opera 11.64.  Firefox did the best overall and also won both the memory consumption tests.  I personally take these kinds of comparisons with a huge bucket of salt.  Indeed, quoting from the article:

Our tests aren’t the most scientific on the planet, but they do reflect a relatively accurate view of the kind of experience you’d get from each browser, speed-wise.

If you ask me, the text before the “but” contradicts the rest of the sentence.  What I found more interesting was the distinct lack of “lolwut everyone knows Chrome is faster than Firefox” comments, which I’m used to seeing when Firefox does well in these kinds of articles.

Bug Counts

Here are the current bug counts.

  • P1: 25 (-1/+3)
  • P2: 88 (-3/+6)
  • P3: 106 (-0/+4)
  • Unprioritized: 2 (-2/+2)

Nothing too exciting there.

MemShrink progress, week 37

It’s been a huge week for MemShrink.

Add-ons

I filed a bug on testing the top 100 add-ons for memory leaks.  The majority of these add-ons are not available on AMO and so are not subject to the leak checking done during AMO reviews.  The instructions and current test results are here.  Randell Jesup helped identify some some add-ons with unclear identities, and Nils Maier and Archaeopteryx have both tested several add-ons.  A month ago I said that add-ons that leak are the #1 problem for MemShrink, and this bug represents a huge step towards reducing that problem.  We’ll need lots of additional help to test this many add-ons, so please feel free to jump in!

As far as I know, comprehensive testing of the most popular add-ons has never been done before.  Although this testing is aimed at finding memory leaks, it’s quite possible it will uncover various other problems with add-ons.  Indeed, it’s clear just from looking at the list that quite a few of the non-AMO add-ons are of dubious merit, and this has stimulated some interesting discussion on dev-platform about third-party add-ons, i.e. those that are installed into Firefox by an external program.

The following add-ons had leaks fixed or mostly fixed by their authors: McAfee Site Advisor, Ghostery, Spool, Screengrab (fix version), Roboform, UF Comment Board ToolsGeenstijl, Long URL Please, HTML Desktop Notifications.

Nils Maier greatly improved the documentation on common causes of memory leaks in add-ons.  This was a MemShrink:P1 bug.  That doesn’t mean the document can’t be improved further, however;  it’s a wiki so please continue to add to it if you can.

Regression testing

John Schoenick’s excellent areweslimyet.com is getting close to being ready for a public unveiling.  The site is currently live but password-protected, simply to prevent large numbers of people accessing it before it’s ready.  (If you want a sneak peek, contact me for the password.)  More importantly, it identified a large regression in memory consumption caused by the landing of the JS engine’s incremental garbage collector on February 19, which Bill McCloskey is investigating.

Tim Taubert improved Firefox’s regression testing to ensure that no new DocShell and DOMWindow leaks are introduced.  This was a MemShrink:P1 bug because these leaks are quite easy to introduce.  Tim wrote some more about this change here.

Cycle collector

Jan Honza Odvarko’s about:ccdump add-on made it onto AMO this week.  Jan has been using this to hunt down some tricky leaks in Firebug.

Marco Bonardo fixed a leak in PlacesUtils.removeLazyBookmarkObserver() that I won’t pretend to understand, but which was a MemShrink:P1.  Marco also fixed a leak that occurred after bookmarking a page, closing the window, and opening a new window.

Olli Pettay fixed an equally impenetrable (to me) leak involving nsFormFillController::mFocusedInput.

Kyle Huey fixed a leak in the About Firefox dialog.

Andrew McCreight added additional information to cycle collector’s error console logging, which helps with identifying and debugging cycle collector problems.

Miscellaneous

Jason Duell fixed an obscure leak involve web sockets.

Bill McCloskey fixed a minor JS engine leak.

I added a new memory reporter called “js/runtime/gc-marker”.  It starts out at 256KB and I saw it reach 2.8MB when running MemBench.  I also converted the existing source image reporters to the new style, which seems to haved fixed some bogus results in about:memory as a side-effect.

Quote of the week

Adam Overa of Tom’s Hardware did another browser Grand Prix.  Devin Coldewey used the results from the Grand Prix to conclude that Firefox and Chrome are both great, and which you prefer basically comes down to a matter of taste.  While that is an interesting conclusion, what caught my eye was this comment, by Balaji Viswanathan:

I guess the title could have been “All browsers suck equally bad now”. Firefox and Chrome have grown to become more resources hogs – closer to IE. These days I shudder to fire up the firefox as the memory leaks are terrible. Two hours of Firefox with a Facebook tab is enough to gobble up 25% of my memory. With Chrome, I would have 20 rendering processes showing up when I would have just 3 tabs open.

In the first 2 years of Chrome I had probably 2 or 3 crashes. These days, it crashes every week or so. I long for the 2006 era Firefox – clean, simple and lean. Its been a long time since I have used IE actively and with Safari lesser said the better. For the developers, we have to deal with the ultra slow update cycles of IE and ultraspeed update cycles of Chrome. In short, we are getting worse and worse in the browser arena. The market is ripe for a breakthrough.

“2006 era Firefox” would have been Firefox 1.5 or Firefox 2.  Balaji may think he’s nostalgic for browsers of that era, but really he’s nostalgic for the web of that era, which was massively simpler than the web of today.  If Balaji tried Firefox 1.5 or Firefox 2 today — with no HTML5 support, no JavaScript JIT and no hardware acceleration — today, I’m sure he’d quickly decide that 2012-era browsers aren’t so bad after all.  I guess the interesting question is this: why do people blame browsers instead of websites for high resource consumption?

Bug counts

This week’s bug counts:

  • P1: 28 (-5/+5)
  • P2: 129 (-12/+7)
  • P3: 83 (-8/+12)
  • Unprioritized: 2 (-2/+2)

That’s a net reduction of one bug.  But more importantly, there is lots of movement, which is great!  It means that problems are being identified and fixed.

(Finally, commenters please note that I’ve turned on the WP-reCAPTCHA plug-in, and this the first post I’ve written since doing so.  In my testing it’s worked fairly well.  Hopefully it won’t cause problems.)

MemShrink progress, week 32

There wasn’t much MemShrink activity this week in terms of bugs fixed, just bug 718100 and bug 720359.  So I’m going to take the opportunity this week to talk about the bigger picture.

Bug Counts

As a prelude, here are this week’s bug counts.

  • P1: 20 (-4/+0)
  • P2: 131 (-3/+3)
  • P3: 74 (-2/+7)
  • Unprioritized: 4 (-3/+4)

The drop in P1s was just due to bug re-classification;  in particular, three bugs relating to long cycle collector pauses were un-MemShrink’d because they are more about responsiveness, and they are being tracked by Project Snappy.

The Big Ticket Items

David Mandelin asked me today what where the big ticket items for MemShrink.  I’d been looking a lot at the MemShrink:P1 list recently (which is why some were re-classified) and so I was able to break it down into six main areas that cover most of the P1s and various P2s.  I’ll list these from what I think is least important to most important.

#6: Better Script Handling

Internally, a JSScript represents (more or less) the code of a JS function, including things like the internal bytecode that SpiderMonkey generates for it.  The memory used by JSScripts is measured by the “gc-heap/scripts” and “script-data” entries in about:memory.

Luke Wagner did some measurements that showed that most (70–80%) JSScripts created in the browser are never run.  In hindsight, this isn’t so surprising — many websites load libraries like jQuery but only use a fraction of the functions in those libraries.  If SpiderMonkey could be changed to generate bytecode for scripts lazily, it could reduce “script-data” memory usage by 60–70%.  This would also allow the decompiler to be removed, which would be great.

Luke also proposed sharing immutable parts of scripts between web pages.  This would avoid a lot of duplication in the case where you have many tabs open with pages from a single site.

Both of these changes potentially will make the browser faster as well, because SpiderMonkey will spend less time compiling JavaScript source code to bytecode.

No-one is assigned to work on these bugs.  The lazy script creation can be done entirely within the JS engine;  the script sharing requires assistance from Necko.  Luke is currently busy with some other righteous refactorings, but I’m quietly hoping once they’re done he might find time for one or both of these bugs.

#5: Better Memory Reporting

Before you can reduce memory consumption you have to measure it.  about:memory is the critical tool that has facilitated much of MemShrink’s work.  (For example, we never would have known about zombie compartments without it.)  It’s in pretty good shape now but there are two major improvements that can be made.

First, the “heap-unclassified” number (a.k.a “dark matter”) is still typically around 20–25%.  My goal is to reduce that to 10%. This won’t require any great new insights, we already have the tools and data required.  Rather, it’s just a matter of grinding through the list of memory reporters that need to be added and improved.

Second, the resources used by each browser tab are reported in an unwieldy fashion:  JS memory on a per-compartment basis;  layout memory on a per-docshell basis;  DOM memory on a per-window basis.  Only a few internal architectural changes stand in the way of uniting these to provide the oft-requested feature of per-tab memory reporting.  This will be great for users, because if Firefox is using more memory than they’d like, it tells them which tabs they should close in order to free up memory.

I am actively working on both these improvements, and I’m hoping that within a couple of months they’ll be mostly done.

#4: Better Memory Consumption Tracking

One thing we haven’t done well in MemShrink is to improve the state of tracking Firefox’s memory consumption.  We have plenty of anecdotes but not much hard data about the improvements we’ve made, and we don’t have good ways to detect any regressions.  A couple of ideas haven’t gone very far, but some good news is that John Schoenick is making great progress on a proper areweslimyet.com implementation.  John has demonstrated preliminary versions of the site at two MemShrink meetings and it’s looking very promising.  It uses the endurance test framework to make the measurements, and opens lots of pages from the Talos tp5 pageset.

We also hope to use telemetry data to analyze how the memory consumption of each released version of Firefox stacks up.  That analysis would come with a significant delay — weeks or months after each release — but it would be much more comprehensive than any oft-run benchmark, coming from the real-world usage patterns of thousands of users.

#3: Compacting Generational GC

If you look in about:memory, JavaScript memory usage usually dominates.  In particular, the “js-gc-heap” is usually large.  There’s also the “js-gc-heap-unused-fraction” number, often 30% or higher, which tells you how much of that space is unused because of fragmentation.  That percentage overstates things somewhat, because often a good proportion of that unused space (see “js-gc-heap-decommitted”) is decommitted, which means that it’s costing nothing but address space… but that is cold comfort if you’re suffering out-of-memory aborts on Windows due to virtual memory exhaustion.

A compacting garbage collector is one that can move objects around the heap, filling up all those little gaps that constitute fragmentation.  The JS team (especially Bill McCloskey and Terrence Cole) is implementing a compacting generational garbage collector, which is a particular kind that tends to have good performance.  In particular, many objects die young and generational collectors find these quickly, which means that the heap will grow at a significantly slower rate than it currently does.  I could be wrong, but I’m convinced this will be a big win for both memory consumption and speed.

#2: Better Foreground Tab Image Handling

Images are stored in a compressed format (e.g. JPEG, PNG, GIF) on disk.  In order to display them, a browser must decompress (a.k.a decode) the compressed form into a raw pixel form that can easily be ten times larger.  This decoded form can be discarded and regenerated as necessary, and there are trade-offs to be made — for example, if you are too aggressive in discarding decoded images, you might have to decode them again, which will take CPU cycles and the user might see flickering if the decoding occurs in the visible part of the page.

However, Firefox goes way too far in the other direction.  If you open a page in the foreground tab, every single image in that page will be immediately decoded, and none of the decoded data will be discarded unless you switch away to another tab.  For pages that contain many images, this is a recipe for horrific memory consumption, and Firefox does much worse than all the other browsers.  So this is a problem that doesn’t rear its head for all users, but it’s terrible for those that are affected.

There are three MemShrink:P1 bugs relating to this:  one about not decoding all images immediately, one about discarding non-visible decoded images after some time, and one about some infrastructure work that is required for the first two.  As far as I know, no progress has been made on these three bugs, and although two of them are assigned they are not being actively worked on.

(See this discussion on the dev-platform mailing list for more details about this topic.)

#1: Better Detection and Notification of Leaky Add-ons

It’s been the case for several months that when a user complains about Firefox consuming an excessive amount of memory, it’s usually because of one or more add-ons, and the “can you try that again in safe mode?” / “oh yeah, that fixes it” dance is getting tiresome.

Many add-ons leak.  Even popular, well-written ones:  in the past few months leaks have been found in Adblock Plus, Video DownloadHelper, GreaseMonkey and Firebug.  That’s four of the top five add-ons on AMO!  We’re now getting several reports about leaky add-ons a week;  in this week’s MemShrink meeting there were four:  TorButton, NoSquint, Customize Your Web, and 1Password.  I strongly suspect the leaks we know about are just the tip of the iceberg.

Although leaks in add-ons are not Mozilla’s fault, they are Mozilla’s problem:  Firefox gets blamed for the sins of its add-ons.  And it’s not just memory consumption;  the story is the same for performance in general.  Here’s the quote of the week, from a user of 1Password:

I only use a handful of extensions and honestly never suspected 1P, however after disabling it I noticed my FireFox performance increased very noticibly. I’ve been running for 48 hours now without the 1P extension in Firefox and wow what a difference. Browsing is faster, switching is faster, memory usage is way down.

I’ve lost count of the number of stories like this that I’ve heard.  How many users have we lost to Chrome because of these issues, I wonder?

(And it’s not just leaks.  See this analysis of 16 add-ons and their effect on memory consumption when Firefox starts.)

One small step towards improving this situation was made this week:  Jorge Villalobos and Andrew Williamson added a “check for memory leaks” item to the AMO review checklist (under “Memory leaks from content”).  And Kris Maglione added some support for this checking in his Extension Test add-on.  This means that add-ons with obvious memory leaks (and many of them are obvious if you are actively looking for them) will not be accepted by AMO.

So that will prevents leaks in some new add-ons and new versions of established add-ons.  What about existing add-ons?  One idea is that AMO could also have a flag that indicates add-ons that have known memory problems (and other performance problems).  (This flag wouldn’t be an automatic thing, it would only be set once a leak has been confirmed, and after giving the author notification and some time to fix the problem.)  So that would also improve things a bit.

But lots of add-ons aren’t hosted on AMO.  Another idea is to have a stronger mechanism, one that informs the user if they have any add-ons installed that are known to cause high memory consumption (or other bad performance problems).  There is an existing mechanism for blocking add-ons that are known to be malware or exceptionally crashy, so hopefully the warnings could piggy-back on top of that.

Then, we need a better way to detect leaky add-ons.  Currently this is entirely done manually — and a couple of excellent contributors have found leaks on multiple add-ons — but I’m hoping that it’ll be possible to do a much more thorough job by analyzing telemetry data to find out which add-ons are correlated with high memory consumption.  That information could be used to trigger manual checking.

Finally, once you know an add-on leaks, it’s not always easy to work out why.  Tools could help a lot here, if they can be made to work well.

Conclusion

I listed six big areas for improvement. If we fixed all of these I think we’d be in a fantastic position.

Three of them (#5 better memory reporting, #4 better memory consumption tracking, #3 compacting generational GC) have people working on them and are in a good state.

Three of them (#6 better script handling, #2 better foreground image tab handling, #1 better detection and notification of leaky add-ons) don’t have people working on them, as far as I know.  If you are willing and have the skills to contribute to any of these areas, please contact me!

And if you think I’ve overestimated or underestimated the importance of any issue, I’d love to hear about it.  Thanks!