Category Archives: about:memory

MemShrink progress, week 53-54

Week 53-54?  We’ve been going for over a year!  To celebrate MemShrink’s 1st birthday, I wrote about the major achievements we’ve made in the past year.

Memory Reporting

I overhauled the cross-cutting “Other Measurements” shown for JavaScript in about:memory.  It’s now in two parts.  The first part is a functional breakdown of the memory consumption of the main JavaScript runtime, i.e. all JavaScript memory usage outside of web worker threads.  Here’s an example.

194,179,064 B (100.0%) -- js-main-runtime
├──159,064,360 B (81.92%) -- compartments
│  ├───73,850,880 B (38.03%) -- gc-heap
│  │   ├──22,014,080 B (11.34%) -- objects
│  │   │  ├──14,119,616 B (07.27%) ── non-function
│  │   │  └───7,894,464 B (04.07%) ── function
│  │   ├──19,553,168 B (10.07%) ── unused-gc-things
│  │   ├──14,014,208 B (07.22%) -- shapes
│  │   │  ├───6,747,240 B (03.47%) ── tree
│  │   │  ├───4,926,000 B (02.54%) ── dict
│  │   │  └───2,340,968 B (01.21%) ── base
│  │   ├──10,204,960 B (05.26%) ── strings
│  │   ├───5,338,816 B (02.75%) ── scripts
│  │   ├───1,493,472 B (00.77%) ── type-objects
│  │   ├───1,229,456 B (00.63%) ── arena-admin
│  │   └───────2,720 B (00.00%) ── sundries
│  ├───29,459,680 B (15.17%) ── analysis-temporary
│  ├───14,484,704 B (07.46%) ── script-data
│  ├───13,062,744 B (06.73%) ── string-chars
│  ├────9,300,704 B (04.79%) -- shapes-extra
│  │    ├──3,325,952 B (01.71%) ── compartment-tables
│  │    ├──2,845,984 B (01.47%) ── tree-tables
│  │    ├──2,125,056 B (01.09%) ── dict-tables
│  │    └──1,003,712 B (00.52%) ── tree-shape-kids
│  ├────7,688,912 B (03.96%) -- type-inference
│  │    ├──5,750,576 B (02.96%) ── script-main
│  │    ├──1,368,560 B (00.70%) ── object-main
│  │    └────569,776 B (00.29%) ── tables
│  ├────6,232,272 B (03.21%) -- objects
│  │    ├──4,406,720 B (02.27%) ── slots
│  │    ├──1,726,640 B (00.89%) ── elements
│  │    └─────98,912 B (00.05%) ── misc
│  ├────4,256,768 B (02.19%) ── cross-compartment-wrappers
│  └──────727,696 B (00.37%) ── mjit-data
├───19,472,384 B (10.03%) -- gc-heap
│   ├──18,006,016 B (09.27%) ── unused-arenas
│   ├───1,458,176 B (00.75%) ── chunk-admin
│   ├───────8,192 B (00.00%) ── decommitted-arenas
│   └───────────0 B (00.00%) ── unused-chunks
└───15,642,320 B (08.06%) ── runtime

The contents of the “js-main-runtime/compartments” sub-tree is the sum of all the individual compartment entries in the “explicit/js” sub-tree, and the “js-main-runtime” total is the same as the “explicit/js” total.  This new view will become more important when per-tab memory reporting is implemented which will cause each tab’s JavaScript memory consumption to be reported separately.

The second part is designed to show the utilization of the garbage-collected heap.  The new measurements and presentation are more detailed and easier to understand than the old version.  Here’s an example.

93,315,072 B (100.0%) -- js-main-runtime-gc-heap-committed
├──55,755,888 B (59.75%) -- used
│  ├──53,068,256 B (56.87%) ── gc-things
│  ├───1,458,176 B (01.56%) ── chunk-admin
│  └───1,229,456 B (01.32%) ── arena-admin
└──37,559,184 B (40.25%) -- unused
   ├──19,553,168 B (20.95%) ── gc-things
   ├──18,006,016 B (19.30%) ── arenas
   └───────────0 B (00.00%) ── chunks

One thing this makes clear is that our “unused” fraction is often very high.  We’ll need a compacting generation garbage collector to really fix that.  This is being actively worked on, but it’s a large task.

Nathan Froyd had a productive couple of weeks.

These four reporters don’t cover a great deal of memory, but that’s inevitable these days;  all the low-handing “heap-unclassified” fruit is gone, and the remaining improvements will necessarily be small.  We’re gradually closing in on the target of 10%, though I’m not sure we’ll ever get there.

Nils Maier added some more identifying information to JavaScript sandboxes, which will help identify them in about:memory.

Bas Schouten added a memory reporter for VRAM used by Azure.

Fixed Leaks

Robert O’Callahan fixed a leak involving canvas, which I won’t pretend to understand.  This also fixed bug  757749.

Cervantes Yu fixed a B2G memory leak that would leak about 1MB of memory per day.  This was found by Valgrind, under the operation of Julian Seward.

Quote of the Week

A Hacker News thread contained the following comment.

Mozilla needs to focus on making its browser consume less memory and CPU when idle. These are the things that are killing its marketshare and leading users to switch to alternatives. Every time I see initiatives like the password account manager, iOS apps that use web view, a browser based editor and what not I just scratch my head because Firefox marketshare is plummeting and the old refrain “just wait until the next version” was tired since 3.0. Instead the general response is to get defensive and list new features. Yeah and are those new features stopping the user drain? Nope. Firefox is a sluggish memory sucking heap of software that drains resources even when it isn’t being used.

I love and cherish Firefox as a product and a brand, because you have achieved a lot in your past. But now you are failing and it is depressing to see this happen.

That’s not the quote of the week, BTW;  I’ve heard variations of that many times before, and while it had some truth to it 18 months ago, that piece of conventional wisdom is now well past its use-by date.  Pleasingly, there was quite a bit of push-back in response.  My favourite comment, which is the quote of the week, was the following.

Firefox is to Ford what Chrome is to Toyota. Even if Firefox is as reliable and efficient as Chrome these days, some people still see it as being the gas-guzzling unreliable Ford vehicle of yesteryear.

A bad reputation is a difficult thing to lose.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-2/+0)
  • P2: 87 (-10/+9)
  • P3: 106 (-3/+3)
  • Unprioritized: 2 (-2/+2)

A net change of -2 P1 bugs, and -1 P2 bugs.  Not bad!

MemShrink’s 1st Birthday

A year ago today, the first meeting of the MemShrink project took place.  The project had been announced almost three months earlier, but until that first meeting it was basically just me working on it and there wasn’t much momentum.  So I now think of those three months as something of a gestation period, and the first meeting as the true birth of the project.

So, happy birthday, MemShrink!  I’ll take this opportunity to go over some of the accomplishments we’ve made in the past year.

Big Win #1: Better Javascript Heap Management

There have been two huge MemShrink improvements that stand out in my mind.  The first one happened very early on, and it consisted of two related changes, both implemented by Gregor Wagner.

In Firefox 4, the implementation of the JavaScript heap was totally overhauled.  In particular, it was segregated into compartments, each of which held (roughly speaking) the memory for a single domain.  System compartments, i.e. those for JavaScript code that implements parts of Firefox (such as the UI), tend to be long-lived.  In contrast, user compartments, i.e. those for JavaScript code in web pages, tend to be short-lived.  But memory from both system and user compartments was being co-located in 1MB chunks.  What tended to happen is that if you browsed for a long time and then closed many tabs, you’d end up with lots of 1MB chunks that were kept alive by a small amount of memory from system compartments.  This is a form of fragmentation, and it prevented Firefox from releasing lots of memory back to the operating system that it should have been able to.

Fortunately, Gregor came up with a simple fix:  don’t allow system and user compartments to share the same 1MB chunks.  To see just how effective this was, consider this visualization of the JavaScript heap after closing many tabs, without Gregor’s change.

badly fragmented JavaScript heap

Each horizontal line is a 1MB chunk, and each square is a 4KB arena.  White squares are unused arenas, and coloured squares are in use.  Arenas belong to a particular compartment all have the same colour;  there are three compartments alive, all of which are system compartments.  You can see that many 1MB chunks are kept alive by a small number of arenas.

After Gregor’s change, in this scenario the heap ended up looking like this.

non-fragmented JavaScript heap

Much better.

Gregor’s second change related to garbage collection scheduling.  In Firefox 4 the garbage collection heuristics were poorly tuned, and the garbage collector simply didn’t run frequently enough.  Many people who left Firefox open for a while came back to find that JavaScript memory consumption had ballooned, and the machine was unusable due to paging, for seconds or minutes, until the garbage collector kicked in and paging finished.  The fix was simple:  run the garbage collector occasionally, using a timer.

These two fixes of Gregor were the main reason why the first item in the Firefox 7 release notes was “drastically improved memory handling for certain use cases”.  More specifically, Firefox 7 used less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.

Big Win #2: Fewer add-on Leaks

The second huge MemShrink improvement is much more recent.  In fact it hasn’t even made it into a released version of Firefox yet.  And the path towards it was much more circuitous.

I first revamped about:memory in May of last year, and those changes made it into Firefox 6. This started giving us insight into Firefox’s memory consumption, but it wasn’t until I broke up the reporting of JS memory consumption on a per-compartment basis that it really started to make a difference.  (This change made it into Firefox 7, another reason why that release was so good from a memory consumption point of view.)

Per-compartment reporters made some blatant inefficiencies obvious, which were quickly fixed.  More importantly, people quickly noticed that sometimes compartments were present for sites that had been closed long ago.  This is a kind of memory leak that became known as a zombie compartment, and hunting them became a popular enough sport that  more than 100 zombie compartment bug reports have been filed.

Some of the zombie compartments were due to defects in Firefox itself, and these were generally fixed fairly quickly.  However, it soon became clear that the majority of them are due to add-ons.  It’s quite easy to unintentionally create zombie compartments in add-ons.  In the worst case, add-ons could leak the compartment of every single site visited.

This led to some lively discussion about how best to handle these leaks, because they are defects in third-party code that is largely out of Mozilla’s control, and yet they make Firefox look bad.  Fortunately, this discussion became mostly moot when Kyle Huey took advantage of an 8,000 mile trip from home to implement a crazy idea he’d had six months earlier –  when a tab is closed, simply cut the references from other compartments that would keep the compartment of the tab’s page alive.  This turns out to work tremendously well and prevents most of the cases where add-ons can create zombie compartments.  This change is scheduled to ship in Firefox 15, which is due for release in late August, and should more or less fix what I six months ago ranked as the single worst problem affecting Firefox’s memory consumption.

The Key Tool: Memory Reporters and About:memory

I mentioned about:memory above.  It’s worth discussing in more detail, because it’s the single most important tool we’ve created, and has driven a lot of the MemShrink improvements.

Let’s look at about:memory as it was in Firefox 6, which was released about 10 months ago.about:memory screenshot from Firefox 6

Even these early versions had several key properties.

  • about:memory doesn’t need a special build to run;  it works even in released versions.
  • It is trivially easy to use — you just type “about:memory” in the address bar.
  • Because it’s all text you can cut and paste the output, and unlike many web pages it is carefully constructed so that it reproduces beautifully when you do so.  No screenshots are necessary.

These properties massively expand the number of people who are able to detect and report problems relating to memory consumption.  Any Firefox user can potentially do it;  if the user doesn’t already know how, they can be taught in 10 seconds.

Having said that, this old version is almost laughably primitive compared to what we have now.  First, let’s consider the state of the memory reporting infrastructure, which about:memory completely relies on, as it was at that time.

  • There were a fixed number of measurements, which meant there were no individual measurements for variable-numbered entities such as compartments, window objects, or SQLite connections.  Such measurements were only possible once I added support for memory multi-reporters.  We now have dozens of measurements per variable-numbered entity, and thousands of measurements in total.
  • Even once per-compartment reporters were added, we still didn’t have much insight into the memory consumption of the JavaScript code used by Firefox itself (and its add-ons).  But compartment-per-global landed recently and gave us much finer-grained measurements, which (a) gives us some idea of how much memory is being used by JavaScript-based add-ons, and (b) gets us very close to being able to report memory consumption for each tab.  Users have been requesting both of these things for years.
  • Early memory reporters had lots of defects that caused them to report incorrect numbers, but these were obvious only when the numbers were ridiculously large or negative (both of which happened on occasion).  The first of two major improvements in correctness came about as we learned how to structure memory reporters to avoid common defects.  In particular, it’s best to write reporters that are traversal-based rather than counter-based, and to measure heap blocks with malloc_usable_size rather than computing the sizes.  Reporters that use malloc_usable_size also measure slop bytes, which are caused by the heap allocator rounding up allocation requests and can account for 5–10% of the heap.  Early reporters didn’t have these characteristics, but most of them have been converted now.
  • Reporters with these characteristics are doubly-virtuous, because they also integrate well with DMD, a tool I wrote that dynamically checks memory reports.  DMD provided the second major improvement in memory reporter correctness, because it identifies heap blocks that are counted twice, which is a mistake that’s easy to make.
  • DMD also identifies unreported heap blocks, and was crucial in getting the “heap-unclassified” number down.  In the example above it was 68.82%(!), whereas now it’s typically around 15%, which is low enough that people have mostly stopped complaining about it.  Prior to DMD, identifying this “dark matter” was much harder and less systematic.
  • The only tree shown in the above example is “explicit”.  On Linux, Justin Lebar added code to summarize the information in the /proc/<pid>/smaps file in “size”, “rss”, “pss” and “swap” trees.
  • The reporters back then could only report byte measurements.  They can now report unitless counts and percentages, both of which are useful.

Now consider the about:memory page itself, as it was back then.

  • Insignificant measurements were simply omitted, and you had to reload in verbose mode to see them (and the measurements would be re-taken when that happened, potentially losing any interesting measurements).  Now it is possible to expand and collapse arbitrary parts of the trees on the fly.
  • The “Other Measurements” section contained a flat list of measurements.  Trees are now possible, which makes that section more flexible and easier to understand.
  • about:memory’s has been carefully streamlined to consume as little memory as possible, so that it doesn’t perturb what it’s measuring.  This is really important now that we report so many measurements.
  • about:memory now has a companion page, about:compartments, that just lists compartments (and ghost windows).  about:compartments shares a lot of code with about:memory, and it is better than about:memory for certain tasks such as spotting zombie compartments.

All this stuff is important because you make what you measure.  Just about every time we’ve started measuring some new thing, we’ve found and fixed problems as a result.

Smaller Things

The following smaller accomplishments were also made in the past year.

  • John Schoenick used the MozMill endurance tests to build areweslimyet.com, which helps identify regressions in memory consumption.
  • I removed a lot of unnecessary memory waste caused by “clownshoes” allocations.
  • Olli Pettay wrote about:cc and Jan Honza Odvarko wrote about:ccdump.  These closely related tools have been used to find various cycle collector leaks.
  • I wrote 41 progress reports.  I had/have two goals with these:  to keep interested people informed of what it happening, and to provide evidence that Mozilla actually does care about memory consumption in Firefox.

This is only the tip of the iceberg, though.  As I am writing this, there have been 314 MemShrink-tagged bugs marked FIXED in Bugzilla.  In comparison, there are only 221 MemShrink-tagged bugs currently open.  Breaking them down by priority we get the following.

  • P1: 52 fixed, 25 open
  • P2: 135 fixed, 88 open
  • P3: 53 fixed, 106 open
  • Unprioritized: 74 fixed, 2 open

Over 75 people were responsible for these fixes.  I give many thanks to all those people, plus everybody who contributed in any other way to fixes, such as bug reporters and patch reviewers.  And I’d like to give extra thanks to the MemShrink regulars:  Justin Lebar, Kyle Huey, Andrew McCreight, Johnny Stenback, and Jet Villegas.

What’s Next?

There’s no real secret to MemShrink.  So far it’s basically been a long, steady grind, gradually improving tools, fixing leaks, slimming down data structures, and responding to user’s problems.  There are no plans to change that, and we’ll continue our recently-lowered pace of fortnightly meetings for the foreseeable future.  And if we’re lucky, I might be writing a report on MemShrink’s second birthday a year from now!

MemShrink progress, week 51-52

Memory Reporting

Nathan Froyd added much more detail to the DOM and layout memory reporters, as the following example shows.

├──3,268,944 B (03.76%) -- window(http://www.reddit.com/)
│  ├──1,419,280 B (01.63%) -- layout
│  │  ├────403,904 B (00.46%) ── style-sets
│  │  ├────285,856 B (00.33%) -- frames
│  │  │    ├───90,000 B (00.10%) ── nsInlineFrame
│  │  │    ├───72,656 B (00.08%) ── nsBlockFrame
│  │  │    ├───62,496 B (00.07%) ── nsTextFrame
│  │  │    ├───42,672 B (00.05%) ── nsHTMLScrollFrame
│  │  │    └───18,032 B (00.02%) ── sundries
│  │  ├────259,392 B (00.30%) ── pres-shell
│  │  ├────168,480 B (00.19%) ── style-contexts
│  │  ├────160,176 B (00.18%) ── pres-contexts
│  │  ├─────55,424 B (00.06%) ── text-runs
│  │  ├─────45,792 B (00.05%) ── rule-nodes
│  │  └─────40,256 B (00.05%) ── line-boxes
│  ├────986,240 B (01.13%) -- dom
│  │    ├──780,056 B (00.90%) ── element-nodes
│  │    ├──156,184 B (00.18%) ── text-nodes
│  │    ├───39,936 B (00.05%) ── other [2]
│  │    └───10,064 B (00.01%) ── comment-nodes
│  └────863,424 B (00.99%) ── style-sheets

Although these changes slightly improved the coverage of the reporters (i.e. they reduce “heap-unclassified” a bit), the more important effect is that they give greater insight into DOM and layout memory consumption.  [Update:  Nathan blogged about these changes.]

I modified the memory reporter infrastructure and about:memory to allow trees of measurements to be shown in the “Other Measurements” section of about:memory.  The following excerpt shows two sets of measurements that were previously shown in a flat list.

108 (100.0%) -- js-compartments
├──102 (94.44%) ── system
└────6 (05.56%) ── user

3,900,832 B (100.0%) -- window-objects
├──2,047,712 B (52.49%) -- layout
│  ├──1,436,464 B (36.82%) ── style-sets
│  ├────402,056 B (10.31%) ── pres-shell
│  ├────100,440 B (02.57%) ── rule-nodes
│  ├─────62,400 B (01.60%) ── style-contexts
│  ├─────30,464 B (00.78%) ── pres-contexts
│  ├─────10,400 B (00.27%) ── frames
│  ├──────2,816 B (00.07%) ── line-boxes
│  └──────2,672 B (00.07%) ── text-runs
├──1,399,904 B (35.89%) ── style-sheets
└────453,216 B (11.62%) -- dom
     ├──319,648 B (08.19%) ── element-nodes
     ├──123,088 B (03.16%) ── other
     ├────8,880 B (00.23%) ── text-nodes
     ├────1,600 B (00.04%) ── comment-nodes
     └────────0 B (00.00%) ── cdata-nodes

This change improves the presentation of measurements that cross-cut those in the “explicit” tree.  I intend to modify a number of the JS engine memory reports to take advantage of this change.

One interesting bug was this one, where various people were seeing multiple compartments for the same site in about:compartments, as the following excerpt shows.

https://bugzilla.mozilla.org/
https://bugzilla.mozilla.org/, about:blank

It turns out this is not a bug.  The sites in question contain an empty iframe that doesn’t specify a URL, and so the memory reporters are doing exactly the right thing.  Still, a useful case to know about when reading about:compartments.

Add-ons

Justin Lebar fixed the FUEL API, which is used by numerous add-ons including Test Pilot and PDF.js, so that it doesn’t leak most of the objects it creates.  This leak caused 10+ minute shut-down times(!) for one user, so it’s a good one to have fixed.

Kyle Huey briefly broke his own Hueyfix, and then fixed it.  You had us worried for a bit there, Kyle!

asgerklasker fixed a leak in the HttpFox add-on that caused zombie compartments.

Miscellaneous

I restricted the amount of context that is shown in JavaScript error messages.  This fixed a longstanding bug where if you had enabled the javascript.options.strict option in about:config, and you had the error console open or Firebug installed, memory consumption would spike dramatically when viewing pages that contain JavaScript code that triggered many strict warnings.  This bug manifested rarely, but would bring Firefox to its knees when it did.

Asaf Romano fixed a leak that caused a zombie compartment if you used the “Highlight All” checkbox when doing a text search in a page.

Mike Hommey finished importing the new version of jemalloc into the Mozilla codebase, and then wrote about it.  The new version is currently disabled, but we hope to turn it on soon.  Preliminary experiments indicate that the new version is unlikely to affect performance or memory consumption much.  However, it will be good to be on a version that is in sync with the upstream version, instead of having a highly hacked Mozilla-specific version.

Justin Lebar wrote a nice blog post explaining what “ghost windows” are, and how we’ve used them to gauge some recent leak fixes.

Till Schneidereit fixed a garbage collection defect that could cause memory usage to spike in certain cases involving Firefox Sync.

LifeHacker published a new browser performance comparison which looked at Firefox 13, Chrome 19, IE9, and Opera 11.64.  Firefox did the best overall and also won both the memory consumption tests.  I personally take these kinds of comparisons with a huge bucket of salt.  Indeed, quoting from the article:

Our tests aren’t the most scientific on the planet, but they do reflect a relatively accurate view of the kind of experience you’d get from each browser, speed-wise.

If you ask me, the text before the “but” contradicts the rest of the sentence.  What I found more interesting was the distinct lack of “lolwut everyone knows Chrome is faster than Firefox” comments, which I’m used to seeing when Firefox does well in these kinds of articles.

Bug Counts

Here are the current bug counts.

  • P1: 25 (-1/+3)
  • P2: 88 (-3/+6)
  • P3: 106 (-0/+4)
  • Unprioritized: 2 (-2/+2)

Nothing too exciting there.

MemShrink progress, week 49-50

System Compartment Reporting

With the recent landing of compartment-per-global, Firefox now regularly has 200+ system compartments at start-up.  However, most of these compartments didn’t have names, which meant that they were merged into a single “[System Principal]” entry in about:memory and about:compartments.

Until last week, that is, when Nils Maier added identifying information to the vast majority of these.  Here’s a small selection of interesting ones from about:compartments on my own machine.

about:blank
about:compartments
about:memory?verbose

chrome://adblockplus-modules/content/FilterClasses.jsm

chrome://browser/content/bookmarks/bookmarksPanel.xul
chrome://browser/content/browser.xul
chrome://browser/content/places/menu.xml
chrome://browser/content/search/search.xml

chrome://chatzilla/content/browserOverlay.xul

chrome://global/content/bindings/button.xml
chrome://global/content/globalOverlay.xul

chrome://treestyletab/content/treestyletab.xul

file:///home/njn/moz/mi0/o64/dist/bin/components/TelemetryPing.js

jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/bootstrap.js
jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/components/PdfStreamConverter.js

resource:///modules/TelemetryTimestamps.jsm
resource:///modules/sessionstore/DocumentUtils.jsm

resource://gre-resources/hiddenWindow.html

resource://gre/modules/AddonManager.jsm

resource://services-common/preferences.js
resource://services-crypto/WeaveCrypto.js
resource://services-sync/constants.js
resource://services-sync/engines/bookmarks.js

resource://treestyletab-modules/browser.js
resource://treestyletab-modules/lib/animationManager.js

Just from this, it’s obvious that I had about:compartments and about:memory?verbose open at the time.  It’s also obvious that I had the following add-ons installed:  AdBlock Plus, Chatzilla, Tree Style Tab, and pdf.js.  And about:memory now gives at least a partial measurement of how much memory these add-ons are using.  This will help identify add-ons that are using excessive amounts of memory.  (Having said that, I identified the add-on compartments simply by their names.  It’d be great if there was a way to systematically identify them within the code, but I don’t know if that’s possible.)

I also hope people will scrutinize Firefox’s own compartment use closely, and start to file bug reports saying things like “hey, that .jsm module shouldn’t be present, there must be a leak”.  If you want to see what the full list of 200+ looks like, try out a recent Nightly build!

In related news, I also added some new compartment-specific reports, including ones for cross-compartment wrappers.

One consequence of all these change is that the number of entries in about:memory jumped tremendously.  As a result, I aggregated the small entries within each compartment, which reduces the number of entries by a factor of roughly four while still reporting full information for large compartments.  Nils and I also made about:memory more efficient, so the amount of memory required to generate each line dropped by about 20%.  about:memory still takes up memory itself, but it does so at a level that I’m fairly happy with.

Add-ons

For a change, the biggest MemShrink-related news in this report wasn’t related to add-ons!  But there was still some interesting movement there.

Justin Lebar uncovered some evidence that the Hueyfix is having a real, positive effect among users.  Telemetry data from Nightly users shows that the number of ghost windows — a concept for which we don’t have good documentation, but they correlate with zombie compartments — has dropped dramatically, as the following graph shows.

ghost windows telemetry data graph

Telemetry data tends to be extremely noisy, so it’s nice to see a clear signal — Kyle’s change made it into Nightly builds on May 5th [Update: that's incorrect, see below] and immediately caused the mean number of ghost windows to drop from roughly three to roughly one.  The variance also dropped dramatically.

Update: Justin just wrote a blog post that explains very nicely what ghost windows are.  That post also explains better the circumstances behind the drop in ghost window numbers;  my explanation above was too simple and got the timing wrong.  Thanks, Justin!

In other add-on news, the following add-ons had leaks fixed: Readability, ProxTube, Youtube MP3 Podcaster.

Firefox vs The New York Times

Robert O’Callahan fixed a leak relating to mouse events that triggered when he visited nytimes.com.  He wrote a great blog post explaining the heroic debugging — searching through full memory dumps! — that was required.  It’s great that Robert found and fixed this, though it’s a shame it took such expertise.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-1/+2)
  • P2: 85 (-2/+4)
  • P3: 102 (-5/+2)
  • Unprioritized: 2 (-3/+2)

Not a great deal of movement.  We only had to triage twelve bugs in today’s MemShrink meeting, which is the fewest we’ve had since we switched to fortnightly meetings.

MemShrink progress, week 42

 Are we slim yet?

John’s Schoenick’s areweslimyet.com (AWSY) has had its password removed and is now open to the public!

areweslimyet.com screenshot

This is a major milestone for the MemShrink project.  It shows the progress we have made (MemShrink started in earnest in June 2011) and will let us identify regressions more easily.

John’s done a wonderful job implementing the site.  It has lots of functionality:  there are multiple graphs, you can zoom in on parts of graphs to see more detail, and you can see revisions, dates and about:memory snapshots for individual runs.

John has also put in a great deal of work refining the methodology to the point where we believe it provides a reasonable facsimile of real-world browsing;  please read the FAQ to understand exactly what is being measured.  Many thanks also to Dave Hunt and the QA team for their work on the Mozmill Endurance Tests, which are at the core of AWSY’s testing.

Update: Hacker News has reported on this.

Ghost windows

Frequent readers of this blog will be familiar with zombie compartments, which are JavaScript compartments that have leaked, due to defects in Firefox or add-ons.  Windows (i.e. window objects) can also be leaked, and often defects that cause compartments leaks will cause window leaks as well.

Justin Lebar has introduced the notion of “ghost windows”.  A ghost window is one that meets the following criteria.

  1. Shows up in about:memory under “window-objects/top(none)”.
  2. Does not share a domain name with any window under “window-objects/active.
  3. Has met criteria (1) and (2) for a moderate amount of time (e.g. two minutes).

The basic idea is that a ghost window has a high chance of representing a genuine leak, and this automated identification of suspicious windows will make leak detection simpler.  Justin has added ghost window tracking to about:memory, about:compartments, and telemetry.  (These three bugs were all marked as MemShrink:P1.)  Ghost window tracking is mostly untested right now, but hopefully it will become another powerful tool for finding memory leaks.

Add-ons

We’ve been tracking leaky add-ons in Bugzilla for a while now, but we’ve never had a good product/component to put them in.  David Lawrence, Byron Jones, Stormy Peters and I have together created a new “Add-ons” component under the “Tech Evangelism” product.  The rationale for putting it under “Tech Evangelism” is that it nicely matches the existing meaning of that phrase — it’s a case where a non-Mozilla entity is writing defective code that interacts with Firefox and hurts users’ experiences with and perceptions of Firefox, and Mozilla can only inform, educate and encourage fixes in that defective code.  This component is only intended for certain classes of common defects (such as leaks) that Mozilla contributors are tracking.  It is not intended for vanilla add-on bugs;  as now, they should be reported through whatever bug-reporting mechanism each add-on uses.  I’ve updated the existing open bugs that track leaky add-ons to use this new component.

Leaks in the following add-ons were fixed:  Video DownloadHelper (the 2nd most popular add-on on AMO!), Scrapbook Plus, Amazon Price Tracker.

Bug counts

This week’s bug counts:

  • P1: 21 (-5/+0)
  • P2: 137 (-1/+7)
  • P3: 90 (-1/+3)
  • Unprioritized: 1 (-1/+1)

Good progress on the P1 bugs!

A new reporting schedule

Many of the weekly MemShrink reports lately have been brief.  From now on I plan to write a report every two weeks.  This will make things easier for me and will also ensure each report is packed full of interesting things.  See you again in two weeks!

MemShrink progress, week 39

Add-ons

Versions of McAfee’s SiteAdvisor add-on prior to 3.4.1.195 have been soft-blocked due to an extreme memory leak.  This means that it will be disabled within Firefox, but users can re-enable it if they want to.  According to McAfee, most users should have been upgraded to 3.4.1.195 by now.  Unfortunately, version 3.4.1.195 still has some leaks, and progress on fixing them has stalled.  (If SiteAdvisor were an AMO add-on there’s a good chance it would have been downgraded to “preliminarily reviewed” by now.)

Leaks in the Lastpass, Fireshot, and Amabay add-ons have been fixed.

Miscellaneous

Two weeks ago I mentioned that the not-yet-public areweslimyet.com detected a large regression in memory consumption caused by incremental garbage collection.  Bill McCloskey adjusted the GC and CC heuristics in order to fix this.

Justin Lebar previously added monitoring of available physical and virtual memory monitoring on Windows;  if either goes below a threshold Firefox attempts to free up memory.  This week he added monitoring of available commit space.

When you switch to a new tab, decoded image data from the old tab is kept until a 20 second timer expires.  This is a good idea because you might switch back to the original tab quickly.  However, until this week, this don’t-discard-it-immediately behaviour was also used when a tab is closed!  Justin Lebar landed a patch which separates these two cases, i.e. it discards decoded image data immediately when a tab is closed.

I tweaked the presentation of the “window-objects” sub-tree in about:memory so that the memory used within each browser tab is more obvious.  This is another step towards true per-tab memory reporting.  Here’s an example for a tab in which I navigated first to valgrind.org and then to www.mozilla.org;  the URL within the “top(…)” line is the one shown in the address bar.

│   ├──1,933,824 B (02.68%) -- top(http://www.mozilla.org/, id=13)
│   │  ├──1,497,480 B (02.08%) -- active
│   │  │  ├──1,496,456 B (02.08%) -- window(http://www.mozilla.org/)
│   │  │  │  ├────876,912 B (01.22%) -- layout
│   │  │  │  │    ├──569,456 B (00.79%) ── arenas
│   │  │  │  │    ├──238,224 B (00.33%) ── style-sets
│   │  │  │  │    └───69,232 B (00.10%) ── text-runs
│   │  │  │  ├────316,216 B (00.44%) ── style-sheets
│   │  │  │  └────303,328 B (00.42%) ── dom [2]
│   │  │  └──────1,024 B (00.00%) -- window([system])
│   │  │         └──1,024 B (00.00%) ── dom
│   │  └────436,344 B (00.61%) -- cached
│   │       └──436,344 B (00.61%) -- window(http://valgrind.org/)
│   │          ├──312,496 B (00.43%) -- layout
│   │          │  ├──170,448 B (00.24%) ── style-sets
│   │          │  ├──125,040 B (00.17%) ── arenas
│   │          │  └───17,008 B (00.02%) ── text-runs
│   │          ├───76,600 B (00.11%) ── dom
│   │          └───47,248 B (00.07%) ── style-sheets

Bug counts

This week’s bug counts:

  • P1: 24 (-5/+0)
  • P2: 132 (-2/+3)
  • P3: 92 (-1/+6)
  • Unprioritized: 0 (-1/+0)

We only had to triage five bugs in today’s MemShrink meeting.  I don’t remember ever having such a small number.  As a result, we spent some time reviewing the P1 bugs, and decided that several could be downgraded to P2 because the situation had improved in some fashion.

MemShrink progress, week 38

After last week’s action, this week was quieter for MemShrink.

Add-ons

I mentioned last week the plan to test the top 100 add-ons for memory leaks.  Unfortunately we learned this week that the list I gathered is the top 100 installed add-ons, not the top 100 enabled add-ons.  It’s quite possible that a lot of installed “third-party” add-ons (those installed by a program outside of Firefox, such as anti-virus programs) are disabled, in which case this top 100 list won’t reflect actual usage.

As a result, the top 100 testing is on hold until we can resolve this issue.  Telemetry data may help, though that data might exhibit opt-in biases.  I’m also going to investigate the possibility of changing the daily ping to distinguish between enabled and disabled add-ons, which would give us fully representative data.

Landed patches

I merged the “dom+style” and “layout” trees in about:memory.  This is a step towards per-tab memory reporting.  It also probably breaks about:nosy :(

I also converted the DOM memory reporters to the new style, added measurement of URIs and links, and added measurement of the FramePropertyTable.  These changes reduced about:memory’s “heap-unclassified” by several MBs in common cases.

Bill McCloskey changed the GC marker stack so it shrinks periodically.  Previously it could grow from 256KB to over 2MB and would never be shrunk once that happened.

Bug counts

This week’s bug counts:

  • P1: 29 (-0/+1)
  • P2: 131 (-4/+6)
  • P3: 87 (-3/+7)
  • Unprioritized: 1 (-2/+1)

MemShrink progress, week 37

It’s been a huge week for MemShrink.

Add-ons

I filed a bug on testing the top 100 add-ons for memory leaks.  The majority of these add-ons are not available on AMO and so are not subject to the leak checking done during AMO reviews.  The instructions and current test results are here.  Randell Jesup helped identify some some add-ons with unclear identities, and Nils Maier and Archaeopteryx have both tested several add-ons.  A month ago I said that add-ons that leak are the #1 problem for MemShrink, and this bug represents a huge step towards reducing that problem.  We’ll need lots of additional help to test this many add-ons, so please feel free to jump in!

As far as I know, comprehensive testing of the most popular add-ons has never been done before.  Although this testing is aimed at finding memory leaks, it’s quite possible it will uncover various other problems with add-ons.  Indeed, it’s clear just from looking at the list that quite a few of the non-AMO add-ons are of dubious merit, and this has stimulated some interesting discussion on dev-platform about third-party add-ons, i.e. those that are installed into Firefox by an external program.

The following add-ons had leaks fixed or mostly fixed by their authors: McAfee Site Advisor, Ghostery, Spool, Screengrab (fix version), Roboform, UF Comment Board ToolsGeenstijl, Long URL Please, HTML Desktop Notifications.

Nils Maier greatly improved the documentation on common causes of memory leaks in add-ons.  This was a MemShrink:P1 bug.  That doesn’t mean the document can’t be improved further, however;  it’s a wiki so please continue to add to it if you can.

Regression testing

John Schoenick’s excellent areweslimyet.com is getting close to being ready for a public unveiling.  The site is currently live but password-protected, simply to prevent large numbers of people accessing it before it’s ready.  (If you want a sneak peek, contact me for the password.)  More importantly, it identified a large regression in memory consumption caused by the landing of the JS engine’s incremental garbage collector on February 19, which Bill McCloskey is investigating.

Tim Taubert improved Firefox’s regression testing to ensure that no new DocShell and DOMWindow leaks are introduced.  This was a MemShrink:P1 bug because these leaks are quite easy to introduce.  Tim wrote some more about this change here.

Cycle collector

Jan Honza Odvarko’s about:ccdump add-on made it onto AMO this week.  Jan has been using this to hunt down some tricky leaks in Firebug.

Marco Bonardo fixed a leak in PlacesUtils.removeLazyBookmarkObserver() that I won’t pretend to understand, but which was a MemShrink:P1.  Marco also fixed a leak that occurred after bookmarking a page, closing the window, and opening a new window.

Olli Pettay fixed an equally impenetrable (to me) leak involving nsFormFillController::mFocusedInput.

Kyle Huey fixed a leak in the About Firefox dialog.

Andrew McCreight added additional information to cycle collector’s error console logging, which helps with identifying and debugging cycle collector problems.

Miscellaneous

Jason Duell fixed an obscure leak involve web sockets.

Bill McCloskey fixed a minor JS engine leak.

I added a new memory reporter called “js/runtime/gc-marker”.  It starts out at 256KB and I saw it reach 2.8MB when running MemBench.  I also converted the existing source image reporters to the new style, which seems to haved fixed some bogus results in about:memory as a side-effect.

Quote of the week

Adam Overa of Tom’s Hardware did another browser Grand Prix.  Devin Coldewey used the results from the Grand Prix to conclude that Firefox and Chrome are both great, and which you prefer basically comes down to a matter of taste.  While that is an interesting conclusion, what caught my eye was this comment, by Balaji Viswanathan:

I guess the title could have been “All browsers suck equally bad now”. Firefox and Chrome have grown to become more resources hogs – closer to IE. These days I shudder to fire up the firefox as the memory leaks are terrible. Two hours of Firefox with a Facebook tab is enough to gobble up 25% of my memory. With Chrome, I would have 20 rendering processes showing up when I would have just 3 tabs open.

In the first 2 years of Chrome I had probably 2 or 3 crashes. These days, it crashes every week or so. I long for the 2006 era Firefox – clean, simple and lean. Its been a long time since I have used IE actively and with Safari lesser said the better. For the developers, we have to deal with the ultra slow update cycles of IE and ultraspeed update cycles of Chrome. In short, we are getting worse and worse in the browser arena. The market is ripe for a breakthrough.

“2006 era Firefox” would have been Firefox 1.5 or Firefox 2.  Balaji may think he’s nostalgic for browsers of that era, but really he’s nostalgic for the web of that era, which was massively simpler than the web of today.  If Balaji tried Firefox 1.5 or Firefox 2 today — with no HTML5 support, no JavaScript JIT and no hardware acceleration — today, I’m sure he’d quickly decide that 2012-era browsers aren’t so bad after all.  I guess the interesting question is this: why do people blame browsers instead of websites for high resource consumption?

Bug counts

This week’s bug counts:

  • P1: 28 (-5/+5)
  • P2: 129 (-12/+7)
  • P3: 83 (-8/+12)
  • Unprioritized: 2 (-2/+2)

That’s a net reduction of one bug.  But more importantly, there is lots of movement, which is great!  It means that problems are being identified and fixed.

(Finally, commenters please note that I’ve turned on the WP-reCAPTCHA plug-in, and this the first post I’ve written since doing so.  In my testing it’s worked fairly well.  Hopefully it won’t cause problems.)

MemShrink progress, week 36

Lots of activity this week.

Leaky add-ons

I created a new devmo wiki page documenting common causes of memory leaks in add-ons.  I based this on a lovely selection of examples provided by Nils Maier.  Unfortunately, I know almost nothing about writing add-ons and so I’m not happy with the current state of the documentation.  I’ve taken it as far as I can, but it very likely has errors, leaves out important cases, conflates distinct concepts, and generally is not as good as it should be.  This documentation is really important — we have good tools for identifying when add-on leaks occur but we don’t yet have good tools for identifying their causes.  Until we have those tools, documentation is the only way we can help add-on authors fix leaks.  If someone who knows more about add-ons is willing to help please contact me!

Alexandre Poirot fixed a leak in the Add-on SDK that was causing zombie compartments when certain add-ons were disabled.  This also fixed bug 725603 and probably fixes some zombie compartments reported in some other add-ons (e.g. MemChaser).

Jan fixed several leaks in the Galaxytoolbar add-on.  The new version is 2.6.12.

A small number of add-ons have been found that disable Firefox’s XUL cache.  This is a recipe for disastrous performance, and so Matt Basta updated the AMO add-on validator to detect this.

Tools

I landed support for a new page called about:compartments. The exposure of zombie compartments in about:memory has become an extremely powerful leak detection tool.  But using about:memory to find zombies is non-trivial — there are multiple steps and ways to get it wrong.  The motivation for about:compartments is to make the detection of zombie compartments as simple as possible.  Here’s a screenshot:

about:compartments screenshot

Some things to note:

  • It only shows compartments, and all of them.
  • User compartments from web content (usually the interesting ones) are listed separately from system compartments.
  • The garbage and cycle collectors are automatically run when the page is loaded, ensuring that dead compartments aren’t listed.
  • The “More verbose” link at the bottom just causes the truncated URLs to be shown in full.
  • about:compartments shares a lot of code with about:memory.
  • I updated the documentation on zombie compartments accordingly.

Olli Pettay created about:cc, an add-on that can detect various kinds of document leaks relating to cycle collection.  Olli has already found several leaks with this tool.  Jan Honza Odvarko has written about:ccdump, a similar but prettier add-on.  I’m not certain but I think Olli and Jan are now co-ordinating their efforts.

I mentioned Mozilla QA’s MemChaser add-on a few weeks ago.  It lets you track memory usage and GC/CC activity easily via the add-on bar, and also allows logging of memory-related activities.  It’s now available on AMO, which guarantees you’ll receive updates as they’re released.

Jesse Ruderman tweaked his DOM fuzzer and found several new small leaks relating to nsITimers.  Jesse also modified about:memory so that it produces assertions that his fuzzer will catch.  This means that his fuzzer will detect if any memory reporters produce bogus values.

Miscellaneous

Kyle Huey fixed a zombie compartment that occurred when searching within pages with onclick handlers, which is a pretty common operation.

Jeff Muizelaar fixed a huge Mac-only leak relating to text rendering.  This was a recent regression that isn’t present in any released version of Firefox.

Josh Aas fixed a bug in cookie clearing — prior to Josh’s fix, if you cleared all cookies, Firefox would launch an instance of the plugin-container process for every plug-in installed, which could cause freezes and memory spikes.

Andrew Quartey updated the WebGL memory reporters to the new style.

Bug counts

This week’s bug counts:

  • P1: 28 (-1/+3)
  • P2: 134 (-6/+9)
  • P3: 79 (-8/+11)
  • Unprioritized: 2 (-1/+2)

Lots of movement there:  plenty of bugs fixed, but even more new ones.  Roughly 20 of the new ones fell into the following three categories.

  • Zombie compartments found in AMO add-ons by Kris Maglione and Andreas Wagner.
  • Leaks found by Olli Pettay’s new about:cc tool, mentioned above.
  • Leaks found by Jesse Ruderman’s tweaked DOM fuzzer, mentioned above.

This is a good thing!  It shows that new policies and tools are exposing existing problems.  I expect this higher level of new bug filing will continue for a couple of weeks.

MemShrink progress, week 35

Add-ons

Zombie compartments were fixed in the following add-ons:  Customize Your Web (fixed by Rudolf Noe), GridTube (fixed by Costas B.), Do Not Track Plus (fixed by kiran).

Marco Bonardo changed the Places JS services in a way that prevents certain kinds of SQLite connection leaks.  As far as I can tell, this fixes leaks in the Delicious Bookmarks and CyberSearch add-ons.

Finally, Alexandre Poirot fixed a bug in the Add-on SDK that was causing zombie jetpack compartments in some cases when add-ons were disabled.

Memory Reporting

Hugh Nougher added a memory reporter for GPU memory on Windows 7 and Vista.  See the “gpu-committed”, “gpu-dedicated” and “gpu-shared” entries in about:memory’s “Other Measurements” list.

I reduced the amount of memory allocated when generating about:memory by about 35%.

Miscellaneous

Josh Aas implemented unloading for out-of-process plug-ins.  If one is unused for 3 minutes it will be unloaded.

Andrew McCreight improved cycle collector dumping, which is useful in certain debugging cases.

Quote of the week

LifeHacker did some browser performance tests.  Firefox 10 handily won the memory usage test, which involved loading sites in 9 tabs and then measuring.  I personally think this is a pretty meaningless test, but I won’t complain about the good press.

As usual, the comments on the article featured a lot of debate about  whether Firefox is a memory hog or not.  One comment particular caught my attention:

They hardly used to be myths. I saw several times when old FF2x would be taking up over 1.4GB of RAM with just half a dozen tabs open.

Firefox 2 was released in October 2006, and superseded by Firefox 3 in June 2008.  Bad reputations are really difficult to shake.

Bug counts

This week’s bug counts:

  • P1: 26 (-0/+4)
  • P2: 131 (-6/+3)
  • P3: 76 (-2/+3)
  • Unprioritized: 1 (-2/+1)