Category Archives: add-ons

MemShrink progress, week 51-52

Memory Reporting

Nathan Froyd added much more detail to the DOM and layout memory reporters, as the following example shows.

├──3,268,944 B (03.76%) -- window(http://www.reddit.com/)
│  ├──1,419,280 B (01.63%) -- layout
│  │  ├────403,904 B (00.46%) ── style-sets
│  │  ├────285,856 B (00.33%) -- frames
│  │  │    ├───90,000 B (00.10%) ── nsInlineFrame
│  │  │    ├───72,656 B (00.08%) ── nsBlockFrame
│  │  │    ├───62,496 B (00.07%) ── nsTextFrame
│  │  │    ├───42,672 B (00.05%) ── nsHTMLScrollFrame
│  │  │    └───18,032 B (00.02%) ── sundries
│  │  ├────259,392 B (00.30%) ── pres-shell
│  │  ├────168,480 B (00.19%) ── style-contexts
│  │  ├────160,176 B (00.18%) ── pres-contexts
│  │  ├─────55,424 B (00.06%) ── text-runs
│  │  ├─────45,792 B (00.05%) ── rule-nodes
│  │  └─────40,256 B (00.05%) ── line-boxes
│  ├────986,240 B (01.13%) -- dom
│  │    ├──780,056 B (00.90%) ── element-nodes
│  │    ├──156,184 B (00.18%) ── text-nodes
│  │    ├───39,936 B (00.05%) ── other [2]
│  │    └───10,064 B (00.01%) ── comment-nodes
│  └────863,424 B (00.99%) ── style-sheets

Although these changes slightly improved the coverage of the reporters (i.e. they reduce “heap-unclassified” a bit), the more important effect is that they give greater insight into DOM and layout memory consumption.  [Update:  Nathan blogged about these changes.]

I modified the memory reporter infrastructure and about:memory to allow trees of measurements to be shown in the “Other Measurements” section of about:memory.  The following excerpt shows two sets of measurements that were previously shown in a flat list.

108 (100.0%) -- js-compartments
├──102 (94.44%) ── system
└────6 (05.56%) ── user

3,900,832 B (100.0%) -- window-objects
├──2,047,712 B (52.49%) -- layout
│  ├──1,436,464 B (36.82%) ── style-sets
│  ├────402,056 B (10.31%) ── pres-shell
│  ├────100,440 B (02.57%) ── rule-nodes
│  ├─────62,400 B (01.60%) ── style-contexts
│  ├─────30,464 B (00.78%) ── pres-contexts
│  ├─────10,400 B (00.27%) ── frames
│  ├──────2,816 B (00.07%) ── line-boxes
│  └──────2,672 B (00.07%) ── text-runs
├──1,399,904 B (35.89%) ── style-sheets
└────453,216 B (11.62%) -- dom
     ├──319,648 B (08.19%) ── element-nodes
     ├──123,088 B (03.16%) ── other
     ├────8,880 B (00.23%) ── text-nodes
     ├────1,600 B (00.04%) ── comment-nodes
     └────────0 B (00.00%) ── cdata-nodes

This change improves the presentation of measurements that cross-cut those in the “explicit” tree.  I intend to modify a number of the JS engine memory reports to take advantage of this change.

One interesting bug was this one, where various people were seeing multiple compartments for the same site in about:compartments, as the following excerpt shows.

https://bugzilla.mozilla.org/
https://bugzilla.mozilla.org/, about:blank

It turns out this is not a bug.  The sites in question contain an empty iframe that doesn’t specify a URL, and so the memory reporters are doing exactly the right thing.  Still, a useful case to know about when reading about:compartments.

Add-ons

Justin Lebar fixed the FUEL API, which is used by numerous add-ons including Test Pilot and PDF.js, so that it doesn’t leak most of the objects it creates.  This leak caused 10+ minute shut-down times(!) for one user, so it’s a good one to have fixed.

Kyle Huey briefly broke his own Hueyfix, and then fixed it.  You had us worried for a bit there, Kyle!

asgerklasker fixed a leak in the HttpFox add-on that caused zombie compartments.

Miscellaneous

I restricted the amount of context that is shown in JavaScript error messages.  This fixed a longstanding bug where if you had enabled the javascript.options.strict option in about:config, and you had the error console open or Firebug installed, memory consumption would spike dramatically when viewing pages that contain JavaScript code that triggered many strict warnings.  This bug manifested rarely, but would bring Firefox to its knees when it did.

Asaf Romano fixed a leak that caused a zombie compartment if you used the “Highlight All” checkbox when doing a text search in a page.

Mike Hommey finished importing the new version of jemalloc into the Mozilla codebase, and then wrote about it.  The new version is currently disabled, but we hope to turn it on soon.  Preliminary experiments indicate that the new version is unlikely to affect performance or memory consumption much.  However, it will be good to be on a version that is in sync with the upstream version, instead of having a highly hacked Mozilla-specific version.

Justin Lebar wrote a nice blog post explaining what “ghost windows” are, and how we’ve used them to gauge some recent leak fixes.

Till Schneidereit fixed a garbage collection defect that could cause memory usage to spike in certain cases involving Firefox Sync.

LifeHacker published a new browser performance comparison which looked at Firefox 13, Chrome 19, IE9, and Opera 11.64.  Firefox did the best overall and also won both the memory consumption tests.  I personally take these kinds of comparisons with a huge bucket of salt.  Indeed, quoting from the article:

Our tests aren’t the most scientific on the planet, but they do reflect a relatively accurate view of the kind of experience you’d get from each browser, speed-wise.

If you ask me, the text before the “but” contradicts the rest of the sentence.  What I found more interesting was the distinct lack of “lolwut everyone knows Chrome is faster than Firefox” comments, which I’m used to seeing when Firefox does well in these kinds of articles.

Bug Counts

Here are the current bug counts.

  • P1: 25 (-1/+3)
  • P2: 88 (-3/+6)
  • P3: 106 (-0/+4)
  • Unprioritized: 2 (-2/+2)

Nothing too exciting there.

MemShrink progress, week 49-50

System Compartment Reporting

With the recent landing of compartment-per-global, Firefox now regularly has 200+ system compartments at start-up.  However, most of these compartments didn’t have names, which meant that they were merged into a single “[System Principal]” entry in about:memory and about:compartments.

Until last week, that is, when Nils Maier added identifying information to the vast majority of these.  Here’s a small selection of interesting ones from about:compartments on my own machine.

about:blank
about:compartments
about:memory?verbose

chrome://adblockplus-modules/content/FilterClasses.jsm

chrome://browser/content/bookmarks/bookmarksPanel.xul
chrome://browser/content/browser.xul
chrome://browser/content/places/menu.xml
chrome://browser/content/search/search.xml

chrome://chatzilla/content/browserOverlay.xul

chrome://global/content/bindings/button.xml
chrome://global/content/globalOverlay.xul

chrome://treestyletab/content/treestyletab.xul

file:///home/njn/moz/mi0/o64/dist/bin/components/TelemetryPing.js

jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/bootstrap.js
jar:file:///home/njn/moz/mi0/o64/dist/bin/extensions/uriloader@pdf.js.xpi!/components/PdfStreamConverter.js

resource:///modules/TelemetryTimestamps.jsm
resource:///modules/sessionstore/DocumentUtils.jsm

resource://gre-resources/hiddenWindow.html

resource://gre/modules/AddonManager.jsm

resource://services-common/preferences.js
resource://services-crypto/WeaveCrypto.js
resource://services-sync/constants.js
resource://services-sync/engines/bookmarks.js

resource://treestyletab-modules/browser.js
resource://treestyletab-modules/lib/animationManager.js

Just from this, it’s obvious that I had about:compartments and about:memory?verbose open at the time.  It’s also obvious that I had the following add-ons installed:  AdBlock Plus, Chatzilla, Tree Style Tab, and pdf.js.  And about:memory now gives at least a partial measurement of how much memory these add-ons are using.  This will help identify add-ons that are using excessive amounts of memory.  (Having said that, I identified the add-on compartments simply by their names.  It’d be great if there was a way to systematically identify them within the code, but I don’t know if that’s possible.)

I also hope people will scrutinize Firefox’s own compartment use closely, and start to file bug reports saying things like “hey, that .jsm module shouldn’t be present, there must be a leak”.  If you want to see what the full list of 200+ looks like, try out a recent Nightly build!

In related news, I also added some new compartment-specific reports, including ones for cross-compartment wrappers.

One consequence of all these change is that the number of entries in about:memory jumped tremendously.  As a result, I aggregated the small entries within each compartment, which reduces the number of entries by a factor of roughly four while still reporting full information for large compartments.  Nils and I also made about:memory more efficient, so the amount of memory required to generate each line dropped by about 20%.  about:memory still takes up memory itself, but it does so at a level that I’m fairly happy with.

Add-ons

For a change, the biggest MemShrink-related news in this report wasn’t related to add-ons!  But there was still some interesting movement there.

Justin Lebar uncovered some evidence that the Hueyfix is having a real, positive effect among users.  Telemetry data from Nightly users shows that the number of ghost windows — a concept for which we don’t have good documentation, but they correlate with zombie compartments — has dropped dramatically, as the following graph shows.

ghost windows telemetry data graph

Telemetry data tends to be extremely noisy, so it’s nice to see a clear signal — Kyle’s change made it into Nightly builds on May 5th [Update: that's incorrect, see below] and immediately caused the mean number of ghost windows to drop from roughly three to roughly one.  The variance also dropped dramatically.

Update: Justin just wrote a blog post that explains very nicely what ghost windows are.  That post also explains better the circumstances behind the drop in ghost window numbers;  my explanation above was too simple and got the timing wrong.  Thanks, Justin!

In other add-on news, the following add-ons had leaks fixed: Readability, ProxTube, Youtube MP3 Podcaster.

Firefox vs The New York Times

Robert O’Callahan fixed a leak relating to mouse events that triggered when he visited nytimes.com.  He wrote a great blog post explaining the heroic debugging — searching through full memory dumps! — that was required.  It’s great that Robert found and fixed this, though it’s a shame it took such expertise.

Bug Counts

Here are the current bug counts.

  • P1: 23 (-1/+2)
  • P2: 85 (-2/+4)
  • P3: 102 (-5/+2)
  • Unprioritized: 2 (-3/+2)

Not a great deal of movement.  We only had to triage twelve bugs in today’s MemShrink meeting, which is the fewest we’ve had since we switched to fortnightly meetings.

MemShrink progress, week 47–48

Add-ons

The main news in the past two weeks has been about Kyle Huey’s patch that prevents most chrome-to-content leaks, which are the most common kind of add-on leak.  Testing showed it worked beautifully, but caused a knock-on leak in add-ons built with old versions (1.3 and earlier) of the Add-on SDK(This received a lot of attention.)  Kyle then made a slight tweak that fixed that knock-on leak.  So we’re currently still on track for Firefox 15 being “the one that fixes add-on leaks”.

For completeness, here are the add-ons that we know were temporarily affected by that knock-on leak:  Wallflower, Visual Hashing, Translate This!, Easy YouTube Video Downloader.  They (and probably quite a few others) are all working fine again now.

Here’s a quote from an email that one user sent to Kyle this week.

Firefox was leaking about 1.5GB per hour for me. It started with Firefox 3. I tracked it down to Ghostery and NoScript, but even without those addons it leaked about 500MB per hour of browsing.

GC and CC times got up into the 10 second range. Ugly. Really really ugly! And this is on top of the line massively overclocked hardware, too. I had to install a new addon to add a restart button to Firefox, because Firefox froze solid after hitting 2GB of memory usage. I also patched it after every update to allow up to 4GB, buying a little more time…

Then your patch comes along and solves it all… you are awesome man – totally awesome!

Another user — one who uses the leaky Autopager add-oncommented on Kyle’s blog.

Certainly, before this fix I would find that Firefox often became sluggish (input lag, slow paint operations, less than silky smooth scroll animations) as the memory usage built up. It’s hard to say how much various factors contributed to the whole, but GC pauses did undoubtedly cause the scroll animation stuttering.

Restarting was the cure. I haven’t noticed the same symptoms since, and while I haven’t had enough chance to make a conclusive judgement, the signs certainly seem to be good.

I have a full tab strip more often than not, and Fx set to load tabs from last time. This is offset by the wonderful and elegantly simple tabs on demand feature. I’m running a 2 year old laptop with 4GB ram.

And while we’re on the topic, here’s a comment from my blog.

Opened my firefox today, 30+ Tabs (only counting the ones in the active group, the others aren’t loaded), using just little more than 330 MB of RAM. A year ago, with Firefox 4, this would have been impossible. Keep it going!

Good times.

The following add-ons had zombie compartments fixed:  Youtube Ratings Preview, SPDY Indicator  It’s likely these leaks would have been fixed by Kyle’s change, but since Firefox 15 won’t be released until August 28, it’s good that they’ve been fixed now.  (Indeed, the AMO review policy still requires that add-ons not cause zombie compartments with the current release of Firefox;  that policy may be revisited once Firefox 15 is released.)

Compartment-per-global

The other big news is that compartment-per-global (CPG) landed, thanks to the work of various people, especially Luke Wagner and Bobby Holley.  Bobby explained what this means and explored some of the consequences.

CPG will allow lots of things within Firefox to become simpler and faster.  The main disadvantage is, unfortunately, increased memory consumption, as can be seen on areweslimyet.com.  (Thanks to Luke, this increase was less than it could have been.)  This is mostly due to more fragmentation in the JavaScript heap — we now have many more compartments, and each 4KB heap arena cannot be shared between compartments, so there are many more partially empty arenas present.

You might think this would make me bang my head against the wall in frustration, but it doesn’t.  That’s because even if I ignore the many non-MemShrink-related benefits of CPG, there are two big MemShrink-related ones.

First, CPG will enable per-tab memory reporting, something that users have been requesting for years.

Second, CPG will lead to much more detail in about:memory and about:compartments.  For example, Nils Maier has written a patch that makes it obvious all the JavaScript modules that have been loaded.  Another example:  Justin Dolske found that plusone.google.com was doing something silly (constantly creating new iframes?) that caused huge numbers of compartments to be created;  without CPG I think all those globals would have been lumped into a single compartment and the problem would have been much less obvious.  More information in about:memory will lead to more diagnosis of existing problems — particularly leaks of various kinds — in both Firefox and websites.

Memory Reporting

Kevin Locke tweaked the JS memory reporters so that more compartments are distinguished, instead of being lumped together.  This was his first Mozilla patch — well done, Kevin!

Nathan Froyd improved the coverage of the layout memory reporters.  This significantly reduces “heap-unclassified” for huge pages like the single-page version of the HTML5 spec.

Bug counts

Here are the current bug counts.

  • P1: 22 (-2/+3)
  • P2: 83 (-6/+4)
  • P3: 105 (-5/+6)
  • Unprioritized: 3 (-1/+3)

Mostly bouncing around at the moment.

Additional Update on Leaky Add-ons

I wrote last week about leaky add-ons.  Specifically, Kyle Huey landed a patch that in most cases prevents zombie compartments, which are the most common kind of memory leak in add-ons.  However, this change itself caused a different memory leak in some add-ons built with versions 1.3 and earlier of the Add-on SDK.  I described this as “two steps forward, one step back”.

Happily, Kyle landed a patch that slightly delayed the cutting of the chrome-to-content references.  This has the following consequences.

  • It doesn’t compromise the “two steps forward”.
  • It fixes the “one step back”.
  • It should reduce the potential for similar, as-yet-unknown backward steps.
  • The repacking of add-ons with newer versions of the Add-on SDK is now less urgent than it was.  It’s still a good idea, though, because older version had some other, albeit much smaller, leaks.

This is good news.  Firefox 15 is scheduled for release on August 28.  Assuming we don’t hit other problems with these changes prior to release, for users with add-ons there’s a good chance that Firefox 15 will use less memory and suffer fewer annoying pauses.

Once again, it would be great if users of Nightly builds, particularly those that use add-ons, could pay attention to memory consumption and file bugs for any bad behaviour.  Thanks!

Update on Leaky Add-ons

Leaky add-ons are a big problem, but there’s been lots of action relating to them recently.  What follows is a summary of the situation and a request for help from currently Nightly testers who use add-ons.

Two Steps Forward

Kyle Huey’s patch to prevent Chrome-to-Content leaks landed recently.  In theory it would prevent almost all add-ons zombie compartments, which constitute the majority of leaks from add-ons.  And in practice, it appears to be working splendidly.

I did some testing of eight add-ons that are known to leak: McAfee SiteAdvisor 3.4.1(the old version that leaked every content compartment), Firebug 1.9.1, PicPasso 1.0.1, LoL 1.5, YouTube Ratings Preview 1.03, Enter Selects 7, HttpFox 0.8.10, and Link Resolve 0.7.  I tested with a “before” browser build that lacked Kyle’s patch, and an “after” browser build that had it.  For every one of the tested add-ons, I could reproduce the zombie compartment(s) with the “before” build, but could not reproduce any zombie compartment(s) with the “after” build.  In other words, the patch worked perfectly.

To understand how important this is, consider this comment on Kyle’s blog describing the effect of the patch.

I seem to be seeing really significant improvements from this. Normally Fx would start at 170MB and by the end of the day that would often be creeping up to 800 or 900 MB. Today using latest nightly I again started at 170, but now at the end of the day I’m only at 230MB!

That’s a 4x reduction in memory consumption.

What effect does this have?  The obvious benefit of reduced memory consumption is that there will be less potential for paging.  For example, if the above user is on a low-end machine without much RAM, a reduction in memory consumption from 800+MB to 230MB might greatly reduce paging, which would make Firefox much faster.

However, what if the user has a high-end machine with 16GB of RAM?  Then paging isn’t an issue.  But this improvement will still be a big deal on such a machine.  This is because garbage collection and cycle collection cause pauses, and the length of the pauses are roughly proportional to the amount of live heap memory.  (Incremental garbage collection will soon be enabled, which will result in smaller garbage collection pauses, but there are no plans for incremental cycle collection and so cycle collection pauses will still be relevant.)  So even on high-end machines with lots of RAM, leaks can greatly hurt browser performance.  In other words, add-on leaks are a Snappy issue just as much as they are a MemShrink issue.

One Step Back

[Update: Kyle found a simple way to fix this problem with the Add-on SDK.]

So Kyle’s patch is great, but the cutting of the chrome-to-content references was unlikely to be totally free of side-effects.  And sure enough, last week Josh Matthews noticed that Dietrich Ayala’s Wallflower add-on was causing many zombie compartments.  Hmm, wasn’t Kyle’s patch supposed to stop exactly that?  Well, it was expected to stop the most common cases, but there are some other leaks that could still cause zombie compartments.

The weird thing with this leak is that Wallflower is a really simple add-on.  It’s built with the Add-on SDK and is only a few lines of code.  It turns out that Kyle’s patch caused the Add-on SDK to leak badly.  The exact reason is complex;  you can see Kyle’s explanation here.

This sounds really bad.  Lots of add-ons are built with the Add-on SDK.  Have we just exchanged one class of add-on leaks for another?  I don’t believe so, for two reasons.

  1. Kyle’s patch prevents the single most common class of add-on leaks, and the Add-on SDK leak it caused is a much more obscure and unlikely leak.  It’s unfortunate that this obscure leak happened to be in a piece of code that’s used by lots of add-ons.
  2. This obscure leak only occurs in old versions of the Add-on SDK.  The leak doesn’t occur with the latest version, which is 1.6.1.  Indeed, when Dietrich rebuilt Wallflower with version 1.6.1 the problem went away.

Still, the situation is far from ideal, because lots of add-ons are still built with older SDK versions.  So, what to do?  There’s a bug open discussing this question, but it’s gotten bogged down due to differing opinions on what to do.  Version 1.6.1 happened to fix all leaks in the Add-on SDK that were known prior to this problem, so we should be encouraging add-on authors to rebuild with it anyway.

In the meantime, if you use Nightly builds and use add-ons, please monitor about:compartments and file bugs if you find that any add-ons are causing zombie compartments.  If Wallflower is representative, the leaks will be very bad and so shouldn’t be hard to spot.  Firefox 15 is scheduled for release on August 28th;  we need as many affected add-ons to be rebuilt with the latest SDK before that date to minimize potential problems.

Problems with FUEL

Another cause of high memory consumption in add-ons is FUEL.  I know very little about it other than it’s also part of the Add-on SDK sort of a proto-version of the Add-on SDK.  The details are here and here.  This is causing bad performance in Readability and probably other add-ons.  Something to be aware of.

MemShrink progress, week 45–46

FIXING LEAKS

The big news this week is that Kyle Huey made chrome-to-content leaks impossible.  Kyle wrote about this in some detail.  This particular kind of leak can happen in Firefox, but more importantly, it’s responsible for most of the leaks in add-ons that we know about.  In other words, it should greatly reduce the problem of leaky add-ons, a problem I previously identified as the #1 MemShrink issue.  It’ll take a while to see the exact effects and whether they match expectations.  Nonetheless, this is a big deal!

Olli Pettay fixed a leak relating to Geolocation.

The following add-ons had leaks fixed: Collusion, SearchMenu, panchang, HTML5 Extension for Windows Media Player Firefox Plug-in.

Slimmer data structures

I reduced the size of JSScripts (here, here and here).  This increased the number of JSScripts that fit in a GC arena from 31 to 42 (on 32-bit platforms) and from 19 to 28 (on 64-bit platforms).  This reduced the size of the “scripts” entry in about:memory — which is often multiple MB for the compartments of complex web sites and apps like Gmail — by approximately 30%.  (The final two patches in the sequence are being held up by the current tree closure, but will land as soon as that has passed.)

Nathan Froyd reduced the size of mozilla::dom::Link — one of which exists for every <a> or <link> or <area> element shown in a page — by 4 bytes (on 32-bit platforms) and 8 bytes (on 64-bit platforms).

Ehsan Akhgari reduced the size of nsEditor — one of which exists for each text area in a page, and some other places — by 16 bytes on 64-bit platforms.

 Quote of the FORTNIGHT

PC Advisor featured an article about browser performance.  The article criticized the standard browser benchmarks for (a) not representing everyday browsing and (b) underestimating the effects of memory consumption on browser performance — two things that I agree with.

The article also presented some cross-browser memory consumption results from real-world-ish workloads.  The testing was imperfect — for example, it didn’t measure memory consumption after any kind of prolonged usage — but I liked the fact that the author had actually thought about things instead of just mindlessly running the standard benchmarks.

And with my PR hat on, I was happy with the conclusion.

Firefox is the clear winner of the bunch. It was the only browser that did not slow things down and I recommended it for both lower-end mobile devices and high-end desktops.

In general, cross-browser memory consumption comparisons are difficult to do well.  My preferred method is to run some set of benchmarks on a machine with a small amount of RAM, as that gives a genuine indication of the performance effect.  “Browser A was 50% slower on benchmark X when I halved the available RAM, but browser B was only 10% slower” is much more meaningful than “Browser A used 20% more memory on benchmark X than browser B”.  Choosing good benchmarks is still not easy, though.

Bug counts

Here are the current bug counts.

  • P1: 21 (-1/+0)
  • P2: 85 (-14/+15)
  • P3: 104 (-5/+9)
  • Unprioritized: 1 (-0/+1)

I’m thinking about omitting the bug counts from future MemShrink reports.  I don’t feel like they add much, but I’d be interested to hear if people think otherwise.

MemShrink progress, week 43–44

As promised last time, this MemShrink report covers two weeks’ worth of activity.

Melbourne MemShrink Meet-up

This week Kyle Huey, Justin Lebar, Andrew McCreight, Jet Villegas flew to Melbourne to bunker down with me for a MemShrink work week.  This is great for two reasons.  First, it gives all of us an excuse/opportunity to ignore the usual myriad of attention-grabbing tasks and focus solely on MemShrink stuff for a week.  (For example, Jet is the manager of the layout team but he’s managed to do a couple of days of coding!)  Second, it lets us collaborate and pick each others’ brains much more easily and efficiently than normally.

Badd-ons

The following add-ons had zombie compartments fixed.

I’ve said before that add-on leaks are the #1 cause of high memory consumption in Firefox.  Numerous add-ons have had leaks fixed, but this is a slow process.  However, Kyle has been making good progress this week on a patch that has the potential to mitigate the majority of add-on leaks that cause zombie compartments.  It currently passes all tests on try server and is undergoing review.  Watch this space!

Version 1.6 of the Add-on SDK was released, which fixed all remaining known leaks.  Except that a new leak was found shortly before release!  This was found too late to be fixed before the deadline for 1.6, but fortunately, it was quickly fixed in version 1.6.1.  The Add-on SDK developers (especially Alexandre Poirot) have fixed a ton of leaks in the past few months, so it’s great to see this milestone reached.

Miscellaneous

Olli Pettay fixed a leak relating to event listeners.

Henrik Skupin released version 0.3 of the MemChaser add-on.  It features improvements to the UI and the logging.

Jonathan Kew optimized the representation of font character maps.  Depending on the platform, this can save around 1MB of memory, or more if you have lots of fonts installed.

Jared Wein landed an option for making plugin content click-to-play.  This isn’t marked as a MemShrink bug but if a user can ignore such content, it has the potential to reduce memory consumption significantly in some cases.

Bug counts

Here are the current bug counts.

  • P1: 22 (-1/+2)
  • P2: 84 (-56/+3)
  • P3: 100 (-8/+18)
  • Unprioritized: 0 (-1/+0)

Lots of movement there.  This is because we’ve been going through the P2 bugs and closed a lot of them because they were dups, or lacked enough information to do anything useful, or we decided they were no longer worthwhile.  We also downgraded quite a few P2 bugs to P3.  As a result, the P2 list is much more interesting than it was.  And we’re only partway through it!

A new meeting schedule

I mentioned last time that I would be writing these reports every two weeks from now on, instead of weekly.  Similarly, we’ve decided to reduce the frequency of MemShrink meetings to once every two weeks.  The next meeting will be on May 1st.

This change does not reflect a reduction in the amount of work being done on MemShrink bugs.  Rather, it reflects a reduction in the number of new MemShrink bugs being filed.  At the past few meetings we’ve had only a handful of new bugs each time and triage has only taken a few minutes.  (In comparison, in the early days of MemShrink, several times we didn’t get through all the new bugs in an hour.)

This leaves us lots of time for talking about other things;  in fact, more time than we’ve needed.  And time spent in meetings is time not spent fixing problems, so this feels like the right thing to do.

MemShrink progress, week 42

 Are we slim yet?

John’s Schoenick’s areweslimyet.com (AWSY) has had its password removed and is now open to the public!

areweslimyet.com screenshot

This is a major milestone for the MemShrink project.  It shows the progress we have made (MemShrink started in earnest in June 2011) and will let us identify regressions more easily.

John’s done a wonderful job implementing the site.  It has lots of functionality:  there are multiple graphs, you can zoom in on parts of graphs to see more detail, and you can see revisions, dates and about:memory snapshots for individual runs.

John has also put in a great deal of work refining the methodology to the point where we believe it provides a reasonable facsimile of real-world browsing;  please read the FAQ to understand exactly what is being measured.  Many thanks also to Dave Hunt and the QA team for their work on the Mozmill Endurance Tests, which are at the core of AWSY’s testing.

Update: Hacker News has reported on this.

Ghost windows

Frequent readers of this blog will be familiar with zombie compartments, which are JavaScript compartments that have leaked, due to defects in Firefox or add-ons.  Windows (i.e. window objects) can also be leaked, and often defects that cause compartments leaks will cause window leaks as well.

Justin Lebar has introduced the notion of “ghost windows”.  A ghost window is one that meets the following criteria.

  1. Shows up in about:memory under “window-objects/top(none)”.
  2. Does not share a domain name with any window under “window-objects/active.
  3. Has met criteria (1) and (2) for a moderate amount of time (e.g. two minutes).

The basic idea is that a ghost window has a high chance of representing a genuine leak, and this automated identification of suspicious windows will make leak detection simpler.  Justin has added ghost window tracking to about:memory, about:compartments, and telemetry.  (These three bugs were all marked as MemShrink:P1.)  Ghost window tracking is mostly untested right now, but hopefully it will become another powerful tool for finding memory leaks.

Add-ons

We’ve been tracking leaky add-ons in Bugzilla for a while now, but we’ve never had a good product/component to put them in.  David Lawrence, Byron Jones, Stormy Peters and I have together created a new “Add-ons” component under the “Tech Evangelism” product.  The rationale for putting it under “Tech Evangelism” is that it nicely matches the existing meaning of that phrase — it’s a case where a non-Mozilla entity is writing defective code that interacts with Firefox and hurts users’ experiences with and perceptions of Firefox, and Mozilla can only inform, educate and encourage fixes in that defective code.  This component is only intended for certain classes of common defects (such as leaks) that Mozilla contributors are tracking.  It is not intended for vanilla add-on bugs;  as now, they should be reported through whatever bug-reporting mechanism each add-on uses.  I’ve updated the existing open bugs that track leaky add-ons to use this new component.

Leaks in the following add-ons were fixed:  Video DownloadHelper (the 2nd most popular add-on on AMO!), Scrapbook Plus, Amazon Price Tracker.

Bug counts

This week’s bug counts:

  • P1: 21 (-5/+0)
  • P2: 137 (-1/+7)
  • P3: 90 (-1/+3)
  • Unprioritized: 1 (-1/+1)

Good progress on the P1 bugs!

A new reporting schedule

Many of the weekly MemShrink reports lately have been brief.  From now on I plan to write a report every two weeks.  This will make things easier for me and will also ensure each report is packed full of interesting things.  See you again in two weeks!

MemShrink progress, week 41

Add-ons

Jorge Villalobos encouraged add-on authors to check for memory leaks on the official Mozilla add-ons blog.

The following add-ons had memory leaks fixed: FavSync, Download Youtube Videos, Page Speed.

The follow add-ons were downgraded to “preliminary review” status on AMO, giving them less visibility: QuickDrag, Yahoo Toolbar.  The Yahoo Toolbar case is interesting;  it is available on AMO, but only has 47,618 users there.  But the number of installed copies is much larger than this.  (I can’t say how much larger, because the exact numbers are not public knowledge.)  So it’s clear that most people who have it installed got it elsewhere, probably directly from Yahoo.  Given that AMO accounts for only a tiny fraction of the installations, the effect of the downgrade in AMO review status is negligible.  Furthermore, Jorge’s attempts to contact Yahoo appear to have failed.  Finally, the only other means we have for addressing add-ons with problems is block-listing them, but that would be overkill in this case.  So we are at an impasse.  It’s a frustrating situation!

Gecko and Firefox

Tim Taubert fixed two leaks in the recently updated new tabs page.

Jonathan Kew added memory reporters for thebes font objects.  With text-heavy pages open, these can account for 10s of MBs of memory.

Andrew McCreight fixed some shutdown leaks involving the XBL cache.

Igor Bukanov did something with JS principals that I won’t pretend to understand.  This bug wasn’t marked with the MemShrink tag, but it gave good Talos results, reducing the number of malloc calls and the maximum heap size on several platforms.

Bug counts

This week’s bug counts:

  • P1: 26 (-2/+4)
  • P2: 131 (-1/+3)
  • P3: 88 (-5/+1)
  • Unprioritized: 0 (-0/+1)

Nothing spectacular there.

MemShrink progress, week 40

The past week was fairly quiet for MemShrink.

Add-ons

Our testing of the top 100 installed add-ons is progressing slowly.  We need more help here.  Please join in if you are interested!

The Image Zoom and Baow add-ons were downgraded to “preliminarily reviewed” status because they cause zombie compartments, as per the new AMO policy.  The main effect of the downgrade is that they’ll show up lower in AMO search results.

Miscellaneous

I modified the storage of CSS properties and values to minimize waste caused by alignment.  On 64-bit platforms this reduced the size of a property/value pair from 24 bytes to 18 bytes, which saves about 0.5MB for an instance of Gmail.  (Gmail uses a lots of CSS!)  On 32-bit platforms the reduction is 1/3 the size, with a property/value pair dropping from 12 to 10 bytes.

Justin Lebar capped the amount of memory used for decoded images.  Decoded images in background tabs are currently discarded 20-40 seconds after they stop being visible in the foreground tab.  With Justin’s patch, there is now a second criterion:  if the decoded image cap is exceeded, decoded images in background tabs will be discarded until the cap is met.  (Decoded images in the foreground tab are never discarded, so it’s possible that all decoded images in background tabs are discarded and the cap is still exceeded.)  The cap defaults to 50MiB, but it can be changed in about:config with the image.mem.max_decoded_image_kb option.  The main effect of this change is that decoded images in background tabs will be discarded more quickly than they used to be in some circumstances.

Bug counts

This week’s bug counts:

  • P1: 22 (-2/+0)
  • P2: 129 (-6/+3)
  • P3: 92 (-3/+3)
  • Unprioritized: 0 (-0/+0)

That’s a net reduction of five bugs!  Furthermore, we only had five bugs to triage in today’s MemShrink meeting.  Good signs.