Category: Memory consumption

Categories
add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 47–48

Add-ons

The main news in the past two weeks has been about Kyle Huey’s patch that prevents most chrome-to-content leaks, which are the most common kind of add-on leak.  Testing showed it worked beautifully, but caused a knock-on leak in add-ons built with old versions (1.3 and earlier) of the Add-on SDK.  (This received a lot of attention.)  Kyle then made a slight tweak that fixed that knock-on leak.  So we’re currently still on track for Firefox 15 being “the one that fixes add-on leaks”.

For completeness, here are the add-ons that we know were temporarily affected by that knock-on leak:  Wallflower, Visual Hashing, Translate This!, Easy YouTube Video Downloader.  They (and probably quite a few others) are all working fine again now.

Here’s a quote from an email that one user sent to Kyle this week.

Firefox was leaking about 1.5GB per hour for me. It started with Firefox 3. I tracked it down to Ghostery and NoScript, but even without those addons it leaked about 500MB per hour of browsing.

GC and CC times got up into the 10 second range. Ugly. Really really ugly! And this is on top of the line massively overclocked hardware, too. I had to install a new addon to add a restart button to Firefox, because Firefox froze solid after hitting 2GB of memory usage. I also patched it after every update to allow up to 4GB, buying a little more time…

Then your patch comes along and solves it all… you are awesome man – totally awesome!

Another user — one who uses the leaky Autopager add-on — commented on Kyle’s blog.

Certainly, before this fix I would find that Firefox often became sluggish (input lag, slow paint operations, less than silky smooth scroll animations) as the memory usage built up. It’s hard to say how much various factors contributed to the whole, but GC pauses did undoubtedly cause the scroll animation stuttering.

Restarting was the cure. I haven’t noticed the same symptoms since, and while I haven’t had enough chance to make a conclusive judgement, the signs certainly seem to be good.

I have a full tab strip more often than not, and Fx set to load tabs from last time. This is offset by the wonderful and elegantly simple tabs on demand feature. I’m running a 2 year old laptop with 4GB ram.

And while we’re on the topic, here’s a comment from my blog.

Opened my firefox today, 30+ Tabs (only counting the ones in the active group, the others aren’t loaded), using just little more than 330 MB of RAM. A year ago, with Firefox 4, this would have been impossible. Keep it going!

Good times.

The following add-ons had zombie compartments fixed:  Youtube Ratings Preview, SPDY Indicator  It’s likely these leaks would have been fixed by Kyle’s change, but since Firefox 15 won’t be released until August 28, it’s good that they’ve been fixed now.  (Indeed, the AMO review policy still requires that add-ons not cause zombie compartments with the current release of Firefox;  that policy may be revisited once Firefox 15 is released.)

Compartment-per-global

The other big news is that compartment-per-global (CPG) landed, thanks to the work of various people, especially Luke Wagner and Bobby Holley.  Bobby explained what this means and explored some of the consequences.

CPG will allow lots of things within Firefox to become simpler and faster.  The main disadvantage is, unfortunately, increased memory consumption, as can be seen on areweslimyet.com.  (Thanks to Luke, this increase was less than it could have been.)  This is mostly due to more fragmentation in the JavaScript heap — we now have many more compartments, and each 4KB heap arena cannot be shared between compartments, so there are many more partially empty arenas present.

You might think this would make me bang my head against the wall in frustration, but it doesn’t.  That’s because even if I ignore the many non-MemShrink-related benefits of CPG, there are two big MemShrink-related ones.

First, CPG will enable per-tab memory reporting, something that users have been requesting for years.

Second, CPG will lead to much more detail in about:memory and about:compartments.  For example, Nils Maier has written a patch that makes it obvious all the JavaScript modules that have been loaded.  Another example:  Justin Dolske found that plusone.google.com was doing something silly (constantly creating new iframes?) that caused huge numbers of compartments to be created;  without CPG I think all those globals would have been lumped into a single compartment and the problem would have been much less obvious.  More information in about:memory will lead to more diagnosis of existing problems — particularly leaks of various kinds — in both Firefox and websites.

Memory Reporting

Kevin Locke tweaked the JS memory reporters so that more compartments are distinguished, instead of being lumped together.  This was his first Mozilla patch — well done, Kevin!

Nathan Froyd improved the coverage of the layout memory reporters.  This significantly reduces “heap-unclassified” for huge pages like the single-page version of the HTML5 spec.

Bug counts

Here are the current bug counts.

  • P1: 22 (-2/+3)
  • P2: 83 (-6/+4)
  • P3: 105 (-5/+6)
  • Unprioritized: 3 (-1/+3)

Mostly bouncing around at the moment.

Categories
add-ons Firefox Memory consumption MemShrink

Update on Leaky Add-ons

Leaky add-ons are a big problem, but there’s been lots of action relating to them recently.  What follows is a summary of the situation and a request for help from currently Nightly testers who use add-ons.

Two Steps Forward

Kyle Huey’s patch to prevent Chrome-to-Content leaks landed recently.  In theory it would prevent almost all add-ons zombie compartments, which constitute the majority of leaks from add-ons.  And in practice, it appears to be working splendidly.

I did some testing of eight add-ons that are known to leak: McAfee SiteAdvisor 3.4.1(the old version that leaked every content compartment), Firebug 1.9.1, PicPasso 1.0.1, LoL 1.5, YouTube Ratings Preview 1.03, Enter Selects 7, HttpFox 0.8.10, and Link Resolve 0.7.  I tested with a “before” browser build that lacked Kyle’s patch, and an “after” browser build that had it.  For every one of the tested add-ons, I could reproduce the zombie compartment(s) with the “before” build, but could not reproduce any zombie compartment(s) with the “after” build.  In other words, the patch worked perfectly.

To understand how important this is, consider this comment on Kyle’s blog describing the effect of the patch.

I seem to be seeing really significant improvements from this. Normally Fx would start at 170MB and by the end of the day that would often be creeping up to 800 or 900 MB. Today using latest nightly I again started at 170, but now at the end of the day I’m only at 230MB!

That’s a 4x reduction in memory consumption.

What effect does this have?  The obvious benefit of reduced memory consumption is that there will be less potential for paging.  For example, if the above user is on a low-end machine without much RAM, a reduction in memory consumption from 800+MB to 230MB might greatly reduce paging, which would make Firefox much faster.

However, what if the user has a high-end machine with 16GB of RAM?  Then paging isn’t an issue.  But this improvement will still be a big deal on such a machine.  This is because garbage collection and cycle collection cause pauses, and the length of the pauses are roughly proportional to the amount of live heap memory.  (Incremental garbage collection will soon be enabled, which will result in smaller garbage collection pauses, but there are no plans for incremental cycle collection and so cycle collection pauses will still be relevant.)  So even on high-end machines with lots of RAM, leaks can greatly hurt browser performance.  In other words, add-on leaks are a Snappy issue just as much as they are a MemShrink issue.

One Step Back

[Update: Kyle found a simple way to fix this problem with the Add-on SDK.]

So Kyle’s patch is great, but the cutting of the chrome-to-content references was unlikely to be totally free of side-effects.  And sure enough, last week Josh Matthews noticed that Dietrich Ayala’s Wallflower add-on was causing many zombie compartments.  Hmm, wasn’t Kyle’s patch supposed to stop exactly that?  Well, it was expected to stop the most common cases, but there are some other leaks that could still cause zombie compartments.

The weird thing with this leak is that Wallflower is a really simple add-on.  It’s built with the Add-on SDK and is only a few lines of code.  It turns out that Kyle’s patch caused the Add-on SDK to leak badly.  The exact reason is complex;  you can see Kyle’s explanation here.

This sounds really bad.  Lots of add-ons are built with the Add-on SDK.  Have we just exchanged one class of add-on leaks for another?  I don’t believe so, for two reasons.

  1. Kyle’s patch prevents the single most common class of add-on leaks, and the Add-on SDK leak it caused is a much more obscure and unlikely leak.  It’s unfortunate that this obscure leak happened to be in a piece of code that’s used by lots of add-ons.
  2. This obscure leak only occurs in old versions of the Add-on SDK.  The leak doesn’t occur with the latest version, which is 1.6.1.  Indeed, when Dietrich rebuilt Wallflower with version 1.6.1 the problem went away.

Still, the situation is far from ideal, because lots of add-ons are still built with older SDK versions.  So, what to do?  There’s a bug open discussing this question, but it’s gotten bogged down due to differing opinions on what to do.  Version 1.6.1 happened to fix all leaks in the Add-on SDK that were known prior to this problem, so we should be encouraging add-on authors to rebuild with it anyway.

In the meantime, if you use Nightly builds and use add-ons, please monitor about:compartments and file bugs if you find that any add-ons are causing zombie compartments.  If Wallflower is representative, the leaks will be very bad and so shouldn’t be hard to spot.  Firefox 15 is scheduled for release on August 28th;  we need as many affected add-ons to be rebuilt with the latest SDK before that date to minimize potential problems.

Problems with FUEL

Another cause of high memory consumption in add-ons is FUEL.  I know very little about it other than it’s also part of the Add-on SDK sort of a proto-version of the Add-on SDK.  The details are here and here.  This is causing bad performance in Readability and probably other add-ons.  Something to be aware of.

Categories
add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 45–46

FIXING LEAKS

The big news this week is that Kyle Huey made chrome-to-content leaks impossible.  Kyle wrote about this in some detail.  This particular kind of leak can happen in Firefox, but more importantly, it’s responsible for most of the leaks in add-ons that we know about.  In other words, it should greatly reduce the problem of leaky add-ons, a problem I previously identified as the #1 MemShrink issue.  It’ll take a while to see the exact effects and whether they match expectations.  Nonetheless, this is a big deal!

Olli Pettay fixed a leak relating to Geolocation.

The following add-ons had leaks fixed: Collusion, SearchMenu, panchang, HTML5 Extension for Windows Media Player Firefox Plug-in.

Slimmer data structures

I reduced the size of JSScripts (here, here and here).  This increased the number of JSScripts that fit in a GC arena from 31 to 42 (on 32-bit platforms) and from 19 to 28 (on 64-bit platforms).  This reduced the size of the “scripts” entry in about:memory — which is often multiple MB for the compartments of complex web sites and apps like Gmail — by approximately 30%.  (The final two patches in the sequence are being held up by the current tree closure, but will land as soon as that has passed.)

Nathan Froyd reduced the size of mozilla::dom::Link — one of which exists for every <a> or <link> or <area> element shown in a page — by 4 bytes (on 32-bit platforms) and 8 bytes (on 64-bit platforms).

Ehsan Akhgari reduced the size of nsEditor — one of which exists for each text area in a page, and some other places — by 16 bytes on 64-bit platforms.

 Quote of the FORTNIGHT

PC Advisor featured an article about browser performance.  The article criticized the standard browser benchmarks for (a) not representing everyday browsing and (b) underestimating the effects of memory consumption on browser performance — two things that I agree with.

The article also presented some cross-browser memory consumption results from real-world-ish workloads.  The testing was imperfect — for example, it didn’t measure memory consumption after any kind of prolonged usage — but I liked the fact that the author had actually thought about things instead of just mindlessly running the standard benchmarks.

And with my PR hat on, I was happy with the conclusion.

Firefox is the clear winner of the bunch. It was the only browser that did not slow things down and I recommended it for both lower-end mobile devices and high-end desktops.

In general, cross-browser memory consumption comparisons are difficult to do well.  My preferred method is to run some set of benchmarks on a machine with a small amount of RAM, as that gives a genuine indication of the performance effect.  “Browser A was 50% slower on benchmark X when I halved the available RAM, but browser B was only 10% slower” is much more meaningful than “Browser A used 20% more memory on benchmark X than browser B”.  Choosing good benchmarks is still not easy, though.

Bug counts

Here are the current bug counts.

  • P1: 21 (-1/+0)
  • P2: 85 (-14/+15)
  • P3: 104 (-5/+9)
  • Unprioritized: 1 (-0/+1)

I’m thinking about omitting the bug counts from future MemShrink reports.  I don’t feel like they add much, but I’d be interested to hear if people think otherwise.

Categories
add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 43–44

As promised last time, this MemShrink report covers two weeks’ worth of activity.

Melbourne MemShrink Meet-up

This week Kyle Huey, Justin Lebar, Andrew McCreight, Jet Villegas flew to Melbourne to bunker down with me for a MemShrink work week.  This is great for two reasons.  First, it gives all of us an excuse/opportunity to ignore the usual myriad of attention-grabbing tasks and focus solely on MemShrink stuff for a week.  (For example, Jet is the manager of the layout team but he’s managed to do a couple of days of coding!)  Second, it lets us collaborate and pick each others’ brains much more easily and efficiently than normally.

Badd-ons

The following add-ons had zombie compartments fixed.

I’ve said before that add-on leaks are the #1 cause of high memory consumption in Firefox.  Numerous add-ons have had leaks fixed, but this is a slow process.  However, Kyle has been making good progress this week on a patch that has the potential to mitigate the majority of add-on leaks that cause zombie compartments.  It currently passes all tests on try server and is undergoing review.  Watch this space!

Version 1.6 of the Add-on SDK was released, which fixed all remaining known leaks.  Except that a new leak was found shortly before release!  This was found too late to be fixed before the deadline for 1.6, but fortunately, it was quickly fixed in version 1.6.1.  The Add-on SDK developers (especially Alexandre Poirot) have fixed a ton of leaks in the past few months, so it’s great to see this milestone reached.

Miscellaneous

Olli Pettay fixed a leak relating to event listeners.

Henrik Skupin released version 0.3 of the MemChaser add-on.  It features improvements to the UI and the logging.

Jonathan Kew optimized the representation of font character maps.  Depending on the platform, this can save around 1MB of memory, or more if you have lots of fonts installed.

Jared Wein landed an option for making plugin content click-to-play.  This isn’t marked as a MemShrink bug but if a user can ignore such content, it has the potential to reduce memory consumption significantly in some cases.

Bug counts

Here are the current bug counts.

  • P1: 22 (-1/+2)
  • P2: 84 (-56/+3)
  • P3: 100 (-8/+18)
  • Unprioritized: 0 (-1/+0)

Lots of movement there.  This is because we’ve been going through the P2 bugs and closed a lot of them because they were dups, or lacked enough information to do anything useful, or we decided they were no longer worthwhile.  We also downgraded quite a few P2 bugs to P3.  As a result, the P2 list is much more interesting than it was.  And we’re only partway through it!

A new meeting schedule

I mentioned last time that I would be writing these reports every two weeks from now on, instead of weekly.  Similarly, we’ve decided to reduce the frequency of MemShrink meetings to once every two weeks.  The next meeting will be on May 1st.

This change does not reflect a reduction in the amount of work being done on MemShrink bugs.  Rather, it reflects a reduction in the number of new MemShrink bugs being filed.  At the past few meetings we’ve had only a handful of new bugs each time and triage has only taken a few minutes.  (In comparison, in the early days of MemShrink, several times we didn’t get through all the new bugs in an hour.)

This leaves us lots of time for talking about other things;  in fact, more time than we’ve needed.  And time spent in meetings is time not spent fixing problems, so this feels like the right thing to do.

Categories
about:compartments about:memory add-ons compartments Firefox Memory consumption MemShrink

MemShrink progress, week 42

 Are we slim yet?

John’s Schoenick’s areweslimyet.com (AWSY) has had its password removed and is now open to the public!

areweslimyet.com screenshot

This is a major milestone for the MemShrink project.  It shows the progress we have made (MemShrink started in earnest in June 2011) and will let us identify regressions more easily.

John’s done a wonderful job implementing the site.  It has lots of functionality:  there are multiple graphs, you can zoom in on parts of graphs to see more detail, and you can see revisions, dates and about:memory snapshots for individual runs.

John has also put in a great deal of work refining the methodology to the point where we believe it provides a reasonable facsimile of real-world browsing;  please read the FAQ to understand exactly what is being measured.  Many thanks also to Dave Hunt and the QA team for their work on the Mozmill Endurance Tests, which are at the core of AWSY’s testing.

Update: Hacker News has reported on this.

Ghost windows

Frequent readers of this blog will be familiar with zombie compartments, which are JavaScript compartments that have leaked, due to defects in Firefox or add-ons.  Windows (i.e. window objects) can also be leaked, and often defects that cause compartments leaks will cause window leaks as well.

Justin Lebar has introduced the notion of “ghost windows”.  A ghost window is one that meets the following criteria.

  1. Shows up in about:memory under “window-objects/top(none)”.
  2. Does not share a domain name with any window under “window-objects/active.
  3. Has met criteria (1) and (2) for a moderate amount of time (e.g. two minutes).

The basic idea is that a ghost window has a high chance of representing a genuine leak, and this automated identification of suspicious windows will make leak detection simpler.  Justin has added ghost window tracking to about:memory, about:compartments, and telemetry.  (These three bugs were all marked as MemShrink:P1.)  Ghost window tracking is mostly untested right now, but hopefully it will become another powerful tool for finding memory leaks.

Add-ons

We’ve been tracking leaky add-ons in Bugzilla for a while now, but we’ve never had a good product/component to put them in.  David Lawrence, Byron Jones, Stormy Peters and I have together created a new “Add-ons” component under the “Tech Evangelism” product.  The rationale for putting it under “Tech Evangelism” is that it nicely matches the existing meaning of that phrase — it’s a case where a non-Mozilla entity is writing defective code that interacts with Firefox and hurts users’ experiences with and perceptions of Firefox, and Mozilla can only inform, educate and encourage fixes in that defective code.  This component is only intended for certain classes of common defects (such as leaks) that Mozilla contributors are tracking.  It is not intended for vanilla add-on bugs;  as now, they should be reported through whatever bug-reporting mechanism each add-on uses.  I’ve updated the existing open bugs that track leaky add-ons to use this new component.

Leaks in the following add-ons were fixed:  Video DownloadHelper (the 2nd most popular add-on on AMO!), Scrapbook Plus, Amazon Price Tracker.

Bug counts

This week’s bug counts:

  • P1: 21 (-5/+0)
  • P2: 137 (-1/+7)
  • P3: 90 (-1/+3)
  • Unprioritized: 1 (-1/+1)

Good progress on the P1 bugs!

A new reporting schedule

Many of the weekly MemShrink reports lately have been brief.  From now on I plan to write a report every two weeks.  This will make things easier for me and will also ensure each report is packed full of interesting things.  See you again in two weeks!

Categories
add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 41

Add-ons

Jorge Villalobos encouraged add-on authors to check for memory leaks on the official Mozilla add-ons blog.

The following add-ons had memory leaks fixed: FavSync, Download Youtube Videos, Page Speed.

The follow add-ons were downgraded to “preliminary review” status on AMO, giving them less visibility: QuickDrag, Yahoo Toolbar.  The Yahoo Toolbar case is interesting;  it is available on AMO, but only has 47,618 users there.  But the number of installed copies is much larger than this.  (I can’t say how much larger, because the exact numbers are not public knowledge.)  So it’s clear that most people who have it installed got it elsewhere, probably directly from Yahoo.  Given that AMO accounts for only a tiny fraction of the installations, the effect of the downgrade in AMO review status is negligible.  Furthermore, Jorge’s attempts to contact Yahoo appear to have failed.  Finally, the only other means we have for addressing add-ons with problems is block-listing them, but that would be overkill in this case.  So we are at an impasse.  It’s a frustrating situation!

Gecko and Firefox

Tim Taubert fixed two leaks in the recently updated new tabs page.

Jonathan Kew added memory reporters for thebes font objects.  With text-heavy pages open, these can account for 10s of MBs of memory.

Andrew McCreight fixed some shutdown leaks involving the XBL cache.

Igor Bukanov did something with JS principals that I won’t pretend to understand.  This bug wasn’t marked with the MemShrink tag, but it gave good Talos results, reducing the number of malloc calls and the maximum heap size on several platforms.

Bug counts

This week’s bug counts:

  • P1: 26 (-2/+4)
  • P2: 131 (-1/+3)
  • P3: 88 (-5/+1)
  • Unprioritized: 0 (-0/+1)

Nothing spectacular there.

Categories
add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 40

The past week was fairly quiet for MemShrink.

Add-ons

Our testing of the top 100 installed add-ons is progressing slowly.  We need more help here.  Please join in if you are interested!

The Image Zoom and Baow add-ons were downgraded to “preliminarily reviewed” status because they cause zombie compartments, as per the new AMO policy.  The main effect of the downgrade is that they’ll show up lower in AMO search results.

Miscellaneous

I modified the storage of CSS properties and values to minimize waste caused by alignment.  On 64-bit platforms this reduced the size of a property/value pair from 24 bytes to 18 bytes, which saves about 0.5MB for an instance of Gmail.  (Gmail uses a lots of CSS!)  On 32-bit platforms the reduction is 1/3 the size, with a property/value pair dropping from 12 to 10 bytes.

Justin Lebar capped the amount of memory used for decoded images.  Decoded images in background tabs are currently discarded 20-40 seconds after they stop being visible in the foreground tab.  With Justin’s patch, there is now a second criterion:  if the decoded image cap is exceeded, decoded images in background tabs will be discarded until the cap is met.  (Decoded images in the foreground tab are never discarded, so it’s possible that all decoded images in background tabs are discarded and the cap is still exceeded.)  The cap defaults to 50MiB, but it can be changed in about:config with the image.mem.max_decoded_image_kb option.  The main effect of this change is that decoded images in background tabs will be discarded more quickly than they used to be in some circumstances.

Bug counts

This week’s bug counts:

  • P1: 22 (-2/+0)
  • P2: 129 (-6/+3)
  • P3: 92 (-3/+3)
  • Unprioritized: 0 (-0/+0)

That’s a net reduction of five bugs!  Furthermore, we only had five bugs to triage in today’s MemShrink meeting.  Good signs.

Categories
about:memory add-ons Memory consumption MemShrink

MemShrink progress, week 39

Add-ons

Versions of McAfee’s SiteAdvisor add-on prior to 3.4.1.195 have been soft-blocked due to an extreme memory leak.  This means that it will be disabled within Firefox, but users can re-enable it if they want to.  According to McAfee, most users should have been upgraded to 3.4.1.195 by now.  Unfortunately, version 3.4.1.195 still has some leaks, and progress on fixing them has stalled.  (If SiteAdvisor were an AMO add-on there’s a good chance it would have been downgraded to “preliminarily reviewed” by now.)

Leaks in the Lastpass, Fireshot, and Amabay add-ons have been fixed.

Miscellaneous

Two weeks ago I mentioned that the not-yet-public areweslimyet.com detected a large regression in memory consumption caused by incremental garbage collection.  Bill McCloskey adjusted the GC and CC heuristics in order to fix this.

Justin Lebar previously added monitoring of available physical and virtual memory monitoring on Windows;  if either goes below a threshold Firefox attempts to free up memory.  This week he added monitoring of available commit space.

When you switch to a new tab, decoded image data from the old tab is kept until a 20 second timer expires.  This is a good idea because you might switch back to the original tab quickly.  However, until this week, this don’t-discard-it-immediately behaviour was also used when a tab is closed!  Justin Lebar landed a patch which separates these two cases, i.e. it discards decoded image data immediately when a tab is closed.

I tweaked the presentation of the “window-objects” sub-tree in about:memory so that the memory used within each browser tab is more obvious.  This is another step towards true per-tab memory reporting.  Here’s an example for a tab in which I navigated first to valgrind.org and then to www.mozilla.org;  the URL within the “top(…)” line is the one shown in the address bar.

│   ├──1,933,824 B (02.68%) -- top(http://www.mozilla.org/, id=13)
│   │  ├──1,497,480 B (02.08%) -- active
│   │  │  ├──1,496,456 B (02.08%) -- window(http://www.mozilla.org/)
│   │  │  │  ├────876,912 B (01.22%) -- layout
│   │  │  │  │    ├──569,456 B (00.79%) ── arenas
│   │  │  │  │    ├──238,224 B (00.33%) ── style-sets
│   │  │  │  │    └───69,232 B (00.10%) ── text-runs
│   │  │  │  ├────316,216 B (00.44%) ── style-sheets
│   │  │  │  └────303,328 B (00.42%) ── dom [2]
│   │  │  └──────1,024 B (00.00%) -- window([system])
│   │  │         └──1,024 B (00.00%) ── dom
│   │  └────436,344 B (00.61%) -- cached
│   │       └──436,344 B (00.61%) -- window(http://valgrind.org/)
│   │          ├──312,496 B (00.43%) -- layout
│   │          │  ├──170,448 B (00.24%) ── style-sets
│   │          │  ├──125,040 B (00.17%) ── arenas
│   │          │  └───17,008 B (00.02%) ── text-runs
│   │          ├───76,600 B (00.11%) ── dom
│   │          └───47,248 B (00.07%) ── style-sheets

Bug counts

This week’s bug counts:

  • P1: 24 (-5/+0)
  • P2: 132 (-2/+3)
  • P3: 92 (-1/+6)
  • Unprioritized: 0 (-1/+0)

We only had to triage five bugs in today’s MemShrink meeting.  I don’t remember ever having such a small number.  As a result, we spent some time reviewing the P1 bugs, and decided that several could be downgraded to P2 because the situation had improved in some fashion.

Categories
about:memory add-ons Firefox Memory consumption MemShrink

MemShrink progress, week 38

After last week’s action, this week was quieter for MemShrink.

Add-ons

I mentioned last week the plan to test the top 100 add-ons for memory leaks.  Unfortunately we learned this week that the list I gathered is the top 100 installed add-ons, not the top 100 enabled add-ons.  It’s quite possible that a lot of installed “third-party” add-ons (those installed by a program outside of Firefox, such as anti-virus programs) are disabled, in which case this top 100 list won’t reflect actual usage.

As a result, the top 100 testing is on hold until we can resolve this issue.  Telemetry data may help, though that data might exhibit opt-in biases.  I’m also going to investigate the possibility of changing the daily ping to distinguish between enabled and disabled add-ons, which would give us fully representative data.

Landed patches

I merged the “dom+style” and “layout” trees in about:memory.  This is a step towards per-tab memory reporting.  It also probably breaks about:nosy 🙁

I also converted the DOM memory reporters to the new style, added measurement of URIs and links, and added measurement of the FramePropertyTable.  These changes reduced about:memory’s “heap-unclassified” by several MBs in common cases.

Bill McCloskey changed the GC marker stack so it shrinks periodically.  Previously it could grow from 256KB to over 2MB and would never be shrunk once that happened.

Bug counts

This week’s bug counts:

  • P1: 29 (-0/+1)
  • P2: 131 (-4/+6)
  • P3: 87 (-3/+7)
  • Unprioritized: 1 (-2/+1)
Categories
about:memory add-ons Firefox Garbage Collection Memory consumption MemShrink

MemShrink progress, week 37

It’s been a huge week for MemShrink.

Add-ons

I filed a bug on testing the top 100 add-ons for memory leaks.  The majority of these add-ons are not available on AMO and so are not subject to the leak checking done during AMO reviews.  The instructions and current test results are here.  Randell Jesup helped identify some some add-ons with unclear identities, and Nils Maier and Archaeopteryx have both tested several add-ons.  A month ago I said that add-ons that leak are the #1 problem for MemShrink, and this bug represents a huge step towards reducing that problem.  We’ll need lots of additional help to test this many add-ons, so please feel free to jump in!

As far as I know, comprehensive testing of the most popular add-ons has never been done before.  Although this testing is aimed at finding memory leaks, it’s quite possible it will uncover various other problems with add-ons.  Indeed, it’s clear just from looking at the list that quite a few of the non-AMO add-ons are of dubious merit, and this has stimulated some interesting discussion on dev-platform about third-party add-ons, i.e. those that are installed into Firefox by an external program.

The following add-ons had leaks fixed or mostly fixed by their authors: McAfee Site Advisor, Ghostery, Spool, Screengrab (fix version), Roboform, UF Comment Board Tools,  Geenstijl, Long URL Please, HTML Desktop Notifications.

Nils Maier greatly improved the documentation on common causes of memory leaks in add-ons.  This was a MemShrink:P1 bug.  That doesn’t mean the document can’t be improved further, however;  it’s a wiki so please continue to add to it if you can.

Regression testing

John Schoenick’s excellent areweslimyet.com is getting close to being ready for a public unveiling.  The site is currently live but password-protected, simply to prevent large numbers of people accessing it before it’s ready.  (If you want a sneak peek, contact me for the password.)  More importantly, it identified a large regression in memory consumption caused by the landing of the JS engine’s incremental garbage collector on February 19, which Bill McCloskey is investigating.

Tim Taubert improved Firefox’s regression testing to ensure that no new DocShell and DOMWindow leaks are introduced.  This was a MemShrink:P1 bug because these leaks are quite easy to introduce.  Tim wrote some more about this change here.

Cycle collector

Jan Honza Odvarko’s about:ccdump add-on made it onto AMO this week.  Jan has been using this to hunt down some tricky leaks in Firebug.

Marco Bonardo fixed a leak in PlacesUtils.removeLazyBookmarkObserver() that I won’t pretend to understand, but which was a MemShrink:P1.  Marco also fixed a leak that occurred after bookmarking a page, closing the window, and opening a new window.

Olli Pettay fixed an equally impenetrable (to me) leak involving nsFormFillController::mFocusedInput.

Kyle Huey fixed a leak in the About Firefox dialog.

Andrew McCreight added additional information to cycle collector’s error console logging, which helps with identifying and debugging cycle collector problems.

Miscellaneous

Jason Duell fixed an obscure leak involve web sockets.

Bill McCloskey fixed a minor JS engine leak.

I added a new memory reporter called “js/runtime/gc-marker”.  It starts out at 256KB and I saw it reach 2.8MB when running MemBench.  I also converted the existing source image reporters to the new style, which seems to haved fixed some bogus results in about:memory as a side-effect.

Quote of the week

Adam Overa of Tom’s Hardware did another browser Grand Prix.  Devin Coldewey used the results from the Grand Prix to conclude that Firefox and Chrome are both great, and which you prefer basically comes down to a matter of taste.  While that is an interesting conclusion, what caught my eye was this comment, by Balaji Viswanathan:

I guess the title could have been “All browsers suck equally bad now”. Firefox and Chrome have grown to become more resources hogs – closer to IE. These days I shudder to fire up the firefox as the memory leaks are terrible. Two hours of Firefox with a Facebook tab is enough to gobble up 25% of my memory. With Chrome, I would have 20 rendering processes showing up when I would have just 3 tabs open.

In the first 2 years of Chrome I had probably 2 or 3 crashes. These days, it crashes every week or so. I long for the 2006 era Firefox – clean, simple and lean. Its been a long time since I have used IE actively and with Safari lesser said the better. For the developers, we have to deal with the ultra slow update cycles of IE and ultraspeed update cycles of Chrome. In short, we are getting worse and worse in the browser arena. The market is ripe for a breakthrough.

“2006 era Firefox” would have been Firefox 1.5 or Firefox 2.  Balaji may think he’s nostalgic for browsers of that era, but really he’s nostalgic for the web of that era, which was massively simpler than the web of today.  If Balaji tried Firefox 1.5 or Firefox 2 today — with no HTML5 support, no JavaScript JIT and no hardware acceleration — today, I’m sure he’d quickly decide that 2012-era browsers aren’t so bad after all.  I guess the interesting question is this: why do people blame browsers instead of websites for high resource consumption?

Bug counts

This week’s bug counts:

  • P1: 28 (-5/+5)
  • P2: 129 (-12/+7)
  • P3: 83 (-8/+12)
  • Unprioritized: 2 (-2/+2)

That’s a net reduction of one bug.  But more importantly, there is lots of movement, which is great!  It means that problems are being identified and fixed.

(Finally, commenters please note that I’ve turned on the WP-reCAPTCHA plug-in, and this the first post I’ve written since doing so.  In my testing it’s worked fairly well.  Hopefully it won’t cause problems.)