MemShrink progress, week 39

Add-ons

Versions of McAfee’s SiteAdvisor add-on prior to 3.4.1.195 have been soft-blocked due to an extreme memory leak.  This means that it will be disabled within Firefox, but users can re-enable it if they want to.  According to McAfee, most users should have been upgraded to 3.4.1.195 by now.  Unfortunately, version 3.4.1.195 still has some leaks, and progress on fixing them has stalled.  (If SiteAdvisor were an AMO add-on there’s a good chance it would have been downgraded to “preliminarily reviewed” by now.)

Leaks in the Lastpass, Fireshot, and Amabay add-ons have been fixed.

Miscellaneous

Two weeks ago I mentioned that the not-yet-public areweslimyet.com detected a large regression in memory consumption caused by incremental garbage collection.  Bill McCloskey adjusted the GC and CC heuristics in order to fix this.

Justin Lebar previously added monitoring of available physical and virtual memory monitoring on Windows;  if either goes below a threshold Firefox attempts to free up memory.  This week he added monitoring of available commit space.

When you switch to a new tab, decoded image data from the old tab is kept until a 20 second timer expires.  This is a good idea because you might switch back to the original tab quickly.  However, until this week, this don’t-discard-it-immediately behaviour was also used when a tab is closed!  Justin Lebar landed a patch which separates these two cases, i.e. it discards decoded image data immediately when a tab is closed.

I tweaked the presentation of the “window-objects” sub-tree in about:memory so that the memory used within each browser tab is more obvious.  This is another step towards true per-tab memory reporting.  Here’s an example for a tab in which I navigated first to valgrind.org and then to www.mozilla.org;  the URL within the “top(…)” line is the one shown in the address bar.

│   ├──1,933,824 B (02.68%) -- top(http://www.mozilla.org/, id=13)
│   │  ├──1,497,480 B (02.08%) -- active
│   │  │  ├──1,496,456 B (02.08%) -- window(http://www.mozilla.org/)
│   │  │  │  ├────876,912 B (01.22%) -- layout
│   │  │  │  │    ├──569,456 B (00.79%) ── arenas
│   │  │  │  │    ├──238,224 B (00.33%) ── style-sets
│   │  │  │  │    └───69,232 B (00.10%) ── text-runs
│   │  │  │  ├────316,216 B (00.44%) ── style-sheets
│   │  │  │  └────303,328 B (00.42%) ── dom [2]
│   │  │  └──────1,024 B (00.00%) -- window([system])
│   │  │         └──1,024 B (00.00%) ── dom
│   │  └────436,344 B (00.61%) -- cached
│   │       └──436,344 B (00.61%) -- window(http://valgrind.org/)
│   │          ├──312,496 B (00.43%) -- layout
│   │          │  ├──170,448 B (00.24%) ── style-sets
│   │          │  ├──125,040 B (00.17%) ── arenas
│   │          │  └───17,008 B (00.02%) ── text-runs
│   │          ├───76,600 B (00.11%) ── dom
│   │          └───47,248 B (00.07%) ── style-sheets

Bug counts

This week’s bug counts:

  • P1: 24 (-5/+0)
  • P2: 132 (-2/+3)
  • P3: 92 (-1/+6)
  • Unprioritized: 0 (-1/+0)

We only had to triage five bugs in today’s MemShrink meeting.  I don’t remember ever having such a small number.  As a result, we spent some time reviewing the P1 bugs, and decided that several could be downgraded to P2 because the situation had improved in some fashion.

23 Responses to MemShrink progress, week 39

  1. Is this a memory bug?
    1. Open about:memory
    2. Press F5 a few times
    3. See memory grow > 1 MB per time (mainly js and heap unclassified).
    4. The memory goes down after some time, but never as low as when at 1.

    • Nicholas Nethercote

      That’s expected behaviour. Some JavaScript code is run to generate about:memory. That code allocates some memory. Every so often the garbage collector runs, and the memory consumption drops again.

      A few weeks back I spent some time minimizing the amount of memory allocated while generating about:memory, so the effect is smaller than it used to be.

  2. MemShrink progress posts are one of my favorite things to read on the web. Thanks.

  3. It would like to see a Blogpost on which Memshrink changes make it into a final Firefox version. It is good to know which changes are in trunk, but I also want to know which land in which release.

  4. Thanks for your efforts in Memshrink project. I have been following this blog for the past few weeks and managed to improve my FF’s memory print. Uninstalling Mcafee Site Advisor was a great improvement.

    Few hours ago, I installed FF 11.0, and the improvement was noticeable. From yesterday, to today the difference is significant. To me this means that FF has improved beyond fixing addon leaks. I have numerous addons installed, but as far as I know, none of them received any updates in the past week, which means that the improvement I saw from FF10 to FF11 was not only related to addons.

    I noticed lower start-up memory footprint, which was around 400MB few days ago, and now it is around 270MBs with FF11. And the rate of memory growth with usage is now slower. And the effect of restoring memory by closing tabs is more noticeable. FF10 memory usage would go to 1GB+ after heavy usage, I didn’t notice FF11 going beyond 500MBs so far.

    It’s been only few hours that I used FF11, so I will post an update if the experience changes; but so far so good.

  5. │ ├──27.41 MB (10.93%) -- sqlite
    │ │ ├──13.57 MB (05.41%) -- places.sqlite
    │ │ │ ├──13.35 MB (05.32%) ── cache-used [4]
    │ │ │ └───0.22 MB (00.09%) ++ (2 tiny)
    │ │ ├──13.25 MB (05.28%) ── other
    │ │ └───0.59 MB (00.24%) ++ (9 tiny)
    Ping: https://bugzilla.mozilla.org/show_bug.cgi?id=699708

    • Nicholas Nethercote

      No progress on that bug. Unfortunately it’s caused by some design decisions that are baked fairly deeply into SQLite and can’t be changed easily. There’s no plan to move forward on it.

      However, it looks like you have an older profile which uses the smaller page size and suffers 50% waste. If you create a new profile you should get the larger page size and the waste with drop to 10%. Creating a new profile is a bit of a pain, I admit, though if you use Sync it’s not as bad — up to you if you think the ~11MB you’ll save is worth the effort.

  6. Hi Nicholas,

    As you may remember it was me that originally highlighted the awful memory performance of McAfee Site Advisor. I am sat here on my girlfriend’s laptop again and I am pleased to say that Firefox 11 and the updated 3.4.1.195 gives exactly the kind of memory performance that I would expect if I was using no add-ons at all. Whatever leaks they have left to fix, rest assured from what I can see, they are very minor.

  7. Excellent job creating areweslimyet.com ! I wonder how long (and how much effort) it would have taken to discover and track down the regression without it. Are you planning to add measurements for different workloads as well, to get a better insight?

    • Nicholas Nethercote

      No plans at the moment. We might make some after it’s been public for a while if we feel the need.

  8. @Nicholas

    I wonder what happened to the regression you mentioned about a month ago that the memory performance of FF12 and FF13 was worse than FF11? At that time you said it was a regression happened most likely in early Jan, but it was not identified at that time what the cause was.

    • Nicholas Nethercote

      Good question! As far as we can tell it was a false positive from areweslimyet.com. Something about the measurement was artificially inflating the numbers, I can’t remember the details now, but John tweaked the measurement code and the regression went away. Automated testing of this stuff is hard, it’s easy to get bogus results if you’re not careful.

      That doesn’t absolutely guarantee that no regression occurred, but areweslimyet.com will never provide perfect protection against regressions, and it was enough to satisfy me.

  9. I’m confused. Was the regression real or not? The bug you pointed to says the fix is for FF13, so if 12 has a regression, that’s a huge deal(breaker). If it’s one of those things where it’s not clear the regression ever existed, that’s good–unless that theory turns out to be wrong. But if there was a real regression, shouldn’t the bug fix land in 12?

    • Nicholas Nethercote

      We’ve seen two regressions on AWSY. The one in early January was apparently a false positive. The more recent one was real and involved incremental GC; that particular problem has been fixed, but incremental GC has also been disabled for the moment due to other problems.

  10. Wooooohoo!

    I don’t know how this is possible and whether it will last but currently I’m running Firefox 11, I have the latest Firebug installed, though I’ve not used it all that much this session, I have four tab groups open with a total of about a dozen tabs in three groups and 30 in this current group I’m writing from. Additionally I have 25 active Add-ons and 2 ‘disabled’ Add-Ons installed. My profile is ancient and is causing some annoying issues but nevertheless … MemChaser shows me using just 210MB of memory!!

    I can’t say this is the ultimate victory for MemShrink but it sure as hell is a great milestone in terms of my own anecdotal experiences.

    I’ve still not changed hardware or OS from the days well before Firefox 3 so those elements have remained constant.

    I am not using WebGL so that might be a factor. I’ve tried it a few times over the months but inevitably it gives me a BSOD.

    I have adopted defensive measures against memory bloat/performance drop in the form of Wallflower and ShareMeNot whilst I’ve never enabled the AwesomeBar integration in my Delicious Add-On (tried it a long time ago, it clearly caused bloat, so I turned it off – I’m not a completely stupid user! LOL). In addition I have “Don’t load tabs until selected” enabled which is a PITA as it pretty much defies the whole purpose of tabbed browsing. However at least memory/performance is reasonably stable!

    I still get significant fluctuation in usage so that confirms GC and so forth are working better than ever. Flash is still a memory/performance killer but hey, Rome wasn’t built in a day (see, I can have generous moments :) )

    Congrats to all those working on the MemShrink effort. I sincerely hope it MemShrink is nowhere near finishing up and that regression analysis is rock solid, but IMHO it’s really made substantial progress and being a frequent critic myself, it’s only right that I acknowledge that. Superb effort thus far everyone!

  11. Hi Nicholas,

    I wanted to say thanks for updating this blog and the work everyones doing on this – its making a HUGE difference in speed and memory usage.

    I had a question – Chatzilla comes up as Heap-unclassified, I was wondering if it should be as that or if it should be considered a tab for memory purposes?

    • Nevermind, It reports some of it:

      8.42 MB (03.68%) — top(chrome://chatzilla/content/output-window.html, id=127)
      │ │ └──8.42 MB (03.68%) — active
      │ │ └──8.42 MB (03.68%) — window(chrome://chatzilla/content/output-window.html)
      │ │ ├──4.36 MB (01.90%) ── dom [2]
      │ │ ├──3.98 MB (01.74%) — layout
      │ │ │ ├──3.81 MB (01.66%) ── arenas
      │ │ │ └──0.17 MB (00.07%) ++ (2 tiny)
      │ │ └──0.08 MB (00.04%) ── style-sheets
      │ ├───4.55 MB (01.99%) — top(chrome://chatzilla/content/output-window.html, id=151)
      │ │ └──4.55 MB (01.99%) — active
      │ │ └──4.55 MB (01.99%) — window(chrome://chatzilla/content/output-window.html)
      │ │ ├──2.40 MB (01.05%) ── dom [2]
      │ │ └──2.15 MB (00.94%) ++ (2 tiny)
      │ └───2.70 MB (01.18%) — top(chrome://chatzilla/content/output-window.html, id=143)
      │ └──2.70 MB (01.18%) — active
      │ └──2.70 MB (01.18%) ++ window(chrome://chatzilla/content/output-window.html)

      I still get a rather large heap-unclassified when using Chatzilla though.

      • Nicholas Nethercote

        How big is the heap-unclassified with Chatzilla? I use Chatzilla with a nightly build myself and I rarely see heap-unclassified exceed 20%.

  12. Hi,

    I couldn’t find your contact so I’m asking for access to the areweslimyet.com site here. Can you help me?

  13. I would like to submit a test-case in advocacy for smarter image decoding heuristics in Firefox. Opening this page [1] caused Firefox to eat about 250MB of RAM. This caused the browser and then the system to momentarily freeze as the OS struggles to free memory. About:memory shows over 200MB in images/content/uncompressed-heap.

    Tested on FF 11.0 on OSX 10.5

    [1] http://brendaneich.com/