MemShrink progress, week 34

Add-ons

The AMO add-on review checklist was amended a couple of weeks ago to include checks for zombie compartments.  This change is bearing fruit:  Andreas Wagner and Kris Maglione have found more than 10 submitted add-ons that have leaks.  See here, here, here, here, here.  With the experience they are gaining, we should be able to greatly improve the documentation on common causes of leaks.

In less positive news, two leaks were found in the add-on SDK.  Fortunately the SDK team has proven to be effective at fixing identified leaks quickly in the past, hopefully they’ll do so again!

Finally, Jason Tackaberry fixed a zombie compartment in NoSquint 2.1.2.  The fix is in the latest version (2.1.5).

about:nosy

Andrew Sutherland wrote a extension called about:nosy which is like about:memory on steroids.  It’s oriented around hiding many of the details and instead assigning blame for memory allocations to particular tabs (and similar things).  It also features graphs showing how the memory consumption of each tracked entity changes over time;  this latter feature is quite memory and CPU-intensive, however.  If you are running a recent Nightly build of Firefox, download about:nosy here and you won’t even have to restart Firefox to see it, just type “about:nosy” into the address bar.

Forthcoming changes will make about:memory more like about:nosy, by measuring more things on a per-tab basis.  And for a long time I have had a vague plan that one day someone who knows about UX will write a user-friendly alternative to about:memory that focuses just on per-tab memory consumption, and about:nosy is a good indicator of what that might look like.

Miscellaneous

Saptarshi Guha from the metrics team did an analysis of the physical memory consumption of Firefox 10, 11 and 12, based on telemetry data.  The post is heavy going for those who aren’t experts at statistics and R, but the two main conclusions of interest are (a) that add-ons significantly increase resident memory consumption, and (b) Firefox 12 is consuming more resident memory than Firefox 11.  This latter fact matches something we’d seen in today’s MemShrink meeting when looking at data from John Schoenick’s in-development version of areweslimyet.com — something in early January caused a significant memory consumption regression.  John is working on narrowing down which change caused this.

Gian-Carlo Pascutto overhauled the safe browsing implementation.  I don’t claim to understand this change at all, but I think it reduces memory consumption when the database is updated.  Better explanations from those who understand this change are welcome!

I reduced the amount of memory allocated when generating about:memory by roughly 30%.

I wrote a detailed discussion of the benefits of reducing memory consumption.

Bug Counts

Here are this week’s bug counts.

  • P1: 22 (-0/+0)
  • P2: 134 (-4/+8)
  • P3: 75 (-3/+4)
  • Unprioritized: 2 (-2/+1)

24 Responses to MemShrink progress, week 34

  1. I’m saddened that we didn’t have a mechanism in place to flag the memory regression when it happened in January.

    • This is absolutely a big failure. Nicholas is on record stating that it took two years and six major versions to get Firefox’s memory handling back to what it was in version 3.6. That is unacceptable. Mozilla should take stock before proceeding with any more WebAPI functionality. Is turning Firefox into a HTML5-based operating system, really that crucial, right now? Mobiles will be around for a long time. User’s will always have to opt-in to Firefox on a mobile. Is it really something that you have get into right now, before ensuring fundamentals are fixed? Pleeease make memory management the number one priority until you can guarantee Firefox will never experience unseen memory management regressions ever again because here’s an all too familiar story yet again:

      Today I came home and my PC was unusable. I have a machine with less than 1GB of RAM. Sure that’s not very much these days but it’s been the same for most of Firefox’s lifespan so it’s a good consistent testing foundation. Yet again I get this:

      Firefox 10 using 1.02 GB memory on a 960 MB RAM computer (February 2012)

      Ok so I have NoSquint installed and I just find out it’s got a leak. Maybe the new version will help. However I’ve heard all this maybe this and maybe that before! I’m sick of it!

      Please, oh please, I beg of you Mozilla, get your memory management foundations right! Fix memory consumption and never let it regress! Stop making Firefox imitate a vibrator and get the fundamentals right!. This is not a troll. This is a long-term Firefox user who needs a healthy Firefox / Firebug combination to do his job as a web developer, let alone anything I use it for at home.

      If you think I’m carrying on and that the previous example is probably just a once off:

      Firefox 4 using 1.24 GB memory on a 960 MB RAM computer (April 2011)

      I mean really, I know there’s a been a heap of progress within MemShrink but really, one could be forgiven for asking id much has really changed in a holistic sense, don’t you think?

      • Benedikt P. [:Mic]

        Honestly I feel sad each time I see one of your postings. You’re wasting precious time of your life for no good reason and I can’t possibly understand why one would like to do that :(

    • Caspy,

      I’ll echo your sentiments with regard to the regression in memory consumption. It is indeed sad news. I have noticed rapid improvements in Firefox Beta versions 7 – 11. Things have just got better and better. Faster, smoother, more stable, it’s absolutely brilliant. It would be a shame to see Firefox take a step back.

      Nicholas,

      Please can you keep us informed of John Schoenick’s progress on tracking down the problem? Is there a blog we can follow or bug(s) on Bugzilla specifically for the work he is doing?

      11 Beta 1 is seriously good, please keep up the hard work and good luck to you, John and anyone else dealing with the regression.

      • Nicholas Nethercote

        https://bugzilla.mozilla.org/show_bug.cgi?id=704646 is the bug tracking areweslimyet.com. It currently does measurements once per day, John is in the process of changing that to run either once per hour or (even better) once per commit. Once he does this he’ll be able to re-run the old builds and get a much narrower regression window. John’s been doing a great job, so I hope he’ll have this done by next week. There’s no bug tracking this change, however.

    • Why? It’s not in any released product. It got caught early in the development cycle. And it has been pointed out that automated detection is being worked on, so that ‘memory regressions’ will be caught even early.

      Should you happen to be an Aurora user, you should comfortable with encountering the occasional bug, though it is unlikely you ever noticed the impact of the regression.

      I see absolutely no reason to be sad. In fact, you should be happy about the memshrink guys doing a good job.

      • Bastiaan,

        You’re right. No real reason to be sad and as you and Nicholas have pointed out since I originally posted, it is being worked on and should be sorted out by the time that 12 is released in April.

        I am very happy about Memshrink. If you look at the rest my paragraph in reply to Caspy, I’ve got nothing but love for the project and what it’s done and continues to do for Firefox

  2. Hi, Nicholas! Thank you for your great work making Firefox slim and fast!
    I’ve noticed, however, since about:memory rewrite week ago, network-memory-cache become “leaky”, reporting its size about 2-3Mb, even after long idle periods.

    • Nicholas Nethercote

      2-3MB doesn’t sound unusual at all. Why do you think this is a problem?

      • Shoudn’t network-memory-cache be allocated/deallocated quickly, on-the-fly? I see no reason to keep it allocated (and wasting RAM) after network transfer was completed.
        Also, network.buffer.cache.size=32768 and network.buffer.cache.count=24 by default, so netcache shouldn’t exceed 768Kb. Memleak in netcache?

  3. Great work on getting the memory beast under control! As someone who’s a ‘balloon user’ (i.e. I usually run about 15 tabs, but can shoot up to 50 or so, then back down) Firefox Just Works(TM) now. I don’t have to restart the browser to get it back to a decent state. A lot of the hitches where the browser gets hung up seem to have disappeared as well.

  4. For the safe-browsing implementation changes, there is a summary here if that helps?
    http://www.morbo.org/2012/02/new-safebrowsing-backend.html

    Thank you for the post! :-)

  5. I’m not sure if there is a project snappy related blog anywhere or not, but i’ll just post this here.

    Since I’ve updated to Firefox 11 beta, i’ve noticed a clear night and day improvement to various tasks responsiveness – switching tabs, scrolling, etc. I’m not sure exactly what change was made, or if it was a combination, but the improvement is very nice. Thanks to all involved.

  6. I just tried the about:nosy extension, and was surprised to see about 25 instances of about:blank taking up 3mb each. My best guess is that these are actually tabs in other tab groups, that haven’t been loaded yet, but it would seem like there are better ways of keeping those around than that?

    As always, thanks for the updates, and keep up the good work!

    • Nicholas Nethercote

      Take a look at https://bugzilla.mozilla.org/show_bug.cgi?id=681201 which is about reducing the memory consumption of not-yet-loaded tabs. Some progress has been made, but there’s more to do.

      You’re seeing 3MB per unloaded tab, i.e. 75MB with 25 about:blanks tabs? That sounds really high, in bug 681201 we were seeing typically less than 0.5MB. Add-ons, maybe?

      • It does appear to be profile related (doesn’t happen in a test profile when I restart with tabs in other tab groups), but it doesn’t appear to be add-on related. I just tested with all other add-ons disabled, and I get the same result. I have a dump of about:memory (verbose) and about:support if that will help?

  7. I am currently on my girlfriends laptop round at her house and I am surfing the net using Firefox 10.0.1.
    The memory usage is terrible. It may be because of the two McAfee add-ons that she has, I don’t know, but I can see now first hand how some Firefox users are having an amazing experience and others are still lagging behind. The memory usage as opposed to being 200,000-230,000kb as it would be on my laptop with 11 beta 2 is about 350,000kb or more. That said, there is no loss of performance and that is what matters most. I am not looking for answers or solutions, merely saying that the same or very similar software used on one computer can perform completely differently on another and now I appreciate that even more having witnessed it for myself.

    • Nicholas Nethercote

      If you can disable the McAfee add-ons and try again I’d be *very* interested to know the results. One thing I’ve learnt is that no add-on is above suspicion when it comes to affecting memory consumption.

      • Hi Nicholas,

        I did some very very rudimentary tests earlier at my girlfriends house on her laptop and here is what I found.

        With the two McAfee add-ons running everything was fine until I got to about 8 tabs and then the memory began to increase. Then it just climbed up to about 450,000 kb. If I was running the same tab load on my latop, I would expect just over half that amount of memory to be reported in task manager.

        I have to say that there was no loss of performance, but the amount of memory being used was very high for the tab load.

        The first tab that I had open was google.co.uk. I right clicked on that tab and hit “Close Other Tabs”. The memory stayed at 418,000 kb and that was it. It stayed at that level for about 10 minutes whilst I went and did other stuff. It wasn’t going up or down. 400MB with just Google UK open and nothing else. It’s diabolical.

        So I then disabled the 2 McAfee add-ons and every went back to normal. I used a heavy tab load than before. I opened more tabs with more complex content. I must have had about 15 tabs open and the memory use peaked at 386,000 kb.

        Then I logged out of Facebook and Hotmail, without closing any tabs, and within 10 seconds the memory had dropped to 330,000 kb.

        Then I closed all but 3 tabs and within 30 seconds the memory was about 180,000.

        Then I closed all but one tab and the memory settled at just under 100,000 kb. Back to normal. Exactly how I would expect Firefox 10.0.1 to perform.

        The extensions in question were McAfee ScriptScan for Firefox 14.4.1 and McAfee Site Advisor 3.4.1

        I didn’t have time to test them individually as things got hectic after that, but as you say Nicholas, the seemingly innocent extensions were ruining the memory performance of a perfectly good browser.

        I could still load pages etc and navigate around as fast, but the numbers on task manager were sky high.

  8. Nicholas, thought it might be of interest:
    http://lifehacker.com/5884941/browser-speed-tests-chrome-17-firefox-10-internet-explorer-9-and-opera-1161
    Firefox (and by extension really the Memshrink team) gets props for memory.

  9. Hi Nicholas,

    I did some very very rudimentary tests earlier at my girlfriends house on her laptop and here is what I found.

    With the two McAfee add-ons running everything was fine until I got to about 8 tabs and then the memory began to increase. Then it just climbed up to about 450,000 kb. If I was running the same tab load on my latop, I would expect just over half that amount of memory to be reported in task manager.

    I have to say that there was no loss of performance, but the amount of memory being used was very high for the tab load.

    The first tab that I had open was google.co.uk. I right clicked on that tab and hit \"Close Other Tabs\". The memory stayed at 418,000 kb and that was it. It stayed at that level for about 10 minutes whilst I went and did other stuff. It wasn\’t going up or down. 400MB with just Google UK open and nothing else. It\’s diabolical.

    So I then disabled the 2 McAfee add-ons and every went back to normal. I used a heavy tab load than before. I opened more tabs with more complex content. I must have had about 15 tabs open and the memory use peaked at 386,000 kb.

    Then I logged out of Facebook and Hotmail, without closing any tabs, and within 10 seconds the memory had dropped to 330,000 kb.

    Then I closed all but 3 tabs and within 30 seconds the memory was about 180,000.

    Then I closed all but one tab and the memory settled at just under 100,000 kb. Back to normal. Exactly how I would expect Firefox 10.0.1 to perform.

    The extensions in question were McAfee ScriptScan for Firefox 14.4.1 and McAfee Site Advisor 3.4.1

    I didn\’t have time to test them individually as things got hectic after that, but as you say Nicholas, the seemingly innocent extensions were ruining the memory performance of a perfectly good browser.

    I could still load pages etc and navigate around as fast, but the numbers on task manager were sky high.