MemShrink progress, week 26

It’s been six months since MemShrink started!  Here are this week’s events of note.

Memory Reporting

I integrated DMD support into Firefox.  This is a big deal because DMD is crucial for improving the coverage and correctness of memory reporters.  There are several consequences of this.

  • You no longer need to apply a patch to Firefox if you want to use DMD, you just have to configure with --enable-dmd.  (Full instructions are here.)
  • Any memory reporter written in the new style (i.e. using a nsMallocSizeOfFun function created with NS_MEMORY_REPORTER_MALLOC_SIZEOF_FUN) will automatically get coverage in DMD.
  • DMD finds bugs in memory reporters, but it can also find bugs elsewhere.  For example, this week it found that some BaseShapes were being shared incorrectly, which Brian Hackett fixed.

I updated the layout memory reporters to use the new style.  I also updated the documentation on writing memory reporters, to cover more complicated topics like how to measure classes that involve inheritance.  I also added a request:  if you write a memory reporter, please add me as a co-reviewer.

Johnny Stenback implemented per-window DOM memory reporters, which gives much more detail for DOM memory usage in about:memory.  Here’s an example:

per-window dom reporter output in about:memory

Memory Usage Improvements

Boris Zbarsky lazily initialized some rulehash tables, saving about 45KB per blank tab;  this is a nice win for those people who have on-demand tab loading combined with sessions with many tabs.  The same patch also made one of the rulehash tables smaller when it is initialized, which can cause multi-MB savings on workloads of only a few tabs.

Bobby Holley fixed a bad memory leak in xpconnect that could cause unbounded amounts of memory to be leaked, albeit in somewhat unusual circumstances.  The bug was only in Firefox 9 and Firefox 10;  it’s been fixed for 10, and the patches to fix for 9 are ready to land.

Fabrice Desré fixed a regression that caused a zombie compartment when visiting addons.mozilla.org.

Bug Counts

Here are the current bug counts.

  • P1: 27 (-1/+1)
  • P2: 139 (-5/+3)
  • P3: 62 (-0/+2)
  • Unprioritized: 1 (-0/+1)

That’s a net change of +1, and we only had to triage seven bugs in today’s meeting.  And if the trees hadn’t been closed for the past few days I could have fixed two more P2 bugs.

Just for fun, I also counted the number of RESOLVED FIXED (i.e. genuine, not duplicates or invalid) MemShrink bugs.

  • P1: 29
  • P2: 70
  • P3: 14
  • Unprioritized: 35

(If you’re wondering why the number of unprioritized fixed bugs is so high, it’s because MemShrink bugs are only prioritized in MemShrink meetings, which means that any bug that gets filed and fixed in less than 7 days doesn’t get prioritized.)

So there are more bugs still open than have been fixed, but the rate of bug growth is close to zero.

30 Responses to MemShrink progress, week 26

  1. I am very sensitive to memory usage as my laptop has only 1GB RAM and Win7 uses 60% of it. So I got exited by reading “Converting a Chrome user to Firefox” and gave a try. The web site I use the most is gmail, so I have used it as a benchmark to compare RAM usage (from Win Task Manager).
    Procedure: fresh browser start, one tab, open my gmail account:
    Firefox 3.6 : 70M (firefox.exe) + 10 (plugin-container.exe) + googletalkplugin.exe (9M).
    Chrome 16.0 : 53 + 43 + 11 M (chrome.exe) + googletalkplugin.exe (9M).
    Now uninstall Firefox 3.6 from Control Panel, remove all customization. Delete manually Mozilla Firefox folder on Program Files. Instal Firefox 8.0.1, add-ons: keep only dictionaries, installed by me. Again open single tab, my gmail account:
    Firefox 8.0.1 : 88 M(firefox.exe) + 10 (plugin-container.exe) + googletalkplugin.exe (9M).

    So I am a little frustrated to see that Firefox 8.0.1 uses more memory than Firefox 3.6. Anyway it uses less than Chrome, so I’ll keep using it.

    • I feel your pain, Red, because I jumped from 3.5 right up to 7.01 and then straight to the 8 beta. (I’m now using the 9 beta.) The memory usage is indeed higher overall, but I’m finding the browser itself is actually a little nicer to work with in a lot of respects and it feels significantly faster.

      My hope is that this continued work on MemShrink will eventually drop the memory usage down to 3.0 levels or better (’cause let’s face it, 3.0 was no prize on memory either). I especially hope to see some progress in the Firebug leak, though the developer seems to be waiting for better feedback in terms of improved memory reporting. Hopefully this new stuff will help out in that department and Firebug can fix the leak, which is far more relevant to me than even the other leaks (at least most of them, anyway).

      • Nicholas Nethercote

        “The memory usage is indeed higher overall, but I’m finding the browser itself is actually a little nicer to work with in a lot of respects and it feels significantly faster.”

        Ding ding ding, we have a winner. You shouldn’t really care about memory usage in and of itself, just how it affects your browsing experience in terms of speed and crash rates. (Just imagine if you couldn’t measure memory usage easily — you couldn’t complain about it, you’d have to complain about things you can sense yourself while browsing, i.e. speed and crash rates.) Sometimes using more memory makes things faster. There are lots of time/space trade-offs.

        • If this is the case (speed and crash rates), we still have a winner which is Chrome actually. I personally use Firefox but the reasons for that are totally different from being it responsive. If Chrome fulfilled only few of my wishes which it doesn’t at the moment (e.g. some Greasemonkey scripts would work in it, which do not, or it would implement OpenType features or automatic ligatures use as Firefox has and maybe some more, I cannot remember them now but I ran into them every time I decided to try to switch to Chrome) I would definitely move to Chrome on the basis of speed, responsiveness and these things.
          It’s maybe even true that Chrome uses more memory when many tabs are used, but as you have just mentioned, it’s the responsiveness and speed that you should value, not the memory itself.

          • Nicholas Nethercote

            Yep, that’s totally valid. Everyone has their own reasons for using a particular browser, but memory usage alone should not be one of them.

            (Alos, which browser is faster tends to vary; it’s amazing how many comment threads I’ve seen where someone says “FF is so slow for me, Chrome is wicked fast” and then someone says the exact opposite.)

        • Well the problem is really that if the browser gets up near 1GB of memory use it becomes highly unstable. This is on a 32-bit system. Firefox 9 can handle that better than 3.5 could, but it’s climbing into the danger zone much faster (mostly due to the Firebug leak, I believe). I haven’t been able to isolate a usage pattern for the extreme cases except that it seems to behave worse when the console and script panels in Firebug are both enabled and I’ve been doing a lot of refreshing/debugging. That said, in the ultra-bad cases the biggest memory use has been in heap-unclassified, so I’m hoping the newer reporters in 10 will clarify where things are going wrong.

          From what I see on the P1 bug related to Firebug however, the developer is still waiting for feedback from someone who can provide more detailed insight on how the add-on is leaking–whether it’s a reference it’s holding onto inappropriately, a closure, or what. I strongly suspect the worst of my woes won’t be behind me until Firebug is patched.

          • @Lummox JR Well Mozilla is adding the developer tools into Firefox, so eventually you wouldn’t need Firebug anymore.

  2. The memory in the Task Manager can be wrong in many cases. “Mem Usage” indicates the memory in physical RAM (excluding the swap). If you did not use any swap at a given moment, the value is accurate. If you ran other programs, maybe a part of Firefox will be on the swap.

    The VM Size is not good either. It includes shared memory.

    What I sometime do is noting the current global memory usage, run the program, note the new memory usage, repeat many times.
    Or use “smem” under linux.

  3. Did anyone follow up on bug 708499 comment 20? I can’t find anything.

  4. Always nice to see new progress on memshink, thanks for that and a great browser!

    Oone thing I’m wondering is about the level of analyzing you are doing for mem leaks, as getting into the guts of FF and squeezing few KB here and there seems so far away from the 100s of MB my ‘fox has often lost.. No doubt there are many reasons and they all add up.

    But are you checking if whole tabs are somehow lost etc? Closing them almost always leaves an empty hole in the tab bar, resulting in swiss cheese like tab bar after a while. Closing FF and restarting it somehow opens few of the already closed ones again, just as if they weren’t actually closed.

    Don’t know easy way to reproduce this, but while working with 300+ tabs, opening and closing dozens of tabs daily, it happens every time on restart.

    This has been happening all the way since 3.x series, just like the “crash recovery” not working (whether related to each other or not). By crash recovery problems I mean opening any number of tabs into a stable state, crashing/killing FF, and restarting doesn’t bring back the previous tabs, but usually some older subset of them.

    • Nicholas Nethercote

      Your tab bar has holes in it? That sounds *really* weird. Do you have any add-ons installed? If so, try disabling them (start in safe mode) and see if it still happens. If so, try a new profile, your existing one might be corrupted somehow.

      • Got it! Newer thought about profile corruption, whatever that might more specifically mean. Disabling all add-ons manually didn’t help, but safe mode worked fine, so I created a new profile and installed all the usual add-ons and it still worked. Now if there only was an easy way to restore years of adblock/flashblock/noscript configuration..

        • You should be able to manually move data for addons over. To get the base folder where they store data go to Help-Troubleshooting; and click the Open profile directory button. Each addon should have its own subfolder for its data.

        • KMo, if you can bring up Firefox under the old profile, you should be Export your NoScript settings under the NoScript options page. Afterwards, you should be able to import these into your new profile from the NoScript options page.

        • Well, it was easy enough, though just copying the extensions directory and few configuration files etc didn’t work for the new profile. But reinstalling the add-ons, copying some files like prefs.js, places.sqlite, .. were enough for NoScript and Flashblock, and AdBlock+ just needed import from the old patterns.ini.

          Haven’t had much time for further testing, but it seems the memory usage went up somewhat when creating the new profile, and from 9.0b5 to 9.0b6 upgrade, but the gui now feels snappier and tab bar works correctly. :)

  5. It seems less error prone to just make SizeOfIncludingThis a global template function like:

    template
    size_t SizeOfIncludingThis(const Ty &aTy, nsMallocSizeOfFun aMallocSizeOf)
    {
    return aMallocSizeOf(&aTy, sizeof(aTy)) + aTy.SizeOfExcludingThis(aMallocSizeOf);
    }

    And just make all clients implement only the SizeOfExcludingThis member function.

  6. Does bz’s work in https://bugzilla.mozilla.org/show_bug.cgi?id=700914 reduce memory usage for just about:blank tabs, or any blank frame in your session?

    If the latter, this will be a big win when combined with the work in https://bugzilla.mozilla.org/show_bug.cgi?id=705597

    • Nicholas Nethercote

      I asked on IRC, roc said “any blank frame I think”. How could bz’s change work in tandem with bug 705597? (I read that bug but didn’t really understand it.)

      • bug 705597 is about reducing the number of blank frames that are written to / restored from session restore. In my case I had over 65,000 about:blank entries in my session restore file. So reducing those, as well as the memory used by the remainder, has resulted in a much slimmer Firefox! ~900MB resident now instead of ~2GB.

  7. I dont understand the technicalities here, but I do understand firefox is becoming more and more unusable.
    I’ve lived in fx for the last 5 years and I just wanna say: PLEASE fix it.
    honestly… its got so bad that if I didnt fear g so much I’d use chrome at this point.
    using v 8. on fedora 16, i7|16g, day job is web dev
    typically up to 10 – 20 windows open maybe 100 – 200 tabs
    addons include usual developer tools firebug firephp webdev etc & lots of refreshes
    memory on restart (open all prev tabs) can be around 400 – 500 mg.
    it’ll rise over an hour or 2 to 1 or 2gb
    I’m happy to push reports somewhere if they’ll help. let me know what and where.

    • Nicholas Nethercote

      The trend since FF4 should be that performance is getting better. There’s a good chance that you either have a corrupted profile or add-ons are causing problems. I suggest creating a new profile and re-enable your add-ons one by one. That fixes a lot of people’s problems. I realize it’s annoying, there’s work underway to add a “reset to factory defaults” feature but it’s not done yet.

      Also beware that Firebug is pretty memory hungry, even if you haven’t turned it on for any sites.

      • One thing I’ve noticed in the recent versions of Firefox is that having lots of Flash instances open seems to impact responsiveness a LOT. My plugin-container.exe was typically around 1GB and just switching between tabs could take more than a second. Scrolling pages was often laggy as well. This happens even when all the Flash plugins aren’t busy (lots of Youtube videos, for example, that aren’t playing), and aren’t on the current page.

        I know there’s a lot of stuff that goes on behind the scenes with Flash, so it makes some sense. It’s just that I think it’s getting worse – i know when i installed the 9 beta it was definitely worse than in 8. I finally ended up adding Flashblock and now things are running much smoother.

        • I notice the same thing too, although this blog properly isn;t the place to report, since they now have project snappy.

          Let’s just hope within a a few more version we should have some noticeable performance.

  8. Here heap-unclassified is now down to 17% here; it was 40% not very long ago. It is still the biggest leaf in the about:memory tree, but is now closely followed by the newly-measured about:blank styledata, at 16%, certainly caused by an excessive number of lazy-loaded tabs. Thank you for the great progress you’ve made.

    • Nicholas Nethercote

      Do you have NoScript installed? Normally JS is by far the biggest component. E.g. for my current browser session it’s 54%.

      And you’ll be pleased to know that I’ve started working on https://bugzilla.mozilla.org/show_bug.cgi?id=671299, which is the biggest remaining piece of heap-unclassified.

      • You’re right, JS is at 54% here as well, and js/system principal is 31% which probably couldn’t be reduced by noscript (but maybe safe-mode to disable extensions could make a dent). I was just focusing on big leaves, trying to see the potential impact of reductions. Delta stats in about:memory could be another way to highlight stuff.