MemShrink progress, week 6

Dave Hunt wrote this week about some endurance tests comparing Firefox 4, 5, 6, 7, and 8, which show how Firefox’s memory usage is improving.  Pay most attention to the ‘resident’ numbers for Firefox 6, 7 and 8, which indicate how much physical memory is being used.  The ‘explicit’ numbers show a big drop between 6 and 7, and then a rise between 7 and 8, but they are suspect, so don’t panic yet.

about:memory improved again this week.  I added a reporter for each compartment’s property table, and two reporters js-compartments-system and js-compartments-user which count how many compartments are present. These latter two will be added to telemetry soon.

I also added a reporter that computes the fraction of the JS heap that is unused, called js-gc-heap-unused-fraction.  The results are surprising — numbers like 30% and 50% are common.  (Some preliminary JS heap visualization helps explain why.)  There are some suggestions for short-term improvements to mitigate this (e.g. smaller GC chunks).  However, I suspect that to fix it properly will require a compacting garbage collector — the current mark and sweep algorithm unavoidably leaves lots of holes all over the heap each time it runs.  But I could be wrong!  GCs have a large design space and there may be other solutions.  A non-compacting generational GC would help in its own way too — fewer objects would reach the general heap and so each full heap collection would likely collect less garbage, and thus leave fewer holes.

Here’s this week’s bug count:

  • P1: 30 (+4)
  • P2: 48 (-1)
  • P3: 33 (+0)
  • Unprioritized: 8 (+2)

It’s nice to see the P2 count go down!  Note that bugs remain unprioritized for two reasons.  First, we don’t always get through all the unprioritized bugs in the MemShrink meeting.  Second, sometimes we ask for more investigation or data before assigning a priority.

On the topic of bugs, I closed the meta-bugs for tracking leaks against Firefox releases, and the one for tracking non-leak memory use reductions, as I mooted last week.  I kept open the one for improving memory-related tools because multiple people though it was useful.

9 Responses to MemShrink progress, week 6

  1. At least on my machine, Firefox 3.6 used to use considerably less memory after many hours of usage than Firefox 5.

    I am just curious: why is this, is there a single change in version 4/5 that is causing this? And why is it not easy to go back to the Firefox 3.6 levels of mem usage?

    – a curious user

    • Nicholas Nethercote

      Curious: there were many, many changes in FF4. And several of them worsened memory usage. Some ones off the top of my head:

      The GC changed signficantly, and the tuning of the collection heuristics wasn’t very good.
      The JS method JIT was added, it makes JS faster in many cases but the compiled code takes up space.
      Decompressed images are held on to for longer (arguably too long) in many cases.
      The new HTML5 parser had a leak.

      And there were undoubtedly more.

      The good news is that we’ve fixed most of these, and also fixed some problems that were in FF3.6. FF7 (currently in the Aurora channel) is looking particularly good, it’s memory usage is much lower than that of FF4/5/6.

      Also, the new rapid release schedule, where we put out a new Firefox every 6 weeks, should help prevent bad memory regressions, because we won’t be putting out new versions that have tons and tons of new features.

  2. Nicholas,

    Thanks for all the work you’ve been doing and kudos to the whole MemShrink team. I think this is a necessary and important undertaking.

    I don’t really know how to use Bugzilla, so I thought I’d leave a comment here for you on something I’ve noticed. Firefox seems to have a memory leak when viewing Flash content. The memory doesn’t get returned after closing the tab, nor after clicking on the GC, GC+CC, and “Minimize memory usage” buttons in the about:memory tab. Watching the resident memory when browsing normally, I can see the memory being returned. But watching Flash makes the memory usage go up and it doesn’t go down again. What seems strange to me is that the memory is going up in the main Firefox process, not the separate plugin process for the Flash stuff.

    My setup is Mac OS X 10.7 with 4 GB of RAM. I have observed this behavior in the Fx6 betas and in the Fx8 nightlies.

    Is there some way I can help? Please email me if there is.

    I apologize that this is not the best way to file a bug report.

    Best,

    Tom

    • Nicholas Nethercote

      Tom: thanks for the info. Filing a bug is the best way to report problems — comments on my blog are read by me, but Bugzilla reports are visible to lots of people. It’s not that hard once you know how :)

      Visit bugzilla.mozilla.org, sign up for an account. Then file a new bug. Choose “core” as the product and “general” as the component. Then write a title and a summary. There are tons of other fields, but don’t worry about them, you don’t need to fill them in. And everything can be changed later, so don’t worry about getting anything wrong. Also, add me (“:njn”) in the CC field.

      In the summary, please include specific steps to reproduce; the more specific, the better. For example: what Flash content? E.g. does a single youtube video demonstrate the problem? Or a single Flash game? Also, what does the contents of about:memory look like before and after you’ve hit “minimize memory usage” several times? Also, do you have any add-ons installed? Please list them if you do, you can get a list from the about:support page (under “extensions”).

      Thanks!

  3. A few remarks

    Let me bring to your attention that bug https://bugzilla.mozilla.org/show_bug.cgi?id=342810
    It took a YEAR to be solved
    It was about constant memory leak associated to a flash object being loaded.
    The solution came from an incredible knowledge of the insights of the code https://bugzilla.mozilla.org/show_bug.cgi?id=342810#c102

    But users could have been listened way better, leading to a much faster solution, see https://bugzilla.mozilla.org/show_bug.cgi?id=342810#c107 .

    Example of clue coming from the field : Seamonkey 2.2 runs Gecko 5 but *doesn’t* leak memory like FF5 does…
    Could this help ?

    Last, about fast release cycles with FF 5/6/7/… every six weeks…:
    You can have internal release cycles every six weeks
    But you can’t afford to throw bullshit to your users every six weeks
    You can’t afford to break plugin’s compatibility every six weeks

    I analyze web site stats on some major web sites in France. FF 3.6.18 is above 50% of FF users, it’s lucky for you. FF 5 is not gaining acceptance. FF 4 didn’t either.
    You’re breaking your user base into pieces, having a very different experience of the web. A really bad strategy.

    On the opposite, Google Chrome manages to have 93% of its users run the last version. And nobody knows its version number.

    Come on Mozilla, find your way between code focus/feature focus/release focus.

    People need a *working* web browser that fits into their usage habits and extension requirements, stable and speedy enough.
    Know your clients, know your market.

  4. For the bug counts, it would be useful to show
    P1 (+6/-2)
    instead of just
    P1 (+4)
    to highlight how many bugs are being fixed.

    • Nicholas Nethercote

      B.J.: You’re right. I’ve been too lazy so far to do the extra work required to get those numbers :/

  5. Thanks a lot Nicholas for your efforts. I really really grateful to you.
    I have few suggestion and request:
    1) If you don’t mind, can you post a blog about how to effectively track or take about:memory measures of leaks on site, I am talking about Bug 643651 and also of Valgrind analysis and some preference for developers helping in about:config to find some more details related to bug (if there are any such preferences). You should cover all memory hunting tools which Firefox has in arsenal.
    2) Can you take a look also on this: during I was testing Bug 643651, if I stay on about:memory page, it just take maximum 135 or 140 MB and as soon as I move to WebGL (test url), UI become unresponsive for a while and then I move to site and again switch to about:memory show me 1 GB+ RAM, otherwise I am only able to see just ~135 MB RAM, so it may mean about:memory don’t measure inactive tabs RAM properly. I maybe totally wrong but you should better double check. If this is my dumbness then forgive me.
    3) If possible for you, make a try build with DOM reporters enabled (landed some memory reporting helpers) so maybe we can find some more leaks and fixes for current.

    Maybe these comment will be useless but do consider it.

    Regards…