MemShrink progress report, week 24

Something that happened a while ago but I failed to notice was that Chris Leary added a regexp cache which, among other things, is flushed on every GC.  This fixed an existing problem where cold regexps were not being discarded, which was a MemShrink:P1 bug because 10s or even 100s of MBs of compiled regexp code could accumulate in some circumstances.  Nice work, Chris!

Andrew McCreight fixed two leaks (here and here) involving WeakMaps.  WeakMaps are an EcmaScript 5 feature and so are not used that much at the moment, but their popularity will increase over time.

I landed some memory reporter infrastructure changes.  These will make it much easier to integrate DMD with Firefox, which will help drive about:memory’s “heap-unclassified” number down.  They also do some sanity checking of memory reporters, and this checking has already found some bugs in existing reporters.

Terrence Cole made the JS engine to a “shrink” GC on memory pressure events, such as when about:memory’s “minimize memory usage” button is pressed.  A “shrink” GC is one that causes unused pages to be decommitted.

I wrote two pieces of documentation.

  • The first is a guide to zombie compartments, including instructions on how to test if an add-on causes them.  This guide is similar in spirit to an old blog post of mine, but explains things more carefully.  There is a QA test day planned for Friday, December 16, and hopefully some extensive add-on leak testing will happen on that day.
  • The second is a guide to implementing memory reporters.  Please read it if you ever have to implement one.

In publicity news, ZDNet published a story about how Google’s +1 buttons consume a lot of memory, particular in the newly redesigned Google Reader.  The author used Firefox’s about:memory page to determine this, which enabled him to point the finger at Google’s JS code instead of Firefox.  (Dietrich Ayala wrote about this topic previously, and wrote the Wallflower add-on in response;  the Antisocial subscription for AdBlock Plus apparently has the same effect.)

Here’s the current bug count.

  • P1: 27 (-5/+1)
  • P2: 139 (-4/+11)
  • P3: 61 (-0/+1)
  • Unprioritized: 0 (-4/+0)

The P1s went down because in today’s MemShrink meeting we reprioritized several that are now less important than they were.

28 Responses to MemShrink progress report, week 24

  1. Great! I have been looking for WallFlower for a long time ever since i have been noticing stupid Facebook and Google +1 is taking over my memory reporter page. I am assuming since WallFlower is lighter in functions, it should also be lighter in resources compare to Ad Block?

    • I guess i was too excited before i tested it

      ├────4.14 MB (01.30%) — compartment(https://plusone.google.com/_/+1/fastbutton?url=
      │ │ ├──2.38 MB (00.75%) — gc-heap
      │ │ │ └──2.38 MB (00.75%) — (8 omitted)
      │ │ └──1.76 MB (00.55%) — (9 omitted)
      │ ├────3.27 MB (01.03%) — compartment(http://www.facebook.com
      │ │ ├──1.75 MB (00.55%) — gc-heap
      │ │ │ └──1.75 MB (00.55%) — (8 omitted)

      • The ZDNet article is correct. Go to a feed with very short articles in Google Reader (such as BBC News), and keep scrolling down, one page at a time, causing the +1 button to load on all articles (apparently they need to be displayed for the +1 button to show).

        Memory for the plusone.google… compartment will keep going up. Didn’t take very long to get to 100 MB.

  2. First, thanks for all you do. You rock.
    Your updates help community members to better be involved & excited about Firefox.

    Also, I meant to comment on a previous post but didn’t. I think your efforts at spreading yourself some and getting behind several memory bugs and pushing were valuable and made a significant difference.
    I do hope you haven’t completely ceased in this rallying role.

    Perhaps this is a silly question, but do we essentially press the minimize memory usage button when all windows are minimized? (and also I wonder if there are any other guaranteed zero interaction moments we can capitalize on?)

    • Nicholas Nethercote

      “Minimize memory usage” triggers a “memory pressure” event. I believe that event is also triggered on Firefox Mobile when it is backgrounded. I don’t know of any other cases when it is triggered. There is something called the “idle service” that can detect when the browser is unused, though I don’t know much about it, but maybe we should trigger a memory pressure event then too.

      • Yes, if the memory pressure event is not being fired when the browser detects idleness or when all windows are minimized, I really think it should be.
        I say “when all windows are minimized” because that’s a time of guaranteed non-interaction, a time when we need to worry less about running UI-pausing events.
        This seems like memshrink territory to me.
        I’d go pursue this myself, as it could be a cheap win, but I’m not really a developer/programmer.

  3. Reprioritized or deprioritized? Hmm I can smell something brewing and it’s not yummy coffee.

    • Probably partial implementation, and/or some of the places where the buggy code was being called were modified to no longer do so.

    • Nicholas Nethercote

      You caught me! I’m just juking the stats just to look good. Oh well, it was worth a try, I guess I’ll be fired in disgrace now.

  4. The GC Shrink change sounds very handy however when and how often is it called automatically rather than user-initiated from about:memory?

  5. Nicholas,

    With just under 3 weeks left in the Nightly development period for Firefox 11, are there going to be lots of big memshrink bugs landing in that time and therefore making it into the release version of Firefox 11 on 13th March 2012, or will we have to wait until Firefox 12?

    How do you envisage the memshrink improvements going into Firefox from now on? Will there be another big release like Firefox 7 where there are obvious improvements or are they going to be more incremental in nature from now on?

    I’m on the beta channel and 9 seems a lot better than 7 or 8. Just wondered if that kind of improvement will keep on coming. :-)

    • Nicholas Nethercote

      It’ll be incremental. Bugs will land when they’re ready, that’s the point of the rapid release cycle. The fact that FF7 had a lot of memory use improvements wasn’t planned, it just happened. Consequently it became “the memory usage release”, but that was a label applied after development had finished.

      In what ways does 9 seem better? The memory usage in particular, or just general responsiveness?

      • Actually I guess we will see another couple big jumps when incremental and generational GC lands, right?
        What about object shrink, will that be a noticeable improvement? Does it affect only memory or we can expect increased JS performance due to more code fitting in the CPU cache?

        • Nicholas Nethercote

          Incremental GC probably won’t affect memory usage much. It’s more about responsiveness, avoiding long pauses by running each GC in lots of small pieces instead instead of one big piece. But you’re right that generational GC should make a big improvement when it happens.

          Object shrink will be a good improvement. I’m not sure if you’ll notice it if you don’t look at memory measurements, though.

          As for performance, reducing memory always potentially helps, but it’s really hard to quantify, esp. as it depends greatly on your machine’s hardware, e.g. how much RAM you have.

      • Thanks for that Nicholas. FF9 is better in two ways that are apparent to me. First off everything seems to be happening that little bit quicker than on FF7 or FF8. Secondly it’s using less memory than previous versions.

        I have 9.0 b3 installed at the moment and it runs like a dream. Bear in mind however that the only add-ons I have installed are a British English dictionary for spell checking when typing on blogs etc like this one, and Test Pilot to help feed information back to Mozilla. I count myself as someone who doesn’t use add-ons, because the two I have installed make a negligible amount of different to the performance of the browser.

        Now when I have 7 or 8 tabs open memory seems to remain at 200,000 kb – 230,000 kb. Before it was 280,000 on just 5 tabs. I always browse the same 12 or so websites with consistency, so the variables are the same over time.

        I also make a note of roughly what the memory consumption is and then go and cook and eat for about an hour, come back and it’s the same if not less than it was. on FF 6 it would be twice what it was. It was dreadful. But since FF7 it’s been getting better and better.

        MY experience is really very good and I have nothing to complain about although that may have something to do with the way I maintain my laptop, the fact that I have FF set to NOT remember my browsing history, use no add-ons worth talking about and never have more than about 9 tabs open. Right now I have 8 tabs, that’s pretty much what I stay at.

        Also, the little circular activity indicators on each tab seem quicker.

        I used to click on a link and the indicator would do it’s black anti-clockwise circles, sometimes doing 5 or 6 rotations before the green clockwise indicator would come in and the page would load. Now, generally speaking, the activity indicator goes black for a split second, doesn’t even get to do quarter of a rotation and then the green indicator kicks in for just a second or two and the page has fully loaded. I know this sounds like an incredibly rudimentary way of measuring speed, but it works because I am comparing the same browsing habits on exactly the same laptop. I also haven’t had what you could call a crash since switching to the beta channel in mid September.

        I am always wary whenever I hear horror stories of someone having a really bad time with memory performance or speed with Firefox. Either they’re trolling or they have a whole bunch of add-ons slowing everything down. Either that or their PC or laptop is slow anyway.

        I am sticking with the beta channel as my default browser for the foreseeable future. It’s great. There is very little loss of stability or performance compared with the release version, you only have to contend with one upgrade per week instead of one a day, and you’re still 6 weeks ahead of most of the pack in terms of new features etc.

        • Nicholas Nethercote

          Thanks for the detailed info. I’m glad FF9 is working well. I don’t know why it’s such an improvement over FF8 for you, and these kinds of things tend to vary greatly with workload. FF9 has JS type inference, which speeds up JS code often and might reduce its memory usage, so maybe that’s a factor. I don’t know what would be speeding up the page loads.

          FF4,5,6 were really bad for the “memory increases while doing nothing” due to some awful GC heuristics. FF7 fixed that.

          W.r.t. performance horror stories, any time you read a discussion thread about browsers, you always have some people saying “OMG Firefox is dog slow for me and Chrome runs like a dream” and then some other people saying the exact opposite. It’s weird, I’d love to know why. (Well, for Firefox add-ons certainly can cause perf problems, and I suspect old, busted profiles can cause problems too.)

  6. I’m really hoping the improvements in FF10 get here soon, because the 9 beta is leakariffic. It seems to have occurred more as of recent versions and I’m not entirely sure it’s not Firebug’s fault, but doing web development I’ve been running into some huge memory-chewing situations.

    • I think that this is Firebug’s fault. I’ve just tried to disable it and the results are that under similar workload, I’m somewhere under 500 MB, whereas with Firebug enabled I was well over 700 MB after some time.

    • And that is when I was actually not using Firebug at all. It was only enabled in the background, everything turned off.

  7. Hello.
    Another good article about MemShrink evolution, thanks. Is it possible to update this page, please?
    https://wiki.mozilla.org/Performance/MemShrink#Minutes_and_Progress_Reports

  8. Hi Nicholas,

    Another question just for my own curiosity.

    How quickly does a Memshrink bug fix get into Firefox?

    For example, if on the week 25 update from you on this blog we read something along the lines of:

    “Really good week last week, Person Y landed fixed Bug X solving problem Z.”

    Does that mean that the big fix in question has already landed in whatever version of Nightly was being worked on at the time or is it a case of it going into Nightly within the next few days? Few weeks? How long?

    • Nicholas Nethercote

      12–18 weeks. I usually only mention bugs that have landed on the mozilla-inbound repository. Those changes then get merged to mozilla-central within 24 hours. If this is right at the start of a 6-week dev cycle, then it’ll be on Nightly for 6 weeks, Aurora for 6 weeks, Beta for 6 weeks. If it lands right at the end of the dev cycle, it’ll be on Nightly for almost no time, Aurora for 6 weeks, Beta for 6 weeks. All this assumes that the change isn’t subsequently backed out, which can happen if it causes problems.

      The rapid release calendar is here: https://wiki.mozilla.org/RapidRelease/Calendar. We’re currently a bit more than halfway through the FF11 dev cycle, so any change that landed today would take about 14 weeks to make it into a release. Each bug’s “target milestone” field also gets filled in when it’s merged to mozilla-central.

  9. “WeakMaps are an EcmaScript 5 feature”
    => They are an ECMAScript 6 feature :-)

    Thanks for all the work on memory and the weekly report!! :-)

  10. Hi,

    I try to bump this bug I’m experiencing on a real application in the real world ;) https://bugzilla.mozilla.org/show_bug.cgi?id=395260

    Just updated it with a working testcase. I think it should at least have the “footprint” keyword.

    Thanks !