MemShrink’s 1st Birthday

A year ago today, the first meeting of the MemShrink project took place.  The project had been announced almost three months earlier, but until that first meeting it was basically just me working on it and there wasn’t much momentum.  So I now think of those three months as something of a gestation period, and the first meeting as the true birth of the project.

So, happy birthday, MemShrink!  I’ll take this opportunity to go over some of the accomplishments we’ve made in the past year.

Big Win #1: Better Javascript Heap Management

There have been two huge MemShrink improvements that stand out in my mind.  The first one happened very early on, and it consisted of two related changes, both implemented by Gregor Wagner.

In Firefox 4, the implementation of the JavaScript heap was totally overhauled.  In particular, it was segregated into compartments, each of which held (roughly speaking) the memory for a single domain.  System compartments, i.e. those for JavaScript code that implements parts of Firefox (such as the UI), tend to be long-lived.  In contrast, user compartments, i.e. those for JavaScript code in web pages, tend to be short-lived.  But memory from both system and user compartments was being co-located in 1MB chunks.  What tended to happen is that if you browsed for a long time and then closed many tabs, you’d end up with lots of 1MB chunks that were kept alive by a small amount of memory from system compartments.  This is a form of fragmentation, and it prevented Firefox from releasing lots of memory back to the operating system that it should have been able to.

Fortunately, Gregor came up with a simple fix:  don’t allow system and user compartments to share the same 1MB chunks.  To see just how effective this was, consider this visualization of the JavaScript heap after closing many tabs, without Gregor’s change.

badly fragmented JavaScript heap

Each horizontal line is a 1MB chunk, and each square is a 4KB arena.  White squares are unused arenas, and coloured squares are in use.  Arenas belong to a particular compartment all have the same colour;  there are three compartments alive, all of which are system compartments.  You can see that many 1MB chunks are kept alive by a small number of arenas.

After Gregor’s change, in this scenario the heap ended up looking like this.

non-fragmented JavaScript heap

Much better.

Gregor’s second change related to garbage collection scheduling.  In Firefox 4 the garbage collection heuristics were poorly tuned, and the garbage collector simply didn’t run frequently enough.  Many people who left Firefox open for a while came back to find that JavaScript memory consumption had ballooned, and the machine was unusable due to paging, for seconds or minutes, until the garbage collector kicked in and paging finished.  The fix was simple:  run the garbage collector occasionally, using a timer.

These two fixes of Gregor were the main reason why the first item in the Firefox 7 release notes was “drastically improved memory handling for certain use cases”.  More specifically, Firefox 7 used less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.

Big Win #2: Fewer add-on Leaks

The second huge MemShrink improvement is much more recent.  In fact it hasn’t even made it into a released version of Firefox yet.  And the path towards it was much more circuitous.

I first revamped about:memory in May of last year, and those changes made it into Firefox 6. This started giving us insight into Firefox’s memory consumption, but it wasn’t until I broke up the reporting of JS memory consumption on a per-compartment basis that it really started to make a difference.  (This change made it into Firefox 7, another reason why that release was so good from a memory consumption point of view.)

Per-compartment reporters made some blatant inefficiencies obvious, which were quickly fixed.  More importantly, people quickly noticed that sometimes compartments were present for sites that had been closed long ago.  This is a kind of memory leak that became known as a zombie compartment, and hunting them became a popular enough sport that  more than 100 zombie compartment bug reports have been filed.

Some of the zombie compartments were due to defects in Firefox itself, and these were generally fixed fairly quickly.  However, it soon became clear that the majority of them are due to add-ons.  It’s quite easy to unintentionally create zombie compartments in add-ons.  In the worst case, add-ons could leak the compartment of every single site visited.

This led to some lively discussion about how best to handle these leaks, because they are defects in third-party code that is largely out of Mozilla’s control, and yet they make Firefox look bad.  Fortunately, this discussion became mostly moot when Kyle Huey took advantage of an 8,000 mile trip from home to implement a crazy idea he’d had six months earlier —  when a tab is closed, simply cut the references from other compartments that would keep the compartment of the tab’s page alive.  This turns out to work tremendously well and prevents most of the cases where add-ons can create zombie compartments.  This change is scheduled to ship in Firefox 15, which is due for release in late August, and should more or less fix what I six months ago ranked as the single worst problem affecting Firefox’s memory consumption.

The Key Tool: Memory Reporters and About:memory

I mentioned about:memory above.  It’s worth discussing in more detail, because it’s the single most important tool we’ve created, and has driven a lot of the MemShrink improvements.

Let’s look at about:memory as it was in Firefox 6, which was released about 10 months ago.about:memory screenshot from Firefox 6

Even these early versions had several key properties.

  • about:memory doesn’t need a special build to run;  it works even in released versions.
  • It is trivially easy to use — you just type “about:memory” in the address bar.
  • Because it’s all text you can cut and paste the output, and unlike many web pages it is carefully constructed so that it reproduces beautifully when you do so.  No screenshots are necessary.

These properties massively expand the number of people who are able to detect and report problems relating to memory consumption.  Any Firefox user can potentially do it;  if the user doesn’t already know how, they can be taught in 10 seconds.

Having said that, this old version is almost laughably primitive compared to what we have now.  First, let’s consider the state of the memory reporting infrastructure, which about:memory completely relies on, as it was at that time.

  • There were a fixed number of measurements, which meant there were no individual measurements for variable-numbered entities such as compartments, window objects, or SQLite connections.  Such measurements were only possible once I added support for memory multi-reporters.  We now have dozens of measurements per variable-numbered entity, and thousands of measurements in total.
  • Even once per-compartment reporters were added, we still didn’t have much insight into the memory consumption of the JavaScript code used by Firefox itself (and its add-ons).  But compartment-per-global landed recently and gave us much finer-grained measurements, which (a) gives us some idea of how much memory is being used by JavaScript-based add-ons, and (b) gets us very close to being able to report memory consumption for each tab.  Users have been requesting both of these things for years.
  • Early memory reporters had lots of defects that caused them to report incorrect numbers, but these were obvious only when the numbers were ridiculously large or negative (both of which happened on occasion).  The first of two major improvements in correctness came about as we learned how to structure memory reporters to avoid common defects.  In particular, it’s best to write reporters that are traversal-based rather than counter-based, and to measure heap blocks with malloc_usable_size rather than computing the sizes.  Reporters that use malloc_usable_size also measure slop bytes, which are caused by the heap allocator rounding up allocation requests and can account for 5–10% of the heap.  Early reporters didn’t have these characteristics, but most of them have been converted now.
  • Reporters with these characteristics are doubly-virtuous, because they also integrate well with DMD, a tool I wrote that dynamically checks memory reports.  DMD provided the second major improvement in memory reporter correctness, because it identifies heap blocks that are counted twice, which is a mistake that’s easy to make.
  • DMD also identifies unreported heap blocks, and was crucial in getting the “heap-unclassified” number down.  In the example above it was 68.82%(!), whereas now it’s typically around 15%, which is low enough that people have mostly stopped complaining about it.  Prior to DMD, identifying this “dark matter” was much harder and less systematic.
  • The only tree shown in the above example is “explicit”.  On Linux, Justin Lebar added code to summarize the information in the /proc/<pid>/smaps file in “size”, “rss”, “pss” and “swap” trees.
  • The reporters back then could only report byte measurements.  They can now report unitless counts and percentages, both of which are useful.

Now consider the about:memory page itself, as it was back then.

  • Insignificant measurements were simply omitted, and you had to reload in verbose mode to see them (and the measurements would be re-taken when that happened, potentially losing any interesting measurements).  Now it is possible to expand and collapse arbitrary parts of the trees on the fly.
  • The “Other Measurements” section contained a flat list of measurements.  Trees are now possible, which makes that section more flexible and easier to understand.
  • about:memory’s has been carefully streamlined to consume as little memory as possible, so that it doesn’t perturb what it’s measuring.  This is really important now that we report so many measurements.
  • about:memory now has a companion page, about:compartments, that just lists compartments (and ghost windows).  about:compartments shares a lot of code with about:memory, and it is better than about:memory for certain tasks such as spotting zombie compartments.

All this stuff is important because you make what you measure.  Just about every time we’ve started measuring some new thing, we’ve found and fixed problems as a result.

Smaller Things

The following smaller accomplishments were also made in the past year.

  • John Schoenick used the MozMill endurance tests to build areweslimyet.com, which helps identify regressions in memory consumption.
  • I removed a lot of unnecessary memory waste caused by “clownshoes” allocations.
  • Olli Pettay wrote about:cc and Jan Honza Odvarko wrote about:ccdump.  These closely related tools have been used to find various cycle collector leaks.
  • I wrote 41 progress reports.  I had/have two goals with these:  to keep interested people informed of what it happening, and to provide evidence that Mozilla actually does care about memory consumption in Firefox.

This is only the tip of the iceberg, though.  As I am writing this, there have been 314 MemShrink-tagged bugs marked FIXED in Bugzilla.  In comparison, there are only 221 MemShrink-tagged bugs currently open.  Breaking them down by priority we get the following.

  • P1: 52 fixed, 25 open
  • P2: 135 fixed, 88 open
  • P3: 53 fixed, 106 open
  • Unprioritized: 74 fixed, 2 open

Over 75 people were responsible for these fixes.  I give many thanks to all those people, plus everybody who contributed in any other way to fixes, such as bug reporters and patch reviewers.  And I’d like to give extra thanks to the MemShrink regulars:  Justin Lebar, Kyle Huey, Andrew McCreight, Johnny Stenback, and Jet Villegas.

What’s Next?

There’s no real secret to MemShrink.  So far it’s basically been a long, steady grind, gradually improving tools, fixing leaks, slimming down data structures, and responding to user’s problems.  There are no plans to change that, and we’ll continue our recently-lowered pace of fortnightly meetings for the foreseeable future.  And if we’re lucky, I might be writing a report on MemShrink’s second birthday a year from now!

41 Responses to MemShrink’s 1st Birthday

  1. Happy birthday!
    I’m a big user of Google Reader and I noticed that, on Firefox 11 for Windows, the memory use increase continuously and never decrease, even if I closed the tab.
    With Firefox 12, that problem disappeared but it’s back with Firefox 13, and only with Google Reader. Other Google applications behave normally.
    The only way to get the memory back is to restart Firefox.
    Hope you can find a solution.
    Thanks for the good job.

    • Nicholas Nethercote

      It’s a known problem: https://bugzilla.mozilla.org/show_bug.cgi?id=750048. Though I’m not sure why you’d see it go away with FF12.

      In FF13 what does about:compartments say? Does it have lots of google-related compartments.

      • Well, GR leaks windows, but the memory should still go away if you close the tab.

        Eric, perhaps you’ve combined GR with a leaky add-on. You could try running Firefox in safe mode.

        If in safe mode the memory really doesn’t go away a few minutes after you close the tab, it would be helpful if you could file a bug report.

      • I think too that the FF12 better for Google Reader. There isn’t so big memory leaking.

  2. The sophistication of the fixes has been increasing steadily over the year too. Hopefully we’ll see even more sophistication over year 2.

  3. As a user (and person who builds sites), thanks. It’s been nice especially seeing Aurora making progress on the memory use from over the last year-plus. I recall earlier times where I had to restart the browser a few times daily, given the frequency with which I was reloading JS-heavy pages during development.

  4. First, thank you yet again both for your work on MemShrink itself and for these updates—both interesting technically and (as you point out) as evidence that these problems are acknowledged and addressed.

    Now, a question: When I look at areweslimyet.com, I can see a huge improvement for upcoming releases (the plugin leak fix?), but also what looks like a general trend of regressions toward the end of the graphs. What’s…well, what’s going on there?

    • Nicholas Nethercote

      Compartment-per-global accounts for most or all of it. I wrote about it here: http://blog.mozilla.org/nnethercote/2012/05/16/memshrink-progress-week-47-48/

      • I think those mayor changes, the ones that cause (or caused) significant changes in memory usage (like the above mentioned Compartment-per-global) should be indicated on the stat-boards of areweslimyet.com. When average users visit the site, all they see is that FF is getting “memory hog” again with FF15, but they do not know the reason behind this change. And they will certainly won’t look after it, if there is no explanation for the changes on the same exact site. At “better cases”, they will simply ignore it, at worse, they will switch to another browser without further thinking.

  5. Stephen Chung

    Chrome used to be my primary browser. Ever since MemShrink started, I’ve been using FireFox more and more. Then it became my primary browser. About six months ago I uninstalled Chrome. And that’s 90% because of the performance gains that MemShrink brought. Happy Birthday!

  6. I (power user and occasional developer) was close to switching to Chrome for performance reasons a year ago. Then the MemShrink project came along, showing to me Mozilla’s commitment to improve the situation. A year later I am on Firefox 13 beta, have no performance complains whatsoever compared to Chrome anymore. On top of that I enjoy the great add-ons provided for Firefox (most importantly, a proper ad blocker).

    Keep up the great work! Kudos! Big thanks!

  7. Thanks for your hard working!

  8. Happy Birthday … and MANY happy returns. Alternatively as Microsoft apparently did with security, are there procedures in place whereby all developers will have tools for, and a mentality of, preventing memory handling problems going forward? Is Mozilla an org that can integrate critical values such as coding with memory management in mind at every step? Or will Mozilla be best served by perhaps always having an over-arching MemShrink-style effort to keep monitoring and/or mopping up the memory handling impact of the broader contributors over time? Either way, hopefully MemShrink’s success thus far will not be wasted in the long term by either regressions or a ‘strategic’ reduction in focus on memory handling.

    Of course there is areweslimyet but even that term implies there will be a time where Mozilla’s memory usage is acceptable. What then?

    BTW it’s very hard to accept your previous criticism of my ‘tone’ when you keep writing articles that touch on similar points, just less ‘colourfully’ :) Pointing out how conceptually simple the two biggest MemShrink wins have been (all be it with hindsight as our best friend) really raises the question of why these issues were not discovered and the fixes implemented long ago. Additionally, and further to my question about Mozilla’s culture and how to best ensure MemShrink’s influence is perpetual, the question of how Netscape or Firefox’s architects allowed the browser to get so bloated in such fundamental ways, really needs to be asked. Cliche time:

    If we don’t learn from out mistakes were are doomed to repeat them!

  9. Jon-Carlos Rivera

    I really appreciate the hard work that the MemShrink team has been putting into SpiderMonkey but in my particular use-case (a raytracer) I don’t see much effect. I don’t know if I have a bug in my code but none of the other major browsers have this problem to the degree that Firefox (Aurora) displays:

    http://i.imgur.com/wvqDA.jpg

    If I try to render at 800×800, the browser will get about halfway through the image and then crash (at the 2GB memory wall). Even IE can complete this simple rendering given enough time.

    The major deallocations at the end of each graph represents a shift-reload of the tab followed by a closing of the tab and loading the same page into a newly created tab. All the browsers except for Firefox and Safari purge the old data immediately at the shift-reload.

    In Firefox, if I were to start a a new rendering in that fresh tab, it would reach the memory limit and crash the browser quickly even though the old tab is no longer around.

  10. I remember back when Mozilla was like “It doesn’t leak at all” and the rest of the world was like “Could Not Reproduce”… I’m glad memshrink has been so successful, because even when I wasn’t using FireFox (did 3.6-7.0 actually exist?) I was never happy with the memory usage or UI of any of the other browsers.

    • I stayed on 3.5 for ages because 3.6 was too dodgy for me–it consumed a lot more memory and its idle CPU usage was way too high. (It also took literally 20 minutes to start up.) I knew from too many reports that 4.0’s memory use would kill my system. Once Firefox 7 was out, I jumped to that and then, disliking 7, went immediately to the 8 beta. I was on the beta channel until Firefox 14 just debuted in beta recently, at which point I had to jump ship because I was seeing 3.6-ish behavior on memory and CPU usage, which I hope is a fluke that will die with that version (or better yet, with the beta).

  11. Would it help to have a button at the top of about:memory that says “Copy to Clipboard”?

  12. The problem seems to be linked with Google +1 button. In about:compartments, there were many lines like this one :
    https://plusone.google.com/_/+1/fastbutton?…
    I use Noscript and, if I forbid gstatics.com scripts in tne GR tab, the problem disappears.
    Thanks for your help.

  13. I got a question not related to this post. When I browse YouTube, I notice that the memory usage of the Firefox process is almost double that of plugin-container process. Of course, since flash is supposedly contained in the plugin-container I expect that Firefox memory usage to not be affected, and all the high memory usage be in the container process.

    Is my expectation false?! What gives?!

    • Nicholas Nethercote

      YouTube pages have plenty of stuff besides the video — comments, ads, other suggested videos, etc. about:memory might give you an idea of how the memory is being used by YouTube.

  14. I hope this steady improvement continues in future versions. I was looking forward to Firefox 15 greatly because of the add-on leak fix, but I have to be honest: 14 is a mess, and it scares me to think that its problems could carry over to 15 even when 15 has so many major promised improvements.

    Firefox 14 used ever-increasing amounts of memory for me and also kept the CPU around 30% when idling. Because it was so badly broken, I had to backtrack to 13 and get off the beta channel. Sadly the 13-to-14 update also trashed my history frequency, which messed up my location bar. I’m hoping when 15 is out in beta I can jump right to it, and that the memory climbing and crazy CPU usage will disappear. I suspect the issue I had was with incremental garbage collection, but according to my about:config that was supposedly disabled if I was reading it right, so I don’t know.

    • For me Firfox 14 is much betetr compared to all other former versions. It starts faster, responds better and works absolutely fine. You should try the Reset feature to rebuild your profile.

    • Nicholas Nethercote

      Can you give more details? Any particular site that causes the problems? Do you have add-ons installed?

      Vague comments along the line of “version X is awful, I’m downgrading” are amazingly frustrating to developers, because they give us no chance to help you.

      • I understand the vagueness doesn’t really help, and I apologize that I can’t really be much clearer on it. I do get enough of that in my own job so I know it’s frustrating to hear “the new version is broken”.

        I do have a number of add-ons; it’d be difficult to list them all here. Firebug is chief among them and is one of the big reasons I’m looking forward to 15. The only new add-on I installed after 14 was “Site identity button colors”, which partially reverse the dreadful new location bar styling in 14. (It does not, sadly, restore favicons there.) I doubt that was it. It is however one of the few new-style add-ons I have installed.

        As far as I know, no one site is causing problems–but there’s no way I know of to track which module or compartment is using the CPU. Maybe about:memory might have been helpful I guess, but at the time I was dealing with it I wasn’t really in a position to spend long diving into the details. What I can tell you is that with my usual bevy of windows open (I have quite a few, many of which have multiple tabs though the unused tabs aren’t loaded till they’re needed), even in a freshly restored session I had idle CPU usage of around 30%–this is on a single-core 3.4 GHz machine–and memory climbed steadily. Again this was just while idling. Since I couldn’t find a changelog, I couldn’t narrow down what had been added to 14 to account for this; some sites I found said incremental GC was turned on in this version and that’s a likely culprit in my book.

        I guess mostly I’m hoping others will come forward with clearer answers as to what’s going amiss, as I have to think now that 14’s in beta those issues will become apparent to more people. Having my CPU constantly churning was untenable though, as was the memory growth (while idling mind you). I don’t really dare return to the 14 beta to run tests on the problem as I rely on the browser too much for work; my hope is rather that whatever’s wrong with 14 is already fixed in 15, and once that hits beta I can hopefully jump back on the beta channel to get the many new benefits in that build. As for the UI snafu, I’ll have to wait for more add-ons to fix that I suppose; an about:config setting to get the favicons back would be better, as would something to restore the old secure site colors.

        • If you use Windows, run an xperf trace to (stackwalking for Profile flag) and look which functions cause the CPU usage. Maybe this helps to see the cause.

          1 note, stackwalking only works with Vista or later, not XP.

  15. I would like Mozilla to aim beyond getting firefox ‘slim’ and put signifcant effort into gettting it as spidery skinny as technicaly possible -or a special build with that goal. Why? well because it would be of great benefit to users of netbook class devices and older computers. It would be of huge use to people in the second and third worlds who are not on the same upgrade cycle as the ‘memory is cheap’ first world. Great for the environment too -to extend the useful life these tools. Feeling slim? Good for you, but no reason to ever get too comfortable.

  16. Ah but i should not have forgotten to say thankyou for the very useful and impressive work by the memshrink team already. Do keep at it please.

  17. Are there any chance that Thunderbird could get some Memshrink-love at some point? I guess, some of the fixes have gone into common code, but there might be some low hanging fruit. It seems that TB is getting larger while FF is getting smaller.

  18. FF12, lion, about 7 tabs, 600Mb. It is normal?
    memory log
    http://pastebin.com/6nmdVx2X

    • Explicit (349M) is much higher than resident (556M). This isn’t normal, even with heavy fragmentation. To troubleshoot this, you should upgrade to the current release (FF13), restart in safe mode, and if there is a difference, bisect enabled addons to see if you can pinpoint one that causes it.

    • Nicholas Nethercote

      First, I need to point you at this: http://blog.mozilla.org/nnethercote/2011/08/29/browser-x-is-using-y-mb-of-memory-with-z-tabs-open-is-a-meaningless-observation/

      But you included about:memory, which is good. As Tobu meant to say, the only odd-looking thing there is that resident is much higher than explicit. Normally I’d expect them to be closer. But then, I just checked about:memory on FF14 (beta) on my 10.7 Mac and resident is 762MB and explicit is 502MB, a ratio of 1.52 compared to your 1.59, i.e. not terribly different.

      FF15 features compartment-per-global which gives a lot more insight into the system compartments, including some of the memory used by add-ons. You could try it if you wanted more information, it’s currently available at aurora.mozilla.org.

      • Here (amd64 nightlies on Ubuntu) resident = explicit + 5% overhead, I had no idea OS X could be that much worse. Will jemalloc fix that, or is there something in the OS memory management which breaks down against anything that isn’t Obj-C style memory pools?

        • Nicholas Nethercote

          Firefox uses jemalloc on Mac now. So I would guess it is a code size issue — maybe the libraries used on OS X are a lot bigger?

  19. sorry men, I said FF12, but I meant FF 13.

    So in the next days i will try to run firefox in safe mode, and see if there are differences, if so, i will enable 1 extension at time, to understand what cause the issue. I will let you know in few days.

  20. First of all thanks for the link, but as you stated, I posted a detailed memory log, that contain also the links to the page opened, so I don’t know if I can give you more log about my situation, (maybe number of plugin, and how many time FF is opened).

    However a little recap of my tries:
    -Starting Firefox in safe mode, seem to “solve” the problem, the memory is about 200-400Mb during the day.
    -Starting Firefox normally with all extension disabled (I use about 8 extension**), don’t solve entirely the problem. Right now, Zero extensions, about 5 tabs, 500Mb, log here http://pastebin.com/MBau6rSQ
    however seem a little better.
    -The memory “peak” (600-700Mb) is always reached in minutes from when i run firefox, and remain stable along the day.

    **extension previously enabled: adblock, session manager, gmail manager, imtranslator, live http headers, xmarks, download statusbar)

    Please let me know if you want do further tests to help to catch “the problem” (if there is one), in any mode you prefer (mail, skype, teamviewer, etc…).

  21. Just want to say congrats. I switched from Chrome to Firefox on OSX for memory usage reasons and haven’t looked back (except occasionally for Chrome’s page translation feature) – Firefox is way outperforming Chrome on this front nowadays.

    Now if we could just see Apple + Microsoft + Google having some similar devotion to memshrinking in their products (OSX, iOS, Android, Chrome, Windows, etc) :)