MemShrink progress, week 9

Firefox 8 graduated to the Aurora channel this week, and the development period for what will become Firefox 9 began.  Lots of MemShrink activity happened this week, and I think all the changes listed below will make it into Firefox 8.

Avoiding Wasted Memory

I have blogged previously about memory wasted by “clownshoes” bugs.   Ed Morley found a webpage that resulted in 700MB of memory being wasted by the PLArena clownshoes bug.  Basically, on platforms where jemalloc is used (Windows, Linux), half the memory allocated by nsPresArena (which is built on top of PLArena) was wasted.  (On Mac the waste was 11%, because the Mac allocator rounds up less aggressively than jemalloc).

Fixing this problem properly for all PLArenas takes time because it requires changes to NSPR, so I made a spot-fix for the nsPresArena case.  This is a particularly big win on very large pages, but it saves around 3MB even on Gmail. This spot-fix has been granted beta approval and so will, barring disaster, make it into Firefox 7.

A Firefox Nightly user did some measurements with different browsers on the problematic page:

  • Firefox 8.0a1 before patch: 2.0 GB
  • Firefox 8.0a1 after patch: 1.3 GB
  • Latest Chrome canary build and dev (15.0.849.0): 1.1GB
  • Webkit2Process of Safari 5.1: 1.05 GB
  • Internet Explorer 9.0.2: 838 MB
  • Latest Opera Next 12.00: 727 MB

So this fix gets Firefox within spitting distance of other browsers, which is good!

In other developments related to avoiding wasted memory:

  • Luke Wagner discovered that, on typical websites, most JSScripts are byte-compiled but never run.  A JSScript roughly corresponds to a JavaScript function.  In hindsight, it’s not such a surprising result — Firefox byte-compiles all loaded JavaScript code, and you can imagine lots of websites use libraries like jQuery but only use a small fraction of the functions in the library.  Making byte-compilation lazy could potentially save MBs of memory per compartment.  But that will require some non-trivial reworking of the JS engine, and so is unlikely to happen in the short-term.
  • Kyle Huey avoided a small amount (~100KB per browser process) of waste due to rounding up in XPT arenas.

Improving about:memory

I made some progress on a Valgrind tool to help identify the memory that is currently reported only by the “heap-unclassified” bucket in about:memory.  It’s called “DMD”, short for “Dark Matter Detector”.  It’s in early stages and I still need to teach it about most of Firefox’s memory reporters, but it’s already spitting out useful data, which led to me and Ehsan Akhgari landing memory reporters for the JS atom table and the Hunspell spell checker.  We also now have some insight (here and here) about memory usage for very large pages.

Mounir Lamouri turned on the memory reporter for the DOM that he’s been working on for some time.  This shows up under “dom” in about:memory.  There are still some cases that require handling;  you can follow the progress of these here.

Andrew McCreight replaced about:memory’s buttons so you can force a cycle collection without also forcing a garbage collection, which may be useful in hunting down certain problems.

Finally, Sander van Veen added the existing “js-compartments-user” and “js-compartments-system” to the statistics collected by telemetry (his first landed patch!), and I did likewise for the “storage/sqlite” reporter.  I also added a new “tjit-data/trace-monitor” memory reporter that accounts for some of the memory used by TraceMonkey.

Miscellaneous

Igor Bukanov tweaked the handling of empty chunks by the JavaScript garbage collector.  That sounds boring until you see the results on Gregor Wagner’s 150-tab stress test: resident memory usage dropped 9.5% with all 150 tabs open, and dropped by 27% after all those tabs were closed.

Brian Hackett fixed a memory leak in type inference, which gets it one step closer to being ready to land.

Christian Höltje fixed a leak in his “It’s All Text” add-on that was causing zombie compartments.  This fix will be in version 1.6.0, which is currently awaiting to receive AMO approval, but can be obtained here in the meantime.  This fix and last week’s fix of a memory leak in LastPass are very encouraging — per-compartment reporters in about:memory have, for the first time, given add-on developers a reasonable tool for identifying memory leaks.  I hope we can continue to improve the situation here.  Several people have asked me for documentation on how to avoid memory leaks in add-ons.  I’m not the person to write that guide (I’m not a Gecko expert and I know almost nothing about add-ons) but hopefully someone else can step up to the plate.

Bug counts

Here’s the change in MemShrink bug counts.

  • P1: 30 (-0, +1)
  • P2: 64 (-4, +6)
  • P3: 36 (-5, +0)
  • Unprioritized: 1 (-2, +1)

Good progress on P3 bugs, but they’re the least important ones.  Other than that, new bugs are still being reported faster than they’re being fixed.  If you want to help but don’t know where to start, feel free to email me or ping me on IRC and I’ll do my best to help get you involved.

 

32 Responses to MemShrink progress, week 9

  1. Can you link the bug counts to the actual bugs when you do these reports? That’d be an easy entry point for people reading this to jump in and start browsing them.

  2. Great you read the Neowin results i posted last week. I thought your didn’t notice. 🙂

    Any idea why we are still nearly double the Mem usage of Opera in the problematic page?

    And it is also good to see we have more bugs added then fix. Why? Because it means we have more headroom for tuning. Future looks good, and that is what users, supporters and media would like to know.

    SQLite Telemetry could be the next big thing.

    • Nicholas Nethercote

      Ed: I have no idea about why we’re double Opera. I know almost nothing about layout and DOM representation, unfortunately.

  3. Another great post – I, like I’m sure many others, really appreciate the time you spend writing these up 🙂

  4. Really appreciate your efforts. Hats off to you..
    To Mozilla Developers King,
    I have two ideas in my mind, so I thought I share with you.
    1) To reduce Addons Overhead and many compartments issue (if possible) up to some extent, why not you guys build “Idle State API” (parity Chrome) so if Mozilla Windows in inactive or minimized for a time being, extension become idle and remove all compartments and reduce activity.
    Limitation: Maybe it is not at all possible, balls in your court.

    2) Why not incorporate “Page Visibility API” (parity Chrome 12+ & IE 10), so pages utilizing this get less CPU usage and it will compensate up to some extent “decode on draw”, you can use its principle in decode on draw as well.
    Limitation: Only site build to use Page Visibility API, can take benefit from it.

    These will be really helpful especially in Fennec (2nd one), only active part will be rendered so less utilization of RAM and less cpu usage means better battery life but limitation is there.
    Mozilla should incorporate Page Visibility API, and then start using gradually it in *.mozila.com, *.mozilla.org and its other sub sites as well.

    If I spitted out some thing dumb, ignore it. I know you are not Gecko guy and Addon chef but why not you file bug or talk with your fellows.

    Sorry for reposting it in new blog post again.

  5. I hope you’re aware of this, but there’s a new bug. Whenever I try to open about:memory, the browser freezes immediately and it doesn’t send any crash report next time I restart it. It can be reproduced. I’ve started to have this issue since 4-5 days.

    Firefox 8.0a1 build: 20110816030741

    • Nicholas Nethercote

      Rustam: I haven’t seen that, and I open about:memory frequently. Can you file a bug in Bugzilla, and include details such as what add-ons you have installed? Thanks.

      • I disabled my add-ons one by one and figured out that about:memory freezes the browser when LastPass (1.75.0) is enabled. The problem persists in latest Aurora and Nightly builds.

  6. In the bug count there are P4 bugs but in the MemShrink page there isn’t the related query. Are they the Unprioritized bugs? Or what else?

    • Nicholas Nethercote

      Fili: we don’t use the Bugzilla “priority” field. Instead we just track the MemShrink priority in the whiteboard by adding a tag like “[MemShrink:P1]”, and we only have P1, P2, P3.

  7. We do have an idleness API, but it’s not quite what you want:

    https://developer.mozilla.org/en/nsIIdleService

    • Thanks for replying. Great! Then I think issue is from addon developer side. Huh!! Can’t do anything.

  8. I’ve got a problem with Firefox 6 in OSX where sometimes it beachballs for 30 seconds at a time, becoming completely unresponsive. This is with no extensions installed and stock plugins. This probably isn’t memory related, but it’s super annoying. I suspect it’s some garbage that accumulated in my profile. Any ideas how to track it down?

    • Nicholas Nethercote

      Mogden, those ones can be hard to track down. It might be a problem with your profile. http://support.mozilla.com/en-US/kb/Managing-profiles has details on creating a new profile. You could try that and see if it helps. If it does, that same page has info on how to import the data from your old profile into your new profile.

      • That would solve the short term problem, but this keeps happening. It takes about 6 months to slow down a profile, it seems (happens both to mine and a family member’s machine).

        Does Mozilla have some strategy to get on top of profile slowness / corruption over time? Users shouldn’t have to manually maintain their profiles to keep the browser behaving normally.

        Sort of like Memshrink, it seems like instrumentation would greatly help to debug these sorts of problems.

    • I see a similar problem in linux. I’ve got pretty good disk subsystem (mirrored 15K rpm SCSI drives with a caching controller), but I run into firefox freezing up for lengthy periods several times a day.

      I’ll try making a new profile and see what happens with that

    • I see a similar problem in Linux. It appears to be Places related, as it tends to happen when accessing bookmarks or using a keyword search. The size of my places.sqlite is 40 MB.

  9. I’m using firefox and opera together and both use lot of memory that end up in increase virtual memory of both. Once I close opera, so the memory decreases, firefox continues to use the virtual memory.
    Is it normal? Should firefox free the virtual memory and return to use the “normal” memory?
    (Actually I didn’t tested much this case but at first glance seemed odd)

    • (This happens with firefox 6, not tried with nightly)

    • Nicholas Nethercote

      That does sound odd. The two programs shouldn’t affect each other’s virtual memory usage like that. Can you double-check your measurements?

  10. one thing that would be extremely useful is some way to find out what sort of memory resources a particular page/tab is using.

    that way those of us who are running up against memory limits can figure out which of the hundreds of pages that we have open are the ones that are the most benefit to close

  11. Why don’t you guys take frecent scripts and merge them all into a single precompiled uberscript in the cache. Then just memory map that file to demand paged executable memory.