MemShrink progress, week 73–74

B2G!  Fennec!  Social API!  They’re all happening, and memory consumption is a big deal for all of them.  Time for a MemShrink report.

B2G

Lots of B2G work is happening, unsurprisingly.  All of these changes have been backported to the Aurora channel.

Kyle Huey landed an important patch that merges system compartments together.  This avoids wasted space caused by having 100s of small compartments.  As Chris Jones said: “It’s a completely different phone with these patches.”  Unfortunately, the preference controlling this behaviour is currently turned off on B2G, because it’s causing some Marionette test failures.  Hopefully they’ll be dealt with soon.  Note that this preference won’t be turned on in desktop builds, and there’s a medium-term plan to avoid this wasted space in a less hacky fashion.

Marco Bonardo disabled Places in B2G, saving about 300 KiB in the main process.

Justin Lebar added code to trigger memory pressure events when lowmemkiller notifications occur.

Justin also tweaked things so that the garbage collector is more likely to run when screenshots are taken.

Jeff Muizelaar fixed a graphics bug that I don’t understand but apparently avoids allocating lots of memory on the B2G unlock screen.

Chris Jones ensured that remote content drop their buffers when they’re hidden.

I tweaked the size of the arena chunks used by XPT, which saved about 120 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I also shrunk the initial size of the SPS hash table, which saves 48 KiB per process on 64-bit builds and 24 KiB on 32-bit builds.

Memory Reporting

Lots of work happened on the memory reporting front.  This was inspired by  “heap-unclassified” typically being much higher on B2G than on desktop.  Desktop Firefox uses a single process and its memory consumption is usually measured in the 100s of MiBs or more.  In contrast, B2G uses one “main” process and then one process per running app, and the smaller ones are typically around 10 MiB.  As a result, the sundry small per-process things that don’t matter much for desktop loom larger on B2G.  (For the same reason, changes that reduce per-process memory by smallish amounts have outsize value on B2G.)  Most of the following changes have also been backported to the Aurora channel.

Nick Hurley added a “explicit/network/disk-cache” memory reporter.  It typically measures 100s of KiBs in desktop Firefox.  (It doesn’t get used on B2G.)

I added memory reporters “explicit/xpcom/component-manager” and “explicit/xpcom/category-manager”.  Together they measure about 280 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/script-namespace-manager”.  It measures just over 150 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added a memory reporter “explicit/xpcom/effective-TLD-service”.  It measures about 128 KiB per process on 64-bit builds, and a bit less on 32-bit builds.

I added some detail to the JS object memory reporters.  This gives slightly more insight into where this memory is going, but the extra memory measured as a result is usually pretty small.

I fixed the “explicit/atom-tables” memory reporter, which was erroneously always reporting 0 bytes.  It now measures anywhere from a few 100 KiBs to several MiBs of memory.  This was a frustrating bug to find.  DMD exists to check that memory reporters are measuring memory properly, and the reporter was doing its measurements just fine.  But DMD cannot check that the measured amounts are added correctly, and that’s what was going wrong here — a line that should have been this:

return n;

instead was this:

return 0;

In other words, the code I had written was doing the difficult part correctly, but botched the trivial part.  How annoying.

Social API

Felipe Gomes fixed a document leak that occurred when the social sidebar was hidden, and disappeared only when it was unhidden.

Felipe also added code to clear the previous profile when Social is toggled off, which prevents a leak.

The social API still has some significant unresolved memory consumption issues, which are a concern.

Fennec

Kartikaya Gupta (a.k.a. Kats) turned on tab expiration for Fennec.  What this means is that Fennec will now unload the contents of background tabs in certain circumstances — when memory is low, and in some cases when a tab hasn’t been viewed for over an hour — and then reload them when they are brought back into the foreground.  Read Kats’ blog post for more details.

(This change was prompted by Project 256meg, which aims to make Fennec usable on phones with only 256 MiB of RAM.  (Fennec currently requires 512 MiB, according to the official specs.)  Kats has just got to the point where Fennec can actually start on 256 MiB devices.  Although Fennec only uses a single process, there is obviously significant overlap between this goal and B2G’s memory reduction goals.)

An obvious follow-up idea is to add tab expiration to desktop Firefox.  But the bug tracking that idea has seen more heated discussion.  The reason is that tab expiration might cause data loss in some cases.  This is less of a problem on Fennec because (as far as I can tell) there is less of an expectation that in-flight data will be saved on mobile devices.  For example, having processes killed due to memory constraints on mobile is much more common than on desktop.  Still, I expect plenty more arguing before this issue is resolved one way or the other.  One option is to allow it on desktop but have it disabled by default.

Add-ons

Michel Gutierrez fixed a zombie compartment in Video DownloadHelper, which is the 2nd most popular add-on at AMO.   Version 4.9.11 has the fix.

We hit a notable milestone on the add-on front this fortnight:  on November 1st hit bug for tracking known leaks in add-ons — filed 16 months ago — had zero blockers.  In other words, we reached the point where no add-ons were known to leak!  Unfortunately this didn’t last long, and at the time of writing the tracking bug has three blocking bugs.  Still, it’s confirmation that what used to be our biggest memory consumption problem is well under control.

Bug Counts

Here are the current bug counts.

  • P1: 20 (-3/+5)
  • P2: 110 (-10/+7)
  • P3: 102 (-3/+8)
  • Unprioritized: 4 (-2/+4)

As always, the current bug lists can be found by following the links on the MemShrink wiki page.

20 Responses to MemShrink progress, week 73–74

  1. Any news on Generational GC?

  2. “The reason is that tab expiration might cause data loss in some cases.”
    And this data can’t be saved in some sort of swap file?

    • Perhaps there are other cases of unsavable data situations, but the first that comes to mind is plugins (aka, Flash & friends).
      One could be in the middle of a game or other data-storing transaction. I don’t know if we have any method to store & restore such states. This would also include Java and even the pause point within a QuickTime movie.
      As Nick said, the problem rings true across the board, but there simply is a lesser expectation on mobile of long term data safety.

    • If you’re suggesting writing it to disk, there’s often information in forms that’s not entered in password fields that I don’t want written to permanent storage. Having to purge my form autocomplete list every second time I need to enter my SSN because the developer used a plaintext input box is bad enough. Having to find an additional place to do so would be even worse.

  3. then what about landing that on desktop for background tabs that don’t have active plugins (with click-to-plugin, this percentage is sure to grow), and use that to pressure plugin makers (adobe, oracle) to come up with a solution to the problem?

    • (this was in reply to Caspy7)

      • To add to what @tom_jones said, there could be a whitelist of websites that do not get “hibernated”:

        1. Websites with plugins (tom’s thought)
        2. Https websites
        3. Pinned tabs

        This would resolve @Dan’s concern about not having to re-enter any PII in a web form. @Dan, I hope you are entering PII only in a form submitted over HTTPS.

        Thanks,
        Manoj

  4. Why not tab expiration for Panorama? Panorama groups stay loaded in memory hours after you stop using them.

  5. The tab expiration is nice for some cases, but I want to have an option to disable it (as well as the “don’t load contents at startup, wait until they are accessed”). I’m willing to wait a bit when I start firefox (not that I really have much choice :-), but after it’s started, running into a delay whenever I select a tab is annoying. Sometimes I take the several minutes to flip through all my windows and do “reload all tabs”, but it’s a hassle.

  6. Since Nicholas you are also behind Generational GC team, I also like to put your attention towards new achievement of Chrome team (We should appreciate where they deserve), so they recently added “Generational DOM GC” as a result on their benchmarks, they saved about 24 MB on Facebook while 235 MB on Google Calendar.
    Here are some links regarding Generational DOM GC:
    https://docs.google.com/document/d/16DeHrzkm3cO9XCPT1aK3Y5qgUxXB3RFmueqQWYmN2rI/edit
    Performance and memory results: https://docs.google.com/document/d/1h0-EsHu7T0sSMuZm5eE0r1e8sCAzY3weLvsDUpOSngE/edit
    Slides: https://docs.google.com/presentation/d/1uifwVYGNYTZDoGLyCb7sXa7g49mWNMW2gaWvMN5NLk8/edit#slide=id.p

  7. Hi Nick,

    Will increasing the image decode chunk size to 16k from 4k (https://bugzilla.mozilla.org/show_bug.cgi?id=811467) have an adverse impact on memory usage for websites like Facebook that are image intensive?

    Thanks,
    Manoj

    • If the data is kept in 16k chunks (vs the change just affecting a buffer/temp memory) the average amount of wasted memory would increase from 2k to 8k per image.

      That said, the bug was marked as complete a few days ago and there’s not a visible bump on are we slim yet’s image tracker; so it appears not.

  8. Hi Nicholas,

    I haven’t posted for a while but thought I would update you with one of my very basic memory tests that I do from time to time.

    Using 18 b1 I opened all my bookmarks in one go. Roughly 70 tabs. I surfed about a bit and the memory usage got up to about 850,000 kb.

    Then I closed all but one tab using the “Close Other Tabs” feature.

    Memory dropped to 430,000 kb but never went any lower with only one tab open.

    I did the test again and when I closed all but one tab again using the “Close Other Tabs” feature I decided that the one tab I leave open this time was the homepage of this blog.

    Again the memory dropped from 850,000 kb, this time stopping at 340,000 kb but never going any lower.

    340,000 kb of memory used with JUST this blog open doesn’t seem right.

    Is this a known issue? Whilst the memory usage when many tabs are loaded is a lot lower than it was 18 months ago, and no longer climbs exponentially of its own accord, the speed with which memory is released once tabs are closed is still either shockingly or non existent.

    My laptop is 2.3 GHz dual core, 4 GB of RAM and Windows 7.

    The only add-ons I have installed are British English Dictionary 1.19.1 and Test Pilot 1.2.2.

    Any ideas?

    • Nicholas Nethercote

      The problem is fragmentation. That 340,000 kb would contain a lot of empty space. The good news is that if you reopen lots of tabs, the space will be filled in, so it’s not a growing problem. Compacting GC will help some with this for the JS heap, but so long as Firefox is single-process this behaviour will occur for the C/C++ heap. That’s one advantage of a multi-process model — if you close a tab and the process is killed, all the memory is definitely recovered.

  9. I meant to say that if you want I can email you my bookmarks file if that would help at all. As far as I can make out there are no sites in there that would be causing memory leaks once the tab was closed, but I guess I don’t know for sure.

    This is one memory test where Chrome still wins because each tabs is a separate process. That said, Chrome’s memory usage with 40 tabs open is well over double what Firefox uses according to the latest Tom’s Hardware tests.

    I know you’re not a fan of the tests they do on Tom’s but it’s still the same tests done on the same system with the same software and hardware installed, same network and internet connection etc. It is at least in that sense a level playing field. It appears Firefox is leading the pack in some areas but not others.

    http://www.tomshardware.com/gallery/MemUse40WBGP14,0101-360351-0-2-6-1-png-.html