MemShrink progress, week 30

Add-ons

This was the week of add-ons in MemShrink-land.

Jared Wein fixed a problem in Firefox that was causing zombie compartments if you viewed a native video with any add-on installed that implements the nsIContentPolicy interface.  Examples of such add-ons are Adblock Plus, GreaseMonkey, and NoScript, which are respectively the #1, #3 and #9 most popular add-ons on AMO!  Welcome to the MemShrink club, Jared.

Speaking of GreaseMonkey, Arantius fixed a bug in it that was causing zombie compartments on some GM scripts when opening a background tabs.

I can’t tell who was responsible for the next fix, because the Add-on SDK folks use Github in a way I don’t understand.   But Myk Melez and/or Gabor Krizsanits greatly reduced the number of compartments used in JetPack-style add-ons.  In one example, the number of compartments dropped from 156 to 8, saving about 20MB of memory.  This was a MemShrink:P1 bug.

Also, as far as I can tell, the same patch also fixed bug 680821 which means that all compartments that hold sandboxes created by JetPack-style add-ons will be marked as belonging to that add-on in about:memory.

Jordan Miner reported that the Delicious Bookmarks add-on is leaking excessive numbers of connections to the Places database.  I tried to reproduce the problem and failed, but I don’t have a Delicious account.  If anyone else who does have a Delicious account is able to reproduce, please let me know or comment in the bug.

Finally, Jorge Villalobos and Andrew Williamson from the add-ons team came to this week’s MemShrink meeting.  We had a very fruitful discussion about how to help add-on authors detect and avoid memory leaks, and how to help add-on users understand which add-ons have high memory consumption.  Stay tuned for more details in the coming weeks!

Three Cheers for New Contributors

Krzysztof Kotlenga, a new contributor, removed an OpenGL cache that was wasting 1GB+ of memory on Linux when hardware acceleration was enabled (it’s currently not enabled by default, but will be at some point in the future).   Great work, Krzysztof!  [Update: I originally wrote WebGL instead of OpenGL, which was incorrect.]

Bug Counts

Here are the current bug counts.

  • P1: 26 (-2/+0)
  • P2: 132 (-14/+3)
  • P3: 67 (-2/+5)
  • Unprioritized: 4 (-0/+4)

That’s a net reduction of six bugs.  The main factor here was that I went through some of our P2s and closed ones that were stale and/or covered by other bugs, and downgraded to P3 a few more that are now known to be less important than we first thought.

Before finishing, I’d like to highlight bug 703427.  Richard Hipp, one of the SQLite developers, has a tiny patch that drastically reduces the amount of memory used by SQLite in Firefox — I ran with it for a while and my SQLite memory consumption dropped from ~15MB to less than 3MB.  The patch also causes a moderate speed drop, but it’s unclear if that speed drop is noticeable to Firefox.  We need someone who understands Firefox’s SQLite usage well to evaluate this patch so that Richard knows if it’s something that should go into a released version of SQLite as an option.  The bug is assigned to Marco Bonardo but he’s currently very busy.  Is there anyway else who knows enough about SQLite to take on this bug?

31 Responses to MemShrink progress, week 30

  1. Kevin Brosnan

    Could we checkin the patch to nightly and use telemetry to see if it regresses performance?

  2. >1,235,302 B (01.38%) — startup-cache
    Shouldn’t this cache be dropped after start?

    • Nicholas Nethercote

      Yes, but it may take a few minutes.

      • Well, after reading his comment, I had a look in about:memory. I am seeing 1,590,611 B of startup-cache, while this session has been open for hours, so I confirm this as a bug. It’s not a lot, but 1-2MB wins here and there will count up eventually :)

      • It does not. I’ve waited few hours. “Minimize” button doesn’t help either.
        Mozilla/5.0 (X11; Linux i686; rv:12.0a1) Gecko/20120110 Firefox/12.0a1

        • An update to my post. Startup cache uses ~3.5Mb at FF startup, then after few minutes goes down to ~1.2Mb. Theese 1.2Mb is mapping of “startupCache.4.little” file, see:
          cat /proc/`pgrep firefox|head -n1`/maps|grep startup
          I’m not sure if this is a bug or not.

          • Nicholas Nethercote

            I talked to Taras, I don’t think it’s a bug. The reporters have been expanded, I now see something like this:

            ├────2,843,584 B (00.96%) — startup-cache
            │ ├──2,842,944 B (00.96%) — mapping
            │ └────────640 B (00.00%) — data

            The mapping is the file, that’s expected to still be there. Because it’s backed by a file it’ll be swapped out after a while and not touched again, so it’s not a big deal. The data is what’s supposed to drop to (almost) zero. I’m not sure why it’s not exactly zero, but Taras said it wasn’t a bug.

  3. Regarding memory usage, you might want to take a look atthis webpage: http://www.head-fi.org/t/478568/multi-iem-review-214-iems-compared-jvc-ha-fxt90-and-ha-fx500-added-1-8-12

    On my 1GB netbook running AMD64 Debian Wheezy, visiting this page with Firefox 9, causes severe swapping. With some luck and lots of patience I might succeed in logging in in tty1 and killing Firefox, but in the worst case, I need to do a hard reset of my system. Chromium on the other hand, opens this site without any problem.

    • Taking the many images on the page into account, my guess is that the problem is probably (once again) Firefox inefficient handling of images where all images in the visible tab are decoded and kept in memory, irregardless of whether an image is currently visible or not. This has been identified as a severe problem a long time ago. Bug 683284 is the tracking bug for this. Unfortunately, despite the potential big wins here, there has been little progress in the last months here so far. :-(

      The site is also very text-heavy, so the patches from bug 703100 might also help here.

    • I have absolutely no idea why such important bug isn’t fixed. But i was pointing out a web site in last week’s Memshrink with similar issues. At least it is a P1 bug so hopefully they will get around….

      • Looks like Nicholas is trying to push the respective developers to renew their efforts to finally fix this issue. 頑張ってよ!

  4. In about:memory, I can see quite a lot of bytes marked as “unused” inside gc-heap > arena. In this session of about 10 tabs loaded in memory, it counts up to over 40MB. Noticable is that 20MB is from Facebook alone and 10MB is from [System Principal]. Other sites are ~1MB or less. I might add that Facebook (and only Facebook) becomes slow after multiple hours of usage. Using Nightly 12.0a1 2012-01-10, but I have been seeing this slowness for a while. It’s only now I went to have a look in about:memory though.

  5. Hi. The common memory leak I’m seeing during my surf sessions is this one about the quick find bar that creates zombie compartments, see https://bugzilla.mozilla.org/show_bug.cgi?id=669845 It can affect many users as soon as they run a quick search, especially on “heavy” pages.

  6. Nit: The cache removed was a OpenGL cache, not a WebGL cache.

  7. Hi Nicholas,

    I am using Firefox 9.0.1 on OS X 10.7 and I noticed that the “resident” memory always stays (more or less) at the peak value for the lifetime of the application. For instance, I open many tabs and the resident number rises to 1 GB. After closing all tabs the “explicit” number will drop approximately to previous levels (no zombie compartments or anything else which is obviously out of order) but “resident” will not. Is this one of the issues which will be solved with Firefox 10 which uses jemalloc instead of the system allocator?

    • Nicholas Nethercote

      Good question, but I don’t know. Hopefully! :)

    • Is this one of the issues which will be solved with Firefox 10 which uses jemalloc instead of the system allocator?

      It should be, yes, with one caveat:

      Even if you measure Firefox 10’s resident memory using the Mac Activity Monitor, you’ll need to open about:memory in order to observe the resident size dropping to its true value.

      The reason is, jemalloc frees pages by calling madvise(MADV_FREE). The page doesn’t actually get removed from Firefox’s address space until the operating system runs low on physical memory, so the activity monitor still counts it against us. When you open about:memory, we forcibly munmap the MADV_FREE’d pages so that the resident size is correct, in about:memory and the activity monitor.

      Please let us know if things don’t look right in FF10. (You can also get it on the beta channel today!)

      • An explanation from the man who got his hands dirty with jemalloc on OS X himself. Thanks :-)

        Because of using exotic variants of Linux for years (including Gentoo) I’ve become weary of using beta software in general (that’s also the reason why I pay premium to use a Mac although it has shortcomings of its own). I’ll report back once Firefox 10 has been pushed to the release channel should the aforementioned oddity persist.

  8. In other add-on news, HTTPS-Everywhere (git version) has reduced their ram usage by an order of magnitude or so: https://trac.torproject.org/projects/tor/ticket/4804

    • Caveat: https-everywhere git isn’t stable at the moment (it has some runaway cpu issues, maybe due to redirect loops).

  9. Hey njn, can you blog about the GC memory visualizer?

  10. Very cool. I was seriously contemplating switching my default browser but your team’s work has kept me with Firefox. The FF9 is heaven!

    The only addon I had memory leak issues with is the Microsoft’s h.264 addon for firefox : http://www.interoperabilitybridges.com/html5-extension-for-wmp-plugin
    I wonder if all these fixes will fix the issue with this addon too.

  11. something that would be useful to have when looking at the memory usage would be the info on how many windows/tabs are open. It seems to me that this would make send in about:memory, possibly with a breakdown of what windows/tabs are part of each compartment each is part of (I have a dozen or windows open with a few hundred tabs between them, but 9 compartments in about:memory, the compartmetns have URLs on them, but where does everything else live?)