MemShrink progress report, week 22

This was a quieter week.

Andrew McCreight finished his static analysis to detect cycle collector leaks.  See the details in the bug (and talk to Andrew) if you are interested in using this analysis.

I shrunk the size of js::HashTable by 4 bytes on 32-bit platforms and 8 bytes on 64-bit platforms.  This saves a few 10s or 100s of KB on typical workloads.

Marco Bonardo decreased the default maximum page size of SQLite connections, which can reduce SQLite memory usage somewhat.

Olli Pettay avoided some wasted space in one of the cycle collector’s data structures.  The cycle collector uses lots of memory but for a short time when it runs;  this change will reduce the size of this memory spike.

Gian-Carlo Pascutto added a memory reporter for one of the data structures used by the url-classifier.  This shows up in about:memory under “explicit/storage/prefixset” and is often over 1MB.

Justin Lebar improved the measurement of nsTArray’s memory usage, which will reduce the size of “heap-unclassified” in about:memory by a small amount.

Justin also wrote a good blog post about the challenges of addressing leaks in add-ons.

We only had seven new MemShrink bugs to triage in today’s meeting;  I’m pretty sure that is the fewest we’ve ever had.  Here are the current bug counts.

  • P1: 29 (+1/-1)
  • P2: 127 (-2/+3)
  • P3: 58 (-3/+2)
  • Unprioritized: 0 (-0/+0)

These counts are notable because the total number (214) is the same as last week!  Maybe the number will start dropping soon.

One thing worth pointing out about the +/- numbers is that if a bug is opened and closed between my weekly reports, it does not get counted in the +/- numbers.  In a way this is good, because it means that duplicate bugs and invalid bugs don’t affect the numbers.  But it also fails to capture bugs that were reported and fixed quickly.  (I usually describe such bugs in my posts, however.)

26 Responses to MemShrink progress report, week 22

  1. The data structure Olli fixed is used while the cycle collector isn’t running, so even better!

    • Nicholas Nethercote

      Oh, good. Any idea how much memory is saved in practice?

      • See comment 3 on the bug. ;) It is about 2kb of memory for every 128 suspected objects. If you set the GC/CC error console printing, it will tell you how many objects were suspected. The number of suspected objects starts out at 0 after a CC, and will go up until the next CC. The number the console prints is is probably higher than the average value. It may not be the high water mark, as things can get removed, but I haven’t ever looked into how the size of the purple buffer varies over time in between CCs.

  2. Just wondering as a layman, how difficult is it for add-on developers to measure the memory usage of their add-on? Are you working on something to make this easier? Also, are there some “educational” materials on best practices to avoid leaks for add-on developers? Perhaps these would help.

  3. Another observation about add-ons: do bundled add-ons receive special attention regarding memory use? Perhaps Mozilla shouldn’t bundle so many add-ons in the first place.

    Some of you might not know this, but at least after Firefox 4 came out, if you loaded firefox.com on a browsers with the primary languauge being set to Chinese, you got redirected to firefox.com.cn (I assume this must be official Mozilla too because of the redirect). The localized Firefox version downloaded from there came with no less than 10 (ten!!) pre-bundled add-ons, a few of which are not particularly well maintained as they got auto-disabled as incompatible during the updates to 5, 6, 7, 8.

    Note that any Chinese user trying to download Firefox from Mozilla will get redirected to firefox.com.cn, so please don’t say that this version, currently identifying it as “MozillaOnline 2011.3″, it not an official Mozilla version.

    • The version name change has me wondering if it’s a Mozilla release, or one done by the Chinese Govt with the redirect being done by the Great Firewall.

      • Mozilla China does indeed release a special localized version, where a lot of the localization is implemented using addons. I’m not familiar with what exactly these changes are, but I think it includes things like a special Chinese start page.

        • Ok. In that case I have to agree with Gaba that letting officially produced addons fall out of compatibility with the latest release is a problem. While I can’t speak to the level of inconvenience their absence causes, it gives the impression that Mozilla is disorganized and doesn’t care about Chinese speaking users.

      • It is not a GFW redirect, as this version was downloaded outside of China :-)

        I think Mozilla definitely does care about Chinese users, after all there is a heavily localized Chinese version! But it does seem that many of the people working on the main codebase are unaware of how heavy these modifications are, and they are simply not properly maintained (some addons get disabled on update)! Also, some of these modifications seem quite unnecessary, not really something localization-specific …

      • Examples of what these addons do: one has shortcuts to programs like Notepad, Paint, My Computer, it takes screenshots, alternate page zoom UI, a speed-dial like thingie, Personas, etc.

  4. Great to read about another week’s worth of hard work from Mozilla devs targeting such a critical aspect of Firefox’s potential.

  5. Bug 692487 really save a lot of memory for me after installing the addons “Expire history by dates” and “Places Maintenance” and running “expire, vacuum”. The latest nightlies are really awesome after tweaking SQLite :)

    • Nicholas Nethercote

      I’m pretty sure the SQLite DBs are vacuumed periodically, something like once a month, when Firefox is idle.

      • I converted the time in the about:support page and last run was 2010. Setting the expire date and running expire, vacuum reduces the size of 50% and the db files are now 50MB and this makes such an huge difference in memory usage.

        • if you converted the time in places.last_vacuum that’s the old preference. we don’t use it anymore, and I’m going to file a bug to remove it…

          the new preference is storage.vacuum.last.places.sqlite so please check that one.

          • I don’t see a value storage.vacuum.last.places.sqlite in about:support only places.last_vacuum. so change this in later versions please.

  6. A quick comment of appreciation to the memshrink team

    I came across this blog when searching for memory issues with FF5. I run a relatively small home machine (Vista, 2MB memory) which is quite lightly used (2 users, browsing, Thunderbird email, some light word processing). After moving to FF5 the machine would hang for up to ten minutes while handling FF page faults, usually when coming back to the machine after a period of inactivity e.g. overnight. This blog persuaded me to be patient rather than immediately switching.

    FF7 and 8 were improvements, but FF would still freeze for about a minute on restart, showing a revolving circle and saying “not responding”. With FF9 this problem has disappeared.

    I don’t know whether there was a particular fix or whether memory usage simply dropped below a threshold on my setup, but thanks to you all – and please keep it up, it really makes a difference!

    • Nicholas Nethercote

      I assume you mean 2GB, not 2MB? :)

      Thanks for being patient! It’s difficult to know what was causing the start-up pauses, but I’m glad FF9 fixed them.

  7. Today I noticed something that looks like a problem regarding memory usage in ff11.0a1 (2011-11-17) (I tried it only in x64 version).
    After opening several tabs in msdn I noticed that in about:memory heap-unclassified reaches ~50% and it won’t go down even if I close all open tabs. To reproduce try to open this page and then open several tabs from the tree vew on the left http://msdn.microsoft.com/en-us/library/windows/desktop/ms632680%28v=vs.85%29.aspx
    Thanks for your great work!

    • Bram Speeckaert

      That sounds exactly like what happened to me using yesterday’s build. Eventually Firefox was taking up 1GB of memory with less than 15 tabs loaded, and heap-unclassified was over 80%! Really weird, since memory usage really does seem to have improved on my end lately (with Firefox rarely using more than 350MB).

      I really wish I had saved my about:memory output now… If it happens again I’ll try to figure out how to reproduce it.

      • Nicholas Nethercote

        It sounds like both of you are suffering from https://bugzilla.mozilla.org/show_bug.cgi?id=702942. One change that might be the culprit has been backed out, if you could try an updated version and see if that fixes the problem, that would be great! (And if you could put any follow-up comments in the bug rather than this blog post, that would also be very helpful.) Thanks.

        • I too had this issue since the Nov 15 Nightly. I downloaded the latest hourly after the backout and everything seems to be smooth now.

          ─26.84 MB (22.54%) — heap-unclassified after 30 minutes or so of browsing, where it would have skyrocketed up to 700, 85% of FF’s usage (a hard limit before everything in Windows freezes due to paging).

        • If the about:memory page could add a reporter for whatever the culprit change consumes, then this issue could turn out to provide some insights on fighting the heap-unclassified size.