MemShrink progress, week 11

This week was quiet in terms of patches landed.

  • Marco Bonardo changed the way the places.sqlite database is handled. I’m reluctant to describe the change in much detail because I’ll probably get something wrong, and Marco told me he’s planning to write a blog post about it soon.  So I’ll just quote from the bug: “Globally on my system (8GBs) I’ve often seen places.sqlite cache going over 100MB, with the patch I plan to force a maximum of 60MB (remember this will vary based on hardware specs), that is a >40% improvement. We may further reduce in future but better being on the safe side for now.”  This was a MemShrink:P1 bug.
  • New contributor Sander van Veen knocked off another bug (with help from his friend Bas Weelinck) when he added more detail to the “mjit-code” entries in about:memory.  This makes it clear how much of JaegerMonkey’s code memory usage is for normal methods vs. memory for compiled regular expressions.
  • I rearranged nsCSSCompressedDataBlock to avoid some unnecessary padding on 64-bit platforms.  This can save a megabyte or two if you have several CSS-heavy (e.g. Gmail) tabs open.   It makes no difference on 32-bit platforms.

But it was a very busy week in terms of bug activity.  Let’s look at the numbers.

  • P1: 29 (-2, +2)
  • P2: 76 (-10, +20)
  • P3: 38 (-1, +2)
  • Unprioritized: 22 (-5, +23)

Several things happened here.

  • Marco Castelluccio looked through old bugs and found a lot (30 or more) that were related to memory usage and tagged them with “MemShrink”.
  • Nine new bugs were filed to reduce about:memory’s “heap-unclassified” number by adding memory reporters;  many of these were thanks to Boris Zbarsky’s insights into the output produced by DMD.
  • I closed out a number of bugs that were incomplete, stale, or finished;  this included some of those newly marked by Marco, and some ones that were already tagged with “MemShrink”.
  • I tagged five leaks that were found with the cppcheck static analysis tool.

We spent the entire MemShrink meeting today triaging unprioritized bugs and we got through 23 of them.  Of the remaining unprioritized bugs, the older ones tagged by Marco and the cppcheck ones (which I tagged after the meeting) constitute most of them.

It’s clear that the rate of problem/improvement identification is outstripping the rate of fixes.  We have a standing agenda item in MemShrink meetings to go through Steve Fink’s ideas list, but we haven’t touched it in the past two meetings because we’ve spent the entire time on triage.  And when we do go through that list, it will only result in more bugs being filed.  I’m hoping that this glut of MemShrink-tagged bugs is temporary and the new bug rate will slow again in the coming weeks.

In the meantime, if you want to help, please look through the lists of open bugs, or contact me if you aren’t sure where to start, and I’ll do my best to find something you can work on.  Thanks!

8 Responses to MemShrink progress, week 11

  1. Well, I want to help but because of non serious response and lack of encouragement, I ditched the idea of this, now I just use my BMO account to vote and track some personal priority bugs.
    I highly appreciate your efforts and really admire you.. Amazing efforts, keep it up!! Appreciated efforts!!

  2. Setting image.mem.decodeondraw to ‘true’ helps a lot on image-heavy pages. Linux, 1Gb RAM.

  3. thanks. I think the SQLLite will reduce the memory usage most.
    Is the places.database.cache_to_memory_percentage setting still relevant?

  4. I was going to comment on the lack of new memory reporters and therefore the size of heap unclassified still being large. It’s good to see the nine new bugs filed on this topic. How difficult is it to add memory reporters? Is it just a matter of finding the right sections of code and calling appropriate functions?

    It seems like you are low on resource NJN? Is Mozilla giving you enough? To get more volunteer help, maybe you could do a few posts to point out how similar issues have been resolved in the past. For example, can you point me to code where memory reporters have been added?

    Is it possible that a developer who has very little C knowledge can help?

    • Nicholas Nethercote

      Adding memory reporters is often as simple as finding the right section of code and calling the right function. That’s *simple*, but not necessarily *easy*, esp. the “finding” part. But most of the memory reporter bugs have people assigned to them, so I’m hopeful that by the time the FF9 dev period ends (Sep 27) a good chunk of them will be done.

      More generally, extra hands would be good, but there’s a limited amount that people with zero Mozilla experience can do. We have one good new contributor (Sander) and I’ve been feeding him bugs that are suitable for someone with very little Mozilla experience, and he’s doing a great job with them, but there are only so many bugs in that category. And MemShrink bugs are spread out over the entire codebase, which also makes it harder for new contributors who say “I want to help with MemShrink”, as opposed to, say, “I want to help with the new JavaScript JIT”.

      Maybe I’m being too pessimistic, but that’s how it feels to me.

      • That’s a shame. I guess I’m familiar with Mozilla as I’ve been following Planet Mozilla for years however I’m not overly familiar with the code base which is I assume what you are referring to?

        Are there any key skills that you would include in a Mozilla familiarity checklist? I’m guessing but perhaps compiling Mozilla from source might be one?

        If code editing is not something I can help with, is there any general bugzilla work I can help with, such as similar work to Marco’s work of finding older bugs that might be relevant for MemShrink, or trying to reduce duplications?

        • Nicholas Nethercote

          Familiarity with the code is what I was referring to, yes.

          Something a non-coder can do is help to reproduce some of the problems filed by users, and narrow down the causes, e.g. sometimes problems occur for a user who has multiple add-ons installed, and the first step is to narrow down if Firefox is at fault, or one of the add-ons.

          Other than that… I’ll give it some thought.

  5. ok, here is the blog entry:

    http://blog.bonardo.net/2011/09/02/changes-to-places-memory-consumption-expiration-and-performances

    And from the changelog I can see that the places.database.cache_to_memory_percentage setting is removed:

    http://hg.mozilla.org/mozilla-central/diff/ed85e33792e0/browser/app/profile/firefox.js

    so it is no longer relevant.