MemShrink progress, week 77–78

DMD

The big news this week is the landing of the native version of DMD.  The old version of DMD is a Valgrind-based tool that has been instrumental in reducing the size of about:memory’s “heap-unclassified”.  (For example, with trunk builds on my Linux desktop machine it’s now frequently less than 10%.)

However, the old version of DMD wasn’t easy to run.  For example, it required patching Valgrind’s source code and re-building it, among other things.  As a result, only a handful of people ever ran it.  Furthermore, because it’s a Valgrind tool it is very slow and will never run on Windows.

In contrast, the new version is much easier to use.  It just requires a slight configuration change at build time and a slight change in how the browser is invoked.

It’s also much faster, especially if you use the sampling mode which trades a small amount of precision for a great deal of speed.  Crucially, this means it is usable on B2G.  Also, it should be possible for people to use it for long-running sessions, which can be helpful for identifying slow leaks.

And while the new version doesn’t currently run on Windows, it should be fairly straightforward to get it working.  If anyone is interested in helping with this, please contact me, or take a look at the open bug.

I will write a more detailed post about the new version some time in the next few days, once I have updated the documentation and finalized a few more tweaks that should improve usability.  In the meantime, Justin Lebar is already using it in earnest to better understand B2G’s memory consumpion (e.g. see here, here, here, here, here, here, here).

Thanks to Mike Hommey for his excellent replace-malloc infrastructure, on which the new version is built, and to Justin for lots of help, especially with Android and B2G details.

B2G

The option that allows B2G to merge system compartments, previously implemented and landed by Kyle Huey, was enabled by default.  This is a big deal, because it’s the single biggest memory consumption improvement B2G has seen and is likely to see before version 1 is released.

Gabriel Svelto reduced the number of unused dirty pages kept around by jemalloc on B2G.  This can save ~4 MiB of memory across all processes, at a potential cost of making some allocations a little slower.  On a device as memory-constrained as the B2G phones, this is a good trade-off.

Social API

Felipe Gomes reduced the memory consumption of the social API when multiple browser windows are open.  This change has been backported to the beta channel, so it will be present in Firefox 18, in time for the bigger publicity push for this feature.  (For those of you that don’t like Facebook, please note that if you don’t enable the social API it will not affect Firefox’s performance in any way.)

Miscellaneous

I added a MEMORY_VSIZE telemetry reporters, which measures virtual memory consumption.  This might help us understand how many Windows users would benefit from 64-bit builds.

Bug counts

Here are the current bug counts.

  • P1: 17 (-3/+3)
  • P2: 126 (-2/+14)
  • P3: 106 (-4/+7)
  • Unprioritized: 0 (-2/+0)

14 Responses to MemShrink progress, week 77–78

  1. Your memory tools don’t run on your current biggest platform? Facepalm

    • Even without the need to patch it’s source first, Valgrind is not an end user tool and mac/linux are the OSes of choice for most of Mozilla’s developers.

    • Nicholas Nethercote

      > Your brand new memory tool doesn’t yet run on the platform that’s hardest to develop for, but you have a bug filed for implementing it and it shouldn’t be much work? Fair enough.

      There, I fixed that for you.

  2. sounds truly like a lot of genuine progress. when the new val grind windows version is ready, hopefully even people as unskilled in compiling and using desktop development tools may be able to run it. will be very handy if windows can hit single figures for heap unclassified as Linux is nearing.

    big thanks to everyone working really hard on memshrink. to think that B2G is targeting deices with as little as 256 Meg is amazingly ambitious but will be great for the raspberry pi, for example.

  3. >This might help us understand how many Windows users would benefit from 64-bit builds.
    Don’t worry. Most of them already left. And Windows 9 is rumored to be 64-bit only.
    Disclaimer: I don’t use 64-bit Windows.

    • Win8 was supposed to be the same. As you can see, it dropped x86 so hard it even learned ARM too.
      Win9 (Win Blue) will be the same, especially after they merged all these stuff instead of creating a separate OS for mobile/tablet platforms. They cannot do a 180° turn now.

  4. Wasn’t the Windows x64 nightly discontinues a few weeks ago?

    Any news on some of the bigger items, such as the real fix to cpg (allocate memory in zones rather than compartments) or generational gc?
    I know those are months of hard work and I don’t see a lot of progress on bz, even though I’m subscribed to all the bugs.

    Some kind of fancy metrics like “generational gc is 85% done” would be awesome.

    • As an interested layman I second this request :)

      Nicholas, I know you’ve worked on quite a few bugs related to rooting – that’s a prerequisite to GGC, right? How much work is left there, and how close would it put us toward having a Generational + Moving GC?

    • Terrence Cole has been making weekly “exact rooting updates” to mozilla.dev.tech.js-engine.internals if you are interested in the nitty gritty details. That’s only the first step towards a generational GC, but it is a large chunk of the work required.

      • If those numbers are counts of places where changes need made, I hope someone can at a minimum work out a semi-automated way to do most of the work.

    • Nicholas Nethercote

      > Wasn’t the Windows x64 nightly discontinues a few weeks ago?

      I believe that decision will be reversed soon.

      > Some kind of fancy metrics like “generational gc is 85% done” would be awesome.

      But probably not accurate. There’s a long way to go.

  5. > I added a MEMORY_VSIZE telemetry reporters, which measures virtual memory consumption.

    ++

  6. Hi Nick,

    There is a noticeable uptick in the memory reported for one test, “After TP5 [+30s]“, over at areweslimyet.com. From a look at the graphs, JS could be the root of the problem. A number of Ion fixes have landed in the last few weeks. Here is the pushlog for the week during which one of the larger spikes was registered: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6fa6e55a93b2&tochange=c8a1314aa449

    Is this expected? As you are investigating, is there something I can do to help test interim builds, etc.?

    Thanks,
    Manoj