Category Archives: Firefox

MemShrink progress, week 113–116

It’s been a relatively quiet four weeks for MemShrink, with 17 bugs fixed.  (Relatedly, in today’s MemShrink meeting we only had to triage 10 bugs, which is the lowest we’ve had for ages.)  Among the fixed bugs were lots for B2G leaks and leak-like things, many of which are hard to explain, but are important for the phone’s stability.

Fabrice Desré made a couple of notable B2G non-leak fixes.

On desktop, Firefox users who view about:memory may notice that it now sometimes mentions more than one process.  This is due to the thumbnails child process, which generates the thumbnails seen on the new tab page, and which occasionally is spawned and runs briefly in the background.  about:memory copes with this child process ok, but the mechanism it uses is sub-optimal, and I’m planning to rewrite it to be nicer and scale better in the presence of multiple child processes, because that’s a direction we’re heading in.

Finally, some sad news:  Justin Lebar, whose name should be familiar to any regular reader of these MemShrink reports, has left Mozilla.  Justin was a core MemShrink-er from the very beginning, and contributed greatly to the success of the project.  Thanks, Justin, and best of luck in the future!

Using include-what-you-use

include-what-you-use (a.k.a. IWYU) is a clang tool that tells you which #include statements should be added and removed from a file.  Nicholas Cameron used it to speed up the building of gfx/layers by 12.5%.  I’ve also used it quite a bit within SpiderMonkey;  I’ve seen smaller build speed improvements but I’ve also been doing it in chunks over time.  Ms2ger started a tracking bug for all places where IWYU has been used in Mozilla code.

Ehsan asked for instructions on setting up IWYU.  There are official instructions, but I thought it might be helpful to document exactly what I did.

First, here is how I installed clang, based on Ehsan’s instructions.  I put the source code under $HOME/local/src and installed the build under $HOME/local.

  mkdir $HOME/local/src
  cd $HOME/local/src
  svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
  cd llvm/tools
  svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
  cd ../..
  mkdir build
  cd build/
  ../configure --enable-optimized --disable-assertions --prefix=$HOME/local
  make
  sudo make install

Then I followed the “Building in-tree” instructions.  The first part is to get the IWYU code.

   cd $HOME/local/src/llvm/tools/clang/tools
   svn checkout http://include-what-you-use.googlecode.com/svn/trunk/ include-what-you-use

The second part was to do the following steps.

  • Edit tools/clang/tools/Makefile and add |include-what-you-use| to the DIRS variable.
  • Edit tools/clang/tools/CMakeLists.txt and add |add_subdirectory(include-what-you-use)|.
  • Re-build clang as per the above instructions.

After that, configure a Mozilla tree for a clang build and then run make with the following options: -j1 -k CXX=/home/njn/local/src/llvm/build/Release/bin/include-what-you-use. I’m not certain if the -j1 is necessary, but since IWYU spits out lots of output, it seemed wise. The -k tells make to keep building even after errors; for some reason, IWYU triggers compilation failure on every file it looks at.

Pipe the output to a file, and you’ll see lots of stuff like this.

../jsarray.h should add these lines:
#include <stdint.h>                     // for uint32_t
#include <sys/types.h>                  // for int32_t
#include "dist/include/js/RootingAPI.h"  // for HandleObject, Handle, etc
#include "jsapi.h"                      // for Value, HandleObject, etc
#include "jsfriendapi.h"                // for JSID_TO_ATOM
#include "jstypes.h"                    // for JSBool
#include "vm/String.h"                  // for JSAtom
namespace JS { class Value; }
namespace js { class ExclusiveContext; }
struct JSContext;

../jsarray.h should remove these lines:

The full include-list for ../jsarray.h:
#include <stdint.h>                     // for uint32_t
#include <sys/types.h>                  // for int32_t
#include "dist/include/js/RootingAPI.h"  // for HandleObject, Handle, etc
#include "jsapi.h"                      // for Value, HandleObject, etc
#include "jsfriendapi.h"                // for JSID_TO_ATOM
#include "jsobj.h"                      // for JSObject (ptr only), etc
#include "jspubtd.h"                    // for jsid
#include "jstypes.h"                    // for JSBool
#include "vm/String.h"                  // for JSAtom
namespace JS { class Value; }
namespace js { class ArrayObject; }  // lines 44-44
namespace js { class ExclusiveContext; }
struct JSContext;

I focused on addressing the “should remove these lines” #includes, and I did it manually. There’s also a script you can use to automatically do everything for you;  I don’t know how well it works.

Note that IWYU’s output is just plain wrong about 5% of the time — i.e. it says you can remove #includes that you clearly cannot.  (A lot of the time this seems to be because it hasn’t realized that a macro is needed.)  I also found that, while it produced output for all .cpp files, it only produced output for some of the .h files.  No idea why. Finally, it doesn’t know about local idioms; in particular, if you have platform-dependent code, its suggestions are often terrible because it only sees the files for the platform you are building on.

Good luck!

MemShrink progress, week 109–112

There’s been a lot of focus on B2G memory consumption in the past four weeks.  Indeed, of the 38 MemShrink bugs fixed in that time, a clear majority of them relate in some way to B2G.

In particular, Justin Lebar, Kyle Huey and Andrew McCreight have done a ton of important work tracking down leaks in both Gecko and Gaia.  Many of these have been reported by B2G partner companies doing stress testing such as opening and closing apps 100s or 1000s of times over long period.  Some examples (including three MemShrink P1s) are here, here, here, here, here, here, here and here.  There are still some P1s remaining (e.g. here, here, here).  This work is painstaking and requires lots of futzing around with low-level tools such as the GC/CC logs, unfortunately.

Relatedly, Justin modified the JS memory reporter to report “notable” strings, which includes smallish strings that are duplicated many times, a case that has occurred on B2G a couple of times.  Justin also moved some of the “heap-*” reports that previously lived in about:memory’s “Other measurements” section into the “explicit” tree.  This makes “explicit” closer to “resident” a lot of the time, which is a useful property.

Finally, Luke Wagner greatly reduced the peak memory usage seen during parsing large asm.js examples.  For the Unreal demo, this reduced the peak from 881MB to 6MB, and reduced start-up time by 1.5 seconds!  Luke also slightly reduced the size of JSScript, which is one of the very common structures on the JS GC heap, thus reducing pressure on the GC heap, which is always a good thing.

 

MemShrink progress, week 105–108

This is the first of the every-four-weeks MemShrink reports that I’m now doing.  The 21 bugs fixed in the past four weeks include 11 leak fixes, which is great, but I won’t bother describing them individually.  Especially when I have several other particularly impressive fixes to describe…

Image Handling

Back in March I described how Timothy Nikkel had greatly improved Firefox’s handling of image-heavy pages.  Unfortunately, the fix had to be disabled in Firefox 22 and Firefox 23 because it caused jerky scrolling on pages with lots of small images, such as Pinterest.

Happily, Timothy has now fixed those problems, and so his previous change has been re-enabled in Firefox 24.  This takes a big chunk out of the #1 item on the MemShrink big ticket items list.  Fantastic news!

Lazy Bytecode Generation

Brian Hackett finished implementing lazy bytecode generation.  This change means that JavaScript functions don’t have bytecode generated for them until they run.  Because lots of websites use libraries like jQuery, in practice a lot of JS functions are never run, and we’ve found this can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.

Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

The completion of this finished off item #5 on the old Memshrink big ticket items list.  Great stuff.  This will be in Firefox 24.

Add-on Memory Reporting

Nils Maier implemented add-on memory reporting in about:memory.  Here’s some example output from my current session.

├───33,345,136 B (05.08%) -- add-ons
│   ├──18,818,336 B (02.87%) ++ {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
│   ├──11,830,424 B (01.80%) ++ {59c81df5-4b7a-477b-912d-4e0fdf64e5f2}
│   └───2,696,376 B (00.41%) ++ treestyletab@piro.sakura.ne.jp/js-non-window/zones/zone(0x7fbd7bf53800)

It’s obvious that Tree Style Tabs is taking up 2.7 MB.  What about the other two entries?  It’s not immediately obvious, but if I look in about:support at the “extensions” section I can see that they are AdBlock Plus and ChatZilla.

If you’re wondering why those add-ons are reported as hex strings, it’s due to a combination of the packaging of each individual add-on, and the fact that the memory reporting code is C++ and the add-on identification code is JS and there aren’t yet good APIs to communicate between the two.  (Yes, it’s not ideal and should be improved, but it’s a good start.)  Also, not all add-on memory is reported, just that in JS compartments;  old-style XUL add-ons in particular can have their memory consumption under-reported.

Despite the shortcomings, this is a big deal.  Users have been asking for this information for years, and we’ve finally got it.  (Admittedly, the fact that we’ve tamed add-on leaks makes it less important than it used to be, but it’s still cool.)  This will also be in Firefox 24.

b2g

Gregor Wagner has landed a nice collection of patches to help the Twitter and Notes+ apps on B2G.

While on the topic of B2G, in today’s MemShrink meeting we discussed the ongoing problem of slow memory leaks in the main B2G process.  Such leaks can cause the phone to crash or become flaky after its been running for hours or days or weeks, and they’re really painful to reproduce and diagnose.  Our partners are finding these leaks when doing multi-hour stress tests as part of their QA processes.  In contrast, Mozilla doesn’t really have any such testing, and as a result we are reacting, flat-footed, to external reports, rather than catching them early ourselves.  This is a big problem because users will rightly expect to have their phones run for weeks (or even months) without rebooting.

Those of us present at the meeting weren’t quite sure how we can improve our QA situation to look for these leaks.  I’d be interested to hear any suggestions.  Thanks!

MemShrink’s 2nd Birthday

June 14, 2013 is the second anniversary of the first MemShrink meeting.  This time last year I took the opportunity to write about the major achievements from MemShrink’s first year.  Unsurprisingly, since then we’ve been picking fruit from higher in the tree, so the advances have been less dramatic.  But it’s been 11 months since I last update the “big ticket items” list, so I will take this opportunity to do so, providing a partial summary of the past year at the same time.

The Old Big Ticket Items List

#5: Better Script Handling

This had two parts.  The first part was the sharing of immutable parts of scripts, which Till Schneidereit implemented.  It can save multiple MiBs of memory, particular if you have numerous tabs open from the same website.

The second part is lazy bytecode generation, which Brian Hackett has implemented and landed, though it hasn’t yet enabled.  Brian is working out the remaining kinks and hopes to land by the end of the current (v24) cycle.    Hopefully he will, because measurements have shown that it can reduce Firefox’s memory consumption by 5% or more on common workloads!  That’s a huge, general improvement.  Furthermore, it significantly reduces the number of things that are allocated on the GC heap (i.e. scripts, strings, objects and shapes that are created when bytecode for a function is generated).  This reduces pressure on the GC which makes it less likely we’ll have bad GC behaviour (e.g. pauses, or too much memory consumption) in cases where the GC heuristics aren’t optimal.

So the first part is done and the second is imminent, which is enough to cross this item off the list.  [Update:  Brian just enabled lazy bytecode on trunk!]

#4: Regain compartment-per-global losses

Bill McCloskey implemented zones, which restructured the heap to allow a certain amount of sharing between zones. This greatly reduced wasted space and reduced memory consumption in common cases by ~5%.

Some more could still be done for this issue.  In particular, it’s not clear if things have improved enough that many small JSMs can be used without wasting memory.  Nonetheless, things have improved enough that I’m happy to take this item off the list.

#3: Boot2Gecko

This item was about getting about:memory (or something similar) working on B2G, and using it to optimize memory.  This was done some time ago and the memory reporters (plus DMD) were enormously helpful in improving memory consumption.  Many of the fixes fell under the umbrella of Operation Slim Fast.

So I will remove this particular item from the list, but memory optimizations for B2G are far from over, as we’ll see below.

#2: Compacting Generational GC

See below.

#1: Better Foreground Tab Image Handling

See below.

The New Big Ticket Items List

#5: pdf.js

pdf.js was recently made the default way of opening PDFs in Firefox, replacing plug-ins such as Adobe Reader.  While this is great in a number of respects, such as security, it’s not as good for memory consumption, because pdf.js can use a lot of memory in at least some cases.  We need to investigate the known cases and improve things.

#4: Dev tools

While we’ve made great progress with memory profiling tools that help Firefox developers, the situation is not nearly as good for web developers.  Google Chrome has heap profiling tools for web developers, and Firefox should too.  (The design space for heap profilers is quite large and so Firefox shouldn’t just copy Chrome’s tools.)  Alexandre Poirot has started work on a promising prototype, though there is a lot of work remaining before any such prototype can make it into a release.

#3: B2G Nuwa

Cervantes Yu and Thinker Li have been working on Nuwa, which aims to give B2G a pre-initialized template process from which every subsequent process will be forked.  This might sound esoteric, but the important part is that it greatly increases the ability for B2G processes to share unchanging data.  In one test run, this increased the number of apps that could be run simultaneously from five to nine, which is obviously a big deal.  The downside is that getting it to work requires some extremely hairy fiddling with low-level code.  Fingers crossed it can be made to work reliably!

Beyond Nuwa, there is still plenty of other ways that B2G can have its memory consumption optimized, as you’d expect in an immature mobile OS.  Although I won’t list anything else in the big ticket items list, work will continue here, as per MemShrink’s primary aim: “MemShrink is a project that aims to reduce the memory consumption of Firefox (on desktop and mobile) and Firefox OS.”

#2: Compacting Generational GC

Generational GC will reduce fragmentation, reduce the working set size, and speed up collections.  Great progress has been made here — the enormous task of exactly rooting the JS engine and the browser is basically finished, helped along greatly by a static analysis implemented by Brian Hackett.  And Terrence Cole and others are well into converting the collector to be generational.  So we’re a lot closer than we were, but there is still some way to go.  So this item is steady at #2.

#1: Better Foreground Tab Image Handling

Firefox still uses much more memory than other browsers on image-heavy pages.  Fortunately, a great deal of progress has been made here.  Timothy Nikkel fixed things so that memory usage when scrolling through image-heavy pages is greatly reduced.  However, this change caused some jank on pages with lots of small images, so it’s currently disabled on the non-trunk branches.  Also, there is still a memory spike when memory-heavy pages are first loaded, which needs to be fixed before this item can be considered done.  So this item remains at #1.

Summary

Three items from the old list (#3, #4, #5) have been ticked off.  Two items remain (#1, #2) — albeit with good progress having been made — and keep their positions on the list.  Three items have been added to the new list (#3, #4, #5).
Let me know if I’ve omitted anything important!

MemShrink progress, week 103–104

I’m back from vacation.  Many thanks to Andrew McCreight for writing two MemShrink reports (here and here) while I was away, and to Justin Lebar for hosting them on his blog.

Fixes

areweslimyet.com proved its value again, identifying a 40 MiB regression relating to layer animations.  Joe Drew fixed the problem quickly.  Thanks, Joe!

Ben Turner changed the GC heuristics for web workers on B2G so that less garbage is allowed to accumulate.  This was a MemShrink:P1.

Bas Schouten fixed a gfx top-crasher that was caused by inappropriate handling of an out-of-memory condition.  This was also a MemShrink:P1.

Randell Jesup fixed a leak in WebRTC code.

Changes to these progress reports

I’ve written over sixty MemShrink progress reports in the past two years.  When I started writing them, I had two major goals.  First, to counter the notion that nobody at Mozilla cares about memory consumption — this was a common comment on tech blogs two years ago.  Second, to keep interested people informed of MemShrink progress.

At this point, I think the first goal has been comprehensively addressed.  (More due to the  fact that we actually have improved Firefox’s memory consumption than because I wrote a bunch of blog posts!)  As a result, I feel like the value of these posts has diminished.  I’m also tired of writing them;  long-time readers may have noticed that they are much terser than they used to be.

Therefore, going forward, I’m only going to post one report every four weeks.  (The MemShrink team will still meet every two weeks, however.)  I’ll also be more selective about what I write about — I’ll focus on the big improvements, and I won’t list every little leak that has been fixed, though I might post links to Bugzilla searches that list those fixes. Finally, I won’t bother recording the MemShrink bug counts.  (In fact, I’ve started that today.)  At this point it’s pretty clear that the P1 list hovers between 15 and 20 most of the time, and the P2 and P3 lists grow endlessly.

Apologies to anyone disappointed by this, but rest easy that it will give me more time to work on actual improvements to Firefox :)

Recent about:memory improvements

The landing of patches from two recent bugs has substantially changed the look of about:memory.  When you load the page all see now is the following.

about:memory screenshot

Measurements aren’t taken away;  you have to click on the “Measure” button for that to happen.  Also, adding ?verbose to the URL no longer has an effect.  If you want verbose output, you need to click on the “verbose” checkbox.

The motivation for this change was that about:memory’s functionality has expanded greatly over the past two years, and cramming more functionality into the existing UI was a bad idea.  There are numerous advantages to the new UI.

  • No need to control behaviour via the URL, which is admittedly an odd way to do it.
  • You can switch between verbose and non-verbose modes without having to reload the page.
  • All the buttons are at the top of the page instead of the bottom, so you don’t have to scroll down.
  • You can trigger a GC/CC/minimize memory usage event without having do a memory measurement.
  • When you save reports to file, it’s clearer that a new measurement will be taken, rather than saving any measurements that are currently displayed on the screen.
  • The buttons are grouped by function, which makes them easier to understand.

There’s also the “Load and diff…” button, which lets you easily find the difference between two saved memory report files.  Here’s some example output taken after closing a tab containing a Google search result.

about:memory diff output

You can see that all the measurements are negative.  The [-] markers on leaf nodes in the tree indicate that they are present in the first file but not in the second file.  Corresponding [+] markers are used for measurements present in the second file but not the first.

MemShrink progress, week 93–94

After lots of activity in the previous month, the past two weeks have been fairly quiet for MemShrink.

AWSY

areweslimyet.com has been proving its worth.

Jonathan Kew reduced the amount of memory taken by fonts on Fennec at start-up, which was detected by AWSY/mobile.  Jonathan also reverted a change that AWSY detected as increasing Fennec memory consumption, and filed a follow-up to investigate further.

Joe Drew fixed a bad regression on AWSY relating to image decoding.  It’s not clear to me if this was a genuine regression that users would have seen, or if it was an artifact of the way AWSY does its measurements.  Either way, it’s good that it was fixed, and props to Joe for doing it so quickly.

Finally, we closed bug 833518, which was for an AWSY regression caused by the new DOM bindings.  This was previously improved by an Aurora-only hack, but enough cases have been translated to the new bindings that we’re naturally down almost to where we were.

Miscellaneous

The mobile team abandoned their goal of making Fennec work on phones with only 256 MiB of memory.  The rationale is that Android phones with only 256 MiB of RAM are uncommon, whereas low-end phones that meet the current minimum of 384 MiB are much more common.  The mobile team will of course continue to look for ways to improve memory consumption in order to make life for users with 384 MiB phones.

I modified the JS engine so that it doesn’t emit bytecode for asm.js functions in the normal case.  This reduced the memory consumption of the Unreal 3 demo used at GDC by about 100 MiB.  I also added a memory reporter for array buffers used by asm.js, which are often quite large and weren’t being measured on 64-bit platforms.

Alexandre Poirot fixed a leak relating to dev tools.

Randell Jesup fixed a small leak in WebRTC.

Help Needed

I’m working on adding a button to about:memory trigger the dumping of memory reporter data to file.  I have a patch awaiting review, but I’m getting a test failure on Windows.  The test saves gzipped memory reports to file, and then immediately loads that saved file (and uncompresses it) and checks the data looks as expected.  This works fine on Mac and Linux, but on Windows I’m sometimes getting incomplete data in the load step.  The file is quite short (just 253 bytes compressed, and 620 bytes uncompressed) and the truncation point varies between runs;  in the most severe occurrence only 9 bytes of uncompressed data were loaded, though the cut-off point seems to vary randomly.

I suspect there’s a file synchronization problem between the save and the load, even though gzclose() is called on the save file before the loading occurs.  If anyone has ideas about what the problem might be, I’d love to hear them.

Update: Nils Maier and an anonymous commenter pointed out the problem — I was using “r” instead of “rb” for the file mode.  On Windows, this causes mangling of EOL chars.

Also, we’re seeing some strange behaviour on Mac OS X where memory managed by jemalloc doesn’t appear to be released back to the OS as it should.  This is both alarming and hard to understand, which is not a good combination.

Good First Bugs

I have two easy bugs assigned to me that I probably won’t get around to for some time.  Both of them would be good first (or second, or third…) bugs.

  • Bug 857382 is about making about:memory handle memory report diffs more elegantly.
  • Bug 798914 is just a minor code clean-up.  Nothing too exciting, but first bugs often aren’t!

Please email or comment in one of the bugs if you are interested in helping.

Bug Counts

Here are the current bug counts.

  • P1: 15 (-0/+2)
  • P2: 138 (-4/+8)
  • P3: 129 (-2/+7)
  • Unprioritized: 1 (-3/+0)

MemShrink progress, week 91–92

Bill McCloskey landed zones, a.k.a. compartment groups.  This mitigates the overhead of each compartment by allowing all compartments within a zone to share arenas and strings.  Roughly speaking, each tab gets its own zone, and there’s a system zone for Firefox’s internal use.  This was a MemShrink:P1 bug.

The following graph from areweslimyet.com shows the impact of zones — about 5/6 of the way along the graph there’s a distinct drop, most noticeable in the dark blue line.  The light green line (settled start-up) showed a ~6 MiB drop, which is ~10%.  Note that the fraction of JS memory in areweslimyet.com is less than that in typical sites, so the drop in the higher lines is smaller than the improvements normal users should see.

areweslimyet.com graph showing improvements due to zones

Avi Halachmi fixed a problem where badly managed gradients could cause spikes of 10s of MiBs when tab animations occurred.  This was a MemShrink:P1 bug.  The fix has been backported to Aurora.

Jed Parsons fixed excessive memory consumption by Persona after visiting the B2G marketplace.  At least, I think that’s what happened;  I won’t pretend to genuinely understand what went on in that bug.  This was a MemShrink:P1 bug.

Fabrice Desré fixed a bad B2G leak relating to error messages.  This bug was fixed before it was triaged in a MemShrink meeting, but it probably would have been a MemShrink:P1 because it could cause the phone to OOM after a few hours.

I removed all uses of nsFixedSizeAllocator.  This was only a small memory consumption win (a few 10s of KiBs) but it cut several hundred lines of code, and removed another low-quality custom allocator (and attractive nuisance) from the codebase.

I added a “js-main-runtime-temporary-peak” memory reporter, which measures the peak size of the main JSRuntime’s “temporary” pool, which mostly holds parse nodes.  These are short-lived but they can be quite large — 10s of MiBs in normal browsing, and we’ve seen it exceed 1.5 GiB on large asm.js examples.  Relatedly, I reduced the size of parse nodes by 12.5% on 64-bit platforms, and 16.7% on 32-bit platforms.

Interesting Open Bugs

Sometimes, especially on B2G, we have excessive memory consumption due to JS strings.  It would be useful to be able to dump the contents of all the strings in memory, to look for ones that shouldn’t be there.  Bug 852010 has been filed for this.  It shouldn’t be too hard, as it would fit in with the existing JSON memory report dumping infrastructure.  This is currently blocking bug 850893, a B2G MemShrink:P1 bug.  Please comment in the bug or contact me if you want to get involved.

Bug 846616 and bug 850545 both are about high “heap-unclassified” values.  Reducing “heap-unclassified” is getting very difficult, because the memory is often allocated in ways we can’t measure by third-party code such as Cairo and graphics drivers.  I suppose in the Cairo case we could put effort into adding some memory profiling infrastructure and try to get it upstreamed, but the driver situation seems pretty hopeless.

Bug Status

Here are the current bug counts.

  • P1: 13 (-3/+4)
  • P2: 134 (-2/+8)
  • P3: 124 (-4/+6)
  • Unprioritized: 4 (-6/+4)

MemShrink progress, week 89–90

IMAGES

The big news this week is that Timothy Nikkel finished bug 689623, which causes us to (for the most part) only decode images that are visible or nearly visible.  This goes a long way towards fixing our problems with decoded images which happens to be the #1 item on the MemShrink big ticket items list.  The fixing of this bug also happened to fix the following bugs in passing:  661304, 682230.

To give you an idea of what changed, consider an image-heavy page like this one.  Prior to Timothy’s change, shortly after loading that page on my machine Firefox’s memory consumption would reach 3 GiB, with 2.7 GiB of that being decoded (uncompressed) images.  If I switched to another tab then 2.45 GiB of those decoded images would be immediately discarded, then after 10–20 seconds the remaining 250 MiB would be.  If I switched back, all the images would immediately be decoded again and memory consumption would climb back to 3 GiB.

After Timothy’s change, Firefox’s memory consumption still peaks around 2.8 GiB just after loading the page, but then quickly drops back to 600 MiB as many of those decoded images are discarded.  After that, even when scrolling quickly through the page I couldn’t get memory consumption to exceed 700 MiB.

This change will make such pages much smoother on desktop machines that don’t have lots of RAM, because there won’t be any paging, which slows things down drastically.   Andrew McCreight wrote a comment that I can’t find now, but basically he said that on one of his machines, that page was extremely slow and painful to scroll through before the change, and after the change it was smooth and pleasant.

The peak that occurs when an image-heavy page first loads is still undesirable, and bug 542158 is open for fixing that, and it will build on the infrastructure that Timothy has developed.  When that’s fixed, it’ll also greatly reduce the likelihood that an image-heavy page will cause an out-of-memory (OOM) crash, which is great for both desktop (esp. 32-bit Windows) and mobile/B2G.

Miscellaneous

Nicolas Pierron fixed a leak in IonMonkey that was causing runaway memory consumption for Shumway demos on Android.  It has been backported to the Aurora and Beta channels.

greatly reduced the size of the property cache.  This is a data structure the JS engine uses to speed up property accesses.  It used to be important, but is now barely necessary thanks to type inference and the JITs.  This reduced memory consumption by 75 KiB per JSRuntime on 32-bit, and by 150 KiB per JSRuntime on 64-bit.  There is one JSRuntime for each process, plus one for each worker, so for B2G it saves around 525 KiB at start-up.

We declared victory in the bug tracking the memory consumption of not-yet-loaded tabs.  When the bug was opened it was over 1 MiB per tab (64-bit builds), it’s now around 200 KiB.  Furthermore, if someone has many (e.g. hundreds) of not-yet-restored tabs they’re probably on a high-end machine, so the motivation for further reducing the memory consumption is low.  There is a bug open about some scripts run for each of these tabs, however;  avoiding these scripts may make start-up faster for heavy tab users.

Marco Bonardo reduced the maximum size of the cache used by each SQLite connection (excluding places.sqlite) from 4 MiB to 2 MiB.  Update: Marco corrected me in the comments — there are four places.sqlite connections, but only one of them (the one for the awesome bar) is excluded from the new 2 MiB limit.

Somebody fixed a leak in the Roboform add-on.  The fix is present in v7.8.7.5.

Olli Pettay improved our memory reporting by adding extra information to some system compartments that previously were undistinguished.  Our experience has been that as soon as somebody adds a new memory measurement, it finds a problem, and this was no exception, leading to the filing of bug 844661 .

Jonathan Kew has been busy.

I wrote a script that can take a diff of two JSON memory report dumps.  This will be very useful when comparing the memory consumption before and after doing something, for example.  If you didn’t know that JSON memory report dumps existed, you’re probably not alone — it’s an undocumented feature that’s currently only available on Linux and Android, and is triggered by sending signal 34 to the browser(!)  These dumps can be loaded by about:memory (see the buttons at the bottom) but there’s currently no easy way to trigger them, which is why today I filed a bug to make it easier.

I removed the pool used for recycling nsNodeInfo objects, which was causing them to never be released back to the OS.  Never-shrinking structures like this can become a problem in longer-running browser sessions — for example, after running MemBench, which opens and closes 150 tabs, this pool was unnecessarily hoarding 4 MiB.

INteresting Open Bugs

In my MemShrink reports I mostly write about bugs that have been fixed.  While this is satisfying and demonstrates progress, it might also give the impression that the MemShrink team has everything under control, when really we could use some help.

(In fact, the term “the MemShrink team” is something of a misnomer since there isn’t any such entity, officially.  “The people who regularly attend MemShrink meetings” would be a more accurate term.  But I’ll use “MemShrink team” as a short-hand for that.)

The MemShrink team has expertise in some areas of Mozilla code, such as DOM, cycle collector, JS engine (partial), Fennec, B2G, and memory profiling, and we tend to make good progress in those areas — we can fix bugs in those areas, and we generally pay attention to how these areas affect memory consumption.

But we lack expertise in areas like graphics, image-handling, layout, text rendering, storage, Jetpack/add-ons, and front-end.  Graphics is a particular problem, because graphics issues, esp. those involving drivers, can cause huge memory consumption blow-ups.  Bug 837542 is an example from the MemShrink:P1 list where gradients are somehow causing memory consumption to spike by 10s or even 100s of MiBs just by opening and closing a few tabs!  We triage bugs like that as well as we can, but often we’re just guessing, and we’re mostly helpless to address such problems.

Therefore, moving forwards I’m going to start mentioning interesting open bugs that don’t have someone working on them.

One example is bug 846173, which I filed when I noticed that fully loading the front page of TechCrunch takes over 100 MiB!  And it’s mostly because of the many Facebook “like” buttons, Google “+1″ buttons, and Twitter “tweet this” buttons — see the about:memory output for the full gory details.  It’s obvious that most of these window objects are basically identical, except for the URL.  Could we do something clever to avoid all this duplication?  Justin Lebar wondered about a copy-on-write scheme.  Are there other ways we could improve this case?

Another example is bug 842003, which is looking for an owner.  Some basic leak-checking infrastructure in the IPC code could potentially detect some big problems for B2G.

In bug 842979 we’re seeing 10s or 100s of MiBs of orphan DOM nodes in long-running Gmail sessions.  It’s not clear if this is a bug in Gmail, or caused by add-ons, or something else.  Please comment in the bug if you have any additional data.

Another one:  DMD, which is one of our most important tools, is pretty much useless on Fennec because it can’t get stack traces.  If anyone who knows about such things would like to investigate, that would be very helpful.  (In fact, I got a crash in the stack tracing code when I most recently tried to run DMD on Mac, which I haven’t had time to investigate.)

Bug Counts

Here are the current bug counts.

  • P1: 12 (-6/+1)
  • P2: 128 (-7/+7)
  • P3: 122 (-0/+5)
  • Unprioritized: 6 (-2/+6)

The closed P1s were 661304, 689623, 837187, 841976, 841993, 842756.  Three of them related to an IPC leak fix that I mentioned in my last report.