Categories
about:memory compartments Firefox Memory consumption MemShrink

MemShrink progress, week 5

There were four main areas of MemShrink progress this week.

Killing zombie compartments

A zombie compartment is one that outlives its tab.  Kyle Huey fixed a very common class of short-lived zombie compartments caused by XmlHttpRequests.  These zombie compartments only lived for a couple of minutes, but this is an important fix because it makes makes memory usage follow tab usage better, and also makes detecting longer-lived zombie compartments much easier.

Alon Zakai has also been doing heroic work tracking down a zombie compartment related to web workers exhibited at coffeekup.org.  I’m sure he’d appreciate additional help from anyone who might be able to provide it.

Improving about:memory

Lots of progress here.

  • I changed the per-compartment memory reporters so that multiple system compartments with the same name are reported separately, instead of being merged.  This made it obvious that JetPack-based add-ons can have dozens of system compartments.  I don’t know if anybody realized this previously, and it’s something of a concern.  The compartments are currently distinguished in about:memory only by their address, as the following example shows.  system principal including its address
    It would be great to add some identifying information that indicates what part of the code is responsible for creating each system compartment.  That might even give us some much-needed per-add-on accounting.
  • Andrew McCreight added a memory reporter for the cycle collector.
  • I added a memory reporter for the Necko (network) memory cache.
  • Justin Lebar fixed a problem with the image memory reporters, but it bounced due to a possible Tp5 RSS/PrivateBytes regression on Mac.  This was surprising, maybe it was just noise?
  • I changed about:memory so that if there are multiple memory reporters with the same name, you can tell they were merged into a single entry.  For example, in the following screenshot you can easily tell that there were four separate connections to the places database.
    places entry indicating duplication

Acting on memory pressure

Justin Lebar made some great progress on triggering memory pressure events when physical or virtual memory is low on Windows.  See comments 28 and 29 on that bug for some nice graphs showing how this kept Firefox responsive when browsing some very image-intensive sites.  This kind of adaptive behaviour is really important for getting Firefox to behave well on the full range of supported devices, from smartphones all the way up to desktop machines with lots of RAM.  Go Justin!

Tweaking MemShrink processes

The MemShrink wiki page used to be vague about the project’s goals.  So this week I made it more precise.

[The] goal is to get the number of MemShrink P1 bugs down to zero. That will mean that all the bad leaks will have been fixed, and also the important auxiliary work (e.g. infrastructure to detect regressions) will be in place.

As a result, Jeff Muizelaar changed areweslimyet.com to point to the list of open MemShrink P1 bugs.  It previously pointed to a Tp5 Talos graph.

On a related noted, we have six open meta-bugs for tracking leaks against Firefox releases, one for tracking non-leak memory use reductions, and one for improving memory-related tools.  (These are listed on the wiki.)  I created these bugs before MemShrink meetings started.  But now that we are using the MemShrink whiteboard annotations assiduously, these tracking bugs don’t seem necessary — having two tracking mechanisms is overkill.  In particular, I think their dependencies aren’t being updated consistently.  So I propose to stop using them and close them.  If you have any objections, please let me know and I’ll reconsider.  If I do close them, I’ll make sure that all bugs blocking them have a MemShrink annotation so they won’t fall through the cracks.

And that segues nicely into the MemShrink bug count for this week:

  • P1: 26 (+2)
  • P2: 49 (+0)
  • P3: 33 (+4)
  • Unprioritized: 6 (+4)

Like last week, this increase mostly reflects the fact that people are coming up with new ideas for improvements.

Finally, thanks to Jesse Ruderman for taking minutes at this week’s meeting (and the previous two).

23 replies on “MemShrink progress, week 5”

Absolutely sensational to read this stuff (except for the JetPack news). Everyone’s doing a great job. Thanks for all the efforts.

Are this week’s changes available in Aurora yet?

pd: The changes aren’t available in Aurora. Aurora (and Beta) are for stabilization, so they only get bug fixes. You’ll have to switch to Nightly if you want these changes straight away, or wait until the 2nd half of August when the current Nightly code gets promoted to Aurora (see https://wiki.mozilla.org/RapidRelease/Calendar).

OK so forgive me for getting a bit confused. I am on Aurora 7.0a2 (2011-07-19) and I get frequent prompts to update (so frequent, they are ‘nightly’), and there’s a hyperlink in the dialogue that takes the user to a generic page that says nothing at all about what the update they are being prompted to install actually contains. A change log would be nice.

Unfortunately I can’t switch to nightlies as I need these really useful things called extensions to work 🙂

pd: you’ll get periodic updates to Aurora as new bug fixes get completed. Every night is more often than I’d expect, though; hopefully that’ll settle down as we further through the dev cycle.

The memory tools one is kind of handy, and those are mostly fairly long term projects, so they aren’t necessarily things that can just be fixed and then you are done forever.

pd: there are two issues. One is that people do still commit bug fixes to Aurora, every day or two. The second thing is that until recently they built a new version of Aurora every day whether or not there were any changes, so you’d get needless updates that didn’t actually do anything. I believe they recently fixed that, so hopefully you won’t get that update prompt quite as often!

Hi,

how to trace heap-unclassified mem usage in Firefox?

here is my output of FF7.0a2:

261.67 MB (100.0%) — explicit
├──105.09 MB (40.16%) — heap-unclassified
├───77.92 MB (29.78%) — js
│ ├──50.20 MB (19.18%) — compartment([System Principal])
│ │ ├──22.84 MB (08.73%) — gc-heap
│ │ │ ├───8.47 MB (03.24%) — objects
│ │ │ ├───6.62 MB (02.53%) — shapes
│ │ │ ├───6.08 MB (02.32%) — arena-unused
│ │ │ ├───1.50 MB (00.57%) — strings
│ │ │ └───0.16 MB (00.06%) — (3 omitted)
│ │ ├──14.84 MB (05.67%) — mjit-code
│ │ ├───3.76 MB (01.44%) — scripts
│ │ ├───3.56 MB (01.36%) — string-chars
│ │ ├───3.00 MB (01.15%) — object-slots
│ │ ├───1.96 MB (00.75%) — mjit-data
│ │ └───0.24 MB (00.09%) — (2 omitted)
│ ├──19.48 MB (07.44%) — gc-heap-chunk-unused
│ ├───5.00 MB (01.91%) — compartment(atoms)
│ │ ├──3.03 MB (01.16%) — string-chars
│ │ ├──1.97 MB (00.75%) — gc-heap
│ │ │ ├──1.73 MB (00.66%) — strings
│ │ │ └──0.24 MB (00.09%) — (6 omitted)
│ │ └──0.00 MB (00.00%) — (6 omitted)
│ ├───2.18 MB (00.83%) — compartment(http://blogs.technet.com/b/dmelanchthon/…)
│ │ └──2.18 MB (00.83%) — (8 omitted)
│ └───1.07 MB (00.41%) — (5 omitted)
├───75.01 MB (28.66%) — storage
│ └──75.01 MB (28.66%) — sqlite
│ ├──59.73 MB (22.83%) — places.sqlite
│ │ ├──59.42 MB (22.71%) — cache-used
│ │ └───0.31 MB (00.12%) — (2 omitted)
│ ├──11.54 MB (04.41%) — urlclassifier3.sqlite
│ │ ├──11.45 MB (04.38%) — cache-used
│ │ └───0.08 MB (00.03%) — (2 omitted)
│ └───3.74 MB (01.43%) — (12 omitted)
├────1.87 MB (00.71%) — images
│ ├──1.36 MB (00.52%) — content
│ │ ├──1.36 MB (00.52%) — used
│ │ │ └──1.36 MB (00.52%) — (2 omitted)
│ │ └──0.00 MB (00.00%) — (1 omitted)
│ └──0.51 MB (00.19%) — (1 omitted)
└────1.79 MB (00.68%) — (2 omitted)

Other Measurements
769.65 MB — vsize
373.42 MB — private
323.38 MB — resident
278.08 MB — heap-committed
245.64 MB — heap-used
95.36 MB — heap-unused
46.00 MB — js-gc-heap
13.12 MB — gfx-d2d-surfacevram
3.70 MB — heap-dirty
1.77 MB — gfx-surface-image
1.62 MB — gfx-d2d-surfacecache
1.31 MB — shmem-allocated
1.31 MB — shmem-mapped
0.24 MB — canvas-2d-pixel-bytes
0.00 MB — gfx-surface-win32

the technet blog was the last page I visited before I looked at the memory values.

what’s the best way to monitor the heap grow to see which sites cause this (xperf creates too large files if you monitor the memory usage)?

André: I managed to reproduce a 50% “heap-unclassified” value on that site, so I’ve added the details to bug 563700, thanks.

Nicholas: If I open a page with some huge images (like this one: https://bug664642.bugzilla.mozilla.org/attachment.cgi?id=539707), close that tab and go to about:memory. Even if i use the ‘Minimize memory usage’ button, i still see 70MB of memory used by images:

158.82 MB (100.0%) — explicit
├───72.84 MB (45.86%) — images
│ ├──72.68 MB (45.77%) — content
│ │ ├──72.68 MB (45.77%) — used
│ │ │ ├──71.70 MB (45.15%) — raw
│ │ │ └───0.98 MB (00.62%) — uncompressed
│ │ └───0.00 MB (00.00%) — (1 omitted)
│ └───0.15 MB (00.10%) — (1 omitted)

This memory is released only when I open a new tab.

What classifies as “unusually high” for heap-unclassified? I have a site that gets around 20-25% on it.
It’s a small amount of ram though, -just 15mb- but not sure if that percentage is something to worry about.

keep up the good work 🙂

gonchuki: 20-25% is quite low! 35–40% is pretty common. Greater than 50% is high.

Nicholas, I want to post results (about:memory) on few bugs, can you tell me when DOM reporter will get into about:memory. Any help, which I can do for you?
Also
http://www.w3.org/2000/10/swap/test/plist/iTunesMusicLibrary.xml
Test case of Bug 291643 also causes 60%+ heap unclassified. Also I noticed one thing, this test case url also uses huge memory on Google Chrome (400 MB round about). I want to clear one thing:
Zombie compartments are such compartments, which lives after even closing tabs???

Ahmad: thanks for the info.

Memory reporters for the DOM are being actively worked on. See https://bugzilla.mozilla.org/show_bug.cgi?id=663271 for the tracking bug. Hopefully they’ll be turned on soon, but I don’t know exactly when they’ll be ready.

As for why Google Chrome uses lots of memory — that page has a *lot* of data, so I expect it’ll use lots of memory in any browser.

Thanks for the XML test case. I added a note to the DOM reporter bug: https://bugzilla.mozilla.org/show_bug.cgi?id=663271#c15.

A zombie compartment is one that stays alive after its tab has closed, yes. But there are some subtleties, see this blog post for full details: http://blog.mozilla.org/nnethercote/2011/07/20/zombie-compartments-recognize-and-report-them-stop-the-screaming/

@Nicholas

with Aurora I get this value right at startup:

46,287,371 B (38.40%) — heap-unclassified

is this ok?

120,527,142 B (100.0%) — explicit
├───60,058,214 B (49.83%) — js
│ ├──29,985,208 B (24.88%) — compartment([System Principal])
│ │ ├──18,067,456 B (14.99%) — gc-heap
│ │ │ ├───8,128,400 B (06.74%) — objects
│ │ │ ├───5,814,480 B (04.82%) — shapes
│ │ │ ├───2,410,568 B (02.00%) — arena-unused
│ │ │ ├───1,587,552 B (01.32%) — strings
│ │ │ ├──────70,576 B (00.06%) — arena-headers
│ │ │ ├──────55,736 B (00.05%) — arena-padding
│ │ │ └─────────144 B (00.00%) — xml
│ │ ├───3,748,574 B (03.11%) — string-chars
│ │ ├───3,463,846 B (02.87%) — scripts
│ │ ├───2,864,216 B (02.38%) — object-slots
│ │ ├───1,310,720 B (01.09%) — mjit-code
│ │ ├─────298,100 B (00.25%) — mjit-data
│ │ ├─────131,072 B (00.11%) — tjit-code
│ │ └─────101,224 B (00.08%) — tjit-data
│ │ ├───74,000 B (00.06%) — allocators-reserve
│ │ └───27,224 B (00.02%) — allocators-main
│ ├──24,254,912 B (20.12%) — gc-heap-chunk-unused
│ ├───4,497,784 B (03.73%) — compartment(atoms)
│ │ ├──2,765,176 B (02.29%) — string-chars
│ │ ├──1,732,608 B (01.44%) — gc-heap
│ │ │ ├──1,574,384 B (01.31%) — strings
│ │ │ ├────148,624 B (00.12%) — arena-unused
│ │ │ ├──────6,768 B (00.01%) — arena-headers
│ │ │ ├──────2,832 B (00.00%) — arena-padding
│ │ │ ├──────────0 B (00.00%) — objects
│ │ │ ├──────────0 B (00.00%) — shapes
│ │ │ └──────────0 B (00.00%) — xml
│ │ ├──────────0 B (00.00%) — object-slots
│ │ ├──────────0 B (00.00%) — scripts
│ │ ├──────────0 B (00.00%) — mjit-code
│ │ ├──────────0 B (00.00%) — mjit-data
│ │ ├──────────0 B (00.00%) — tjit-code
│ │ └──────────0 B (00.00%) — tjit-data
│ │ ├──0 B (00.00%) — allocators-main
│ │ └──0 B (00.00%) — allocators-reserve
│ ├─────734,784 B (00.61%) — gc-heap-chunk-admin
│ ├─────262,144 B (00.22%) — stack
│ ├─────146,932 B (00.12%) — compartment(about:mozilla)
│ │ ├──135,168 B (00.11%) — gc-heap
│ │ │ ├───55,528 B (00.05%) — objects
│ │ │ ├───52,400 B (00.04%) — shapes
│ │ │ ├───26,544 B (00.02%) — arena-unused
│ │ │ ├──────528 B (00.00%) — arena-headers
│ │ │ ├──────168 B (00.00%) — arena-padding
│ │ │ ├────────0 B (00.00%) — strings
│ │ │ └────────0 B (00.00%) — xml
│ │ ├───11,536 B (00.01%) — object-slots
│ │ ├──────228 B (00.00%) — scripts
│ │ ├────────0 B (00.00%) — string-chars
│ │ ├────────0 B (00.00%) — mjit-code
│ │ ├────────0 B (00.00%) — mjit-data
│ │ ├────────0 B (00.00%) — tjit-code
│ │ └────────0 B (00.00%) — tjit-data
│ │ ├──0 B (00.00%) — allocators-main
│ │ └──0 B (00.00%) — allocators-reserve
│ ├─────130,462 B (00.11%) — compartment(about:blank)
│ │ ├──122,880 B (00.10%) — gc-heap
│ │ │ ├───56,432 B (00.05%) — arena-unused
│ │ │ ├───35,320 B (00.03%) — shapes
│ │ │ ├───30,408 B (00.03%) — objects
│ │ │ ├──────480 B (00.00%) — arena-headers
│ │ │ ├──────240 B (00.00%) — arena-padding
│ │ │ ├────────0 B (00.00%) — strings
│ │ │ └────────0 B (00.00%) — xml
│ │ ├────7,240 B (00.01%) — object-slots
│ │ ├──────342 B (00.00%) — scripts
│ │ ├────────0 B (00.00%) — string-chars
│ │ ├────────0 B (00.00%) — mjit-code
│ │ ├────────0 B (00.00%) — mjit-data
│ │ ├────────0 B (00.00%) — tjit-code
│ │ └────────0 B (00.00%) — tjit-data
│ │ ├──0 B (00.00%) — allocators-main
│ │ └──0 B (00.00%) — allocators-reserve
│ ├──────23,026 B (00.02%) — compartment(about:)
│ │ ├──20,480 B (00.02%) — gc-heap
│ │ │ ├──14,440 B (00.01%) — arena-unused
│ │ │ ├───3,592 B (00.00%) — objects
│ │ │ ├───2,320 B (00.00%) — shapes
│ │ │ ├──────80 B (00.00%) — arena-headers
│ │ │ ├──────48 B (00.00%) — arena-padding
│ │ │ ├───────0 B (00.00%) — strings
│ │ │ └───────0 B (00.00%) — xml
│ │ ├───2,432 B (00.00%) — object-slots
│ │ ├─────114 B (00.00%) — scripts
│ │ ├───────0 B (00.00%) — string-chars
│ │ ├───────0 B (00.00%) — mjit-code
│ │ ├───────0 B (00.00%) — mjit-data
│ │ ├───────0 B (00.00%) — tjit-code
│ │ └───────0 B (00.00%) — tjit-data
│ │ ├──0 B (00.00%) — allocators-main
│ │ └──0 B (00.00%) — allocators-reserve
│ ├──────22,962 B (00.02%) — compartment(moz-nullprincipal:{2213d3b1-d9c6-4bc7-93cc-5529c751425e})
│ │ ├──20,480 B (00.02%) — gc-heap
│ │ │ ├──15,040 B (00.01%) — arena-unused
│ │ │ ├───3,192 B (00.00%) — objects
│ │ │ ├───2,120 B (00.00%) — shapes
│ │ │ ├──────80 B (00.00%) — arena-headers
│ │ │ ├──────48 B (00.00%) — arena-padding
│ │ │ ├───────0 B (00.00%) — strings
│ │ │ └───────0 B (00.00%) — xml
│ │ ├───2,368 B (00.00%) — object-slots
│ │ ├─────114 B (00.00%) — scripts
│ │ ├───────0 B (00.00%) — string-chars
│ │ ├───────0 B (00.00%) — mjit-code
│ │ ├───────0 B (00.00%) — mjit-data
│ │ ├───────0 B (00.00%) — tjit-code
│ │ └───────0 B (00.00%) — tjit-data
│ │ ├──0 B (00.00%) — allocators-main
│ │ └──0 B (00.00%) — allocators-reserve
│ └───────────0 B (00.00%) — compartment(moz-nullprincipal:{3347ecff-47e1-4a44-ba9b-f324923e5513})
│ ├──0 B (00.00%) — gc-heap
│ │ ├──0 B (00.00%) — arena-headers
│ │ ├──0 B (00.00%) — arena-padding
│ │ ├──0 B (00.00%) — arena-unused
│ │ ├──0 B (00.00%) — objects
│ │ ├──0 B (00.00%) — strings
│ │ ├──0 B (00.00%) — shapes
│ │ └──0 B (00.00%) — xml
│ ├──0 B (00.00%) — object-slots
│ ├──0 B (00.00%) — string-chars
│ ├──0 B (00.00%) — scripts
│ ├──0 B (00.00%) — mjit-code
│ ├──0 B (00.00%) — mjit-data
│ ├──0 B (00.00%) — tjit-code
│ └──0 B (00.00%) — tjit-data
│ ├──0 B (00.00%) — allocators-main
│ └──0 B (00.00%) — allocators-reserve
├───46,287,371 B (38.40%) — heap-unclassified
├───12,325,584 B (10.23%) — storage
│ └──12,325,584 B (10.23%) — sqlite
│ ├───9,681,408 B (08.03%) — places.sqlite
│ │ ├──9,535,632 B (07.91%) — cache-used
│ │ ├─────97,864 B (00.08%) — stmt-used
│ │ └─────47,912 B (00.04%) — schema-used
│ ├─────715,168 B (00.59%) — other
│ ├─────559,096 B (00.46%) — cookies.sqlite
│ │ ├──557,296 B (00.46%) — cache-used
│ │ ├────1,800 B (00.00%) — schema-used
│ │ └────────0 B (00.00%) — stmt-used
│ ├─────449,616 B (00.37%) — urlclassifier3.sqlite
│ │ ├──362,456 B (00.30%) — cache-used
│ │ ├───84,496 B (00.07%) — stmt-used
│ │ └────2,664 B (00.00%) — schema-used
│ ├─────445,760 B (00.37%) — extensions.sqlite
│ │ ├──362,448 B (00.30%) — cache-used
│ │ ├───76,416 B (00.06%) — stmt-used
│ │ └────6,896 B (00.01%) — schema-used
│ ├─────330,048 B (00.27%) — addons.sqlite
│ │ ├──296,624 B (00.25%) — cache-used
│ │ ├───29,152 B (00.02%) — stmt-used
│ │ └────4,272 B (00.00%) — schema-used
│ ├──────67,976 B (00.06%) — formhistory.sqlite
│ │ ├──66,312 B (00.06%) — cache-used
│ │ ├───1,664 B (00.00%) — schema-used
│ │ └───────0 B (00.00%) — stmt-used
│ ├──────28,712 B (00.02%) — signons.sqlite
│ │ ├──13,256 B (00.01%) — cache-used
│ │ ├──12,624 B (00.01%) — stmt-used
│ │ └───2,832 B (00.00%) — schema-used
│ ├──────12,816 B (00.01%) — search.sqlite
│ │ ├───9,768 B (00.01%) — cache-used
│ │ ├───1,840 B (00.00%) — stmt-used
│ │ └───1,208 B (00.00%) — schema-used
│ ├──────12,248 B (00.01%) — content-prefs.sqlite
│ │ ├───5,856 B (00.00%) — stmt-used
│ │ ├───3,992 B (00.00%) — cache-used
│ │ └───2,400 B (00.00%) — schema-used
│ ├──────11,744 B (00.01%) — downloads.sqlite
│ │ ├───5,984 B (00.00%) — stmt-used
│ │ ├───3,976 B (00.00%) — cache-used
│ │ └───1,784 B (00.00%) — schema-used
│ └──────10,992 B (00.01%) — permissions.sqlite
│ ├───5,744 B (00.00%) — stmt-used
│ ├───3,984 B (00.00%) — cache-used
│ └───1,264 B (00.00%) — schema-used
├──────921,744 B (00.76%) — xpti-working-set
├──────681,422 B (00.57%) — layout
│ ├──681,422 B (00.57%) — all
│ └────────0 B (00.00%) — bidi
└──────252,807 B (00.21%) — images
├──246,248 B (00.20%) — chrome
│ ├──243,840 B (00.20%) — used
│ │ ├──243,840 B (00.20%) — uncompressed
│ │ └────────0 B (00.00%) — raw
│ └────2,408 B (00.00%) — unused
│ ├──2,408 B (00.00%) — uncompressed
│ └──────0 B (00.00%) — raw
└────6,559 B (00.01%) — content
├──6,559 B (00.01%) — used
│ ├──4,288 B (00.00%) — uncompressed
│ └──2,271 B (00.00%) — raw
└──────0 B (00.00%) — unused
├──0 B (00.00%) — raw
└──0 B (00.00%) — uncompressed

Other Measurements
437,170,176 B — vsize
184,647,680 B — resident
171,790,336 B — private
126,095,360 B — heap-committed
118,823,206 B — heap-used
45,088,768 B — js-gc-heap
22,733,608 B — heap-unused
11,307,396 B — gfx-d2d-surfacevram
2,453,504 B — heap-dirty
255,620 B — gfx-surface-image
140,452 B — gfx-d2d-surfacecache
0 B — gfx-surface-win32

Comments are closed.