Categories
about:memory Firefox Memory consumption MemShrink

MemShrink progress, week 7

Lots of good stuff this week.

Blog posts

There were four great blog posts this week relating to memory usage in Firefox.

Web workers

Ben Turner landed a complete reworking of web workers.  This fixed three MemShrink bugs, and probably a number of others.  Ben also added memory reporters so that web workers show up in about:memory.

Add-ons

I always cringe when someone files a bug report complaining about Firefox’s memory usage and they have many (e.g. 10, 20 or more) add-ons installed.  The chances of all of those add-ons behaving themselves is pretty low, unfortunately.  For example, one user found that the CyberSearch 2.0.8 add-on causes the places SQLite database to grow to over 300MB.  Another user found that one or more add-ons caused that same database to grow to 1.2GB(!);  the add-on(s) responsible has not yet been identified.

This kind of thing is terrible for perceptions of Firefox.  Maybe limiting the size of places.sqlite will help?

In better add-on-related news, Jan Honza Odvarko fixed a bad memory leak in Firebug, one that was causing zombie compartments on pages where Firebug hadn’t even been enabled.  Now Firebug only causes zombie compartments for pages where Firebug has been enabled, but Steve Fink is making progress there, as he described in his blog post that I linked to above.

Two other add-ons known to cause memory problems are Lastpass and It’s All Text.

Miscellaneous

Andrew McCreight is making great progress on his static analysis to find cycle collection leaks.  This week he found a problem in nsDOMEventTargetWrapperCache, which Olli Pettay then fixed.

If you ask a heap allocator like jemalloc for a certain number of bytes, it’ll often round that request size up to a number such as a power-of-two.  For example, if you ask for 99 bytes it’ll give you 112 or 128.  I found that this is contributing to the high “heap-unclassified” number in about:memory.  I also stumbled across two cases where Firefox itself deliberately requests a power-of-two-sized block of memory (with the aim of avoiding round-up in the heap allocator) but botches the calculation such that it asks for a block slightly bigger than a power-of-two, which jemalloc then rounds up again.  Luke Wagner fixed the first of these, and I have patches awaiting review to fix the second.

Finally, I should mention bug 658738 and its children.  Dão Gottwald and others have been working tirelessly to fix leaked windows in browser-chrome tests.  This has involved both (a) fixing problems with the tests, and (b) fixing problems with the browser.  Eleven child bugs have already been fixed, and there are four still open.

Bug counts

Here’s the change in MemShrink bug counts.

  • P1: 27 (+1)
  • P2: 53 (+4)
  • P3: 35 (+2)
  • P4: 11 (+5)

Increases all across the board.  Once again I think this reflects the increasing amount of work being done… but I keep saying that.  Therefore, beginning next week, I’ll show how many bugs in each category were fixed and how many were added, e.g. “P1: 27 (-2, +3)”.  That’ll give a better idea of progress.  (I can’t start it until next week because I’ve only today started recording enough info about the open bugs to get those numbers.  And before you suggest that I use a time-based Bugzilla search instead, the problem with that is that I write this progress report at a slightly different time each week, so if I just search during the last week I may double-count or fail to count some bugs.)

21 replies on “MemShrink progress, week 7”

I wonder if anyone has tried a 150-window stress test on Firefox and other browsers? Last time I checked, Firefox used crazy amounts of resources on a per-window basis, and nobody had looked at it because most people use tabs. But if you use a tabbed window manager, you really want to use top-level windows and disable tabs.

Congratz on the great progress. I have previously posted on Mozillazine with similar test. Although i was running slightly less with 135 Tabs only.

Firefox 6, performed really well. With little to no UI sluggish feeling. While Chrome perform… well it was usable, apart from its silly microscopic Tabs.

Firefox 7 made a huge difference, it didn’t feel slow at all. Memory usage was much lower then Chrome. And the whole browsers and webpage are still very responsive.

Looking at all those fixes coming in, Firefox 8 looks to be another good release. ( Firefox 7 will be a great release due to its dramatic improvement ) I just hope we have some additional JS and DOM performance boost to go with it.

And I think something need to be done to those Addon. It is damaging Firefox’s reputation. I think we need to set some restriction to prevent further damage.

How realistically is it that Firefox’ more efficient memory behavior will remain once Electrolysis is turned on by default?

> if I just search during the last week I may
> double-count or fail to count some bugs

If you have a fixed “last week” search. But you could record the date of your last search each week, and tweak the search to search between that time and “now” each time.

Hi Nicholas,

thanks for your hard work to make Firefox better 🙂

Addons have indeed a large impact on the memory usage of the Firefox. Adblock Plus and NoScript use a large amount of RAM, but I can’t live without them.

Here are the results of the latest Aurora:

all addons activated:

Explicit Allocations
117.65 MB (100.0%) — explicit
+—55.87 MB (47.49%) — js
¦ +–31.17 MB (26.50%) — compartment([System Principal])
¦ ¦ +–18.14 MB (15.42%) — gc-heap
¦ ¦ ¦ +—8.64 MB (07.34%) — objects
¦ ¦ ¦ +—5.03 MB (04.27%) — shapes
¦ ¦ ¦ +—2.91 MB (02.47%) — arena-unused
¦ ¦ ¦ +—1.43 MB (01.22%) — strings
¦ ¦ ¦ +—0.13 MB (00.11%) — (3 omitted)
¦ ¦ +—4.26 MB (03.62%) — string-chars
¦ ¦ +—3.53 MB (03.00%) — scripts
¦ ¦ +—2.56 MB (02.18%) — object-slots
¦ ¦ +—2.06 MB (01.75%) — mjit-code
¦ ¦ +—0.62 MB (00.53%) — (3 omitted)
¦ +–19.33 MB (16.43%) — gc-heap-chunk-unused
¦ +—4.24 MB (03.60%) — compartment(atoms)
¦ ¦ +–2.69 MB (02.28%) — string-chars
¦ ¦ +–1.55 MB (01.32%) — gc-heap
¦ ¦ ¦ +–1.53 MB (01.30%) — strings
¦ ¦ ¦ +–0.02 MB (00.01%) — (6 omitted)
¦ ¦ +–0.00 MB (00.00%) — (6 omitted)
¦ +—0.65 MB (00.55%) — gc-heap-chunk-admin
¦ +—0.48 MB (00.41%) — (5 omitted)
+—47.11 MB (40.04%) — heap-unclassified
+—12.18 MB (10.35%) — storage
¦ +–12.18 MB (10.35%) — sqlite
¦ +—9.63 MB (08.19%) — places.sqlite
¦ ¦ +–9.44 MB (08.02%) — cache-used
¦ ¦ +–0.20 MB (00.17%) — (2 omitted)
¦ +—1.88 MB (01.60%) — (10 omitted)
¦ +—0.66 MB (00.56%) — other
+—-1.11 MB (00.94%) — layout
¦ +–1.11 MB (00.94%) — all
¦ +–0.00 MB (00.00%) — (1 omitted)
+—-0.88 MB (00.75%) — xpti-working-set
+—-0.50 MB (00.42%) — (1 omitted)

Other Measurements
410.23 MB — vsize
181.14 MB — resident
165.97 MB — private
120.75 MB — heap-committed
115.46 MB — heap-used
40.00 MB — js-gc-heap
11.28 MB — gfx-d2d-surfacevram
8.54 MB — heap-unused
0.54 MB — heap-dirty
0.50 MB — gfx-surface-image
0.28 MB — gfx-d2d-surfacecache
0.24 MB — canvas-2d-pixel-bytes
0.00 MB — gfx-surface-win32

without ABP:

Main Process

Explicit Allocations
66.75 MB (100.0%) — explicit
+–33.85 MB (50.70%) — heap-unclassified
+–28.87 MB (43.25%) — js
¦ +–24.41 MB (36.57%) — compartment([System Principal])
¦ ¦ +–15.04 MB (22.54%) — gc-heap
¦ ¦ ¦ +—7.42 MB (11.11%) — objects
¦ ¦ ¦ +—5.44 MB (08.15%) — shapes
¦ ¦ ¦ +—2.06 MB (03.09%) — strings
¦ ¦ ¦ +—0.12 MB (00.18%) — (4 omitted)
¦ ¦ +—3.81 MB (05.71%) — string-chars
¦ ¦ +—2.78 MB (04.16%) — scripts
¦ ¦ +—1.61 MB (02.41%) — object-slots
¦ ¦ +—0.75 MB (01.12%) — mjit-code
¦ ¦ +—0.41 MB (00.62%) — (3 omitted)
¦ +—2.09 MB (03.13%) — compartment(atoms)
¦ ¦ +–1.36 MB (02.03%) — string-chars
¦ ¦ +–0.73 MB (01.10%) — gc-heap
¦ ¦ ¦ +–0.73 MB (01.09%) — strings
¦ ¦ ¦ +–0.01 MB (00.01%) — (6 omitted)
¦ ¦ +–0.00 MB (00.00%) — (6 omitted)
¦ +—1.66 MB (02.49%) — gc-heap-chunk-unused
¦ +—0.71 MB (01.07%) — (4 omitted)
+—2.34 MB (03.51%) — storage
¦ +–2.34 MB (03.51%) — sqlite
¦ +–0.54 MB (00.82%) — cookies.sqlite
¦ ¦ +–0.54 MB (00.81%) — cache-used
¦ ¦ +–0.00 MB (00.00%) — (2 omitted)
¦ +–0.51 MB (00.77%) — other
¦ +–0.46 MB (00.69%) — extensions.sqlite
¦ ¦ +–0.35 MB (00.52%) — cache-used
¦ ¦ +–0.11 MB (00.17%) — (2 omitted)
¦ +–0.41 MB (00.62%) — (5 omitted)
¦ +–0.41 MB (00.62%) — places.sqlite
¦ +–0.34 MB (00.50%) — cache-used
¦ +–0.08 MB (00.11%) — (2 omitted)
+—0.88 MB (01.32%) — xpti-working-set
+—0.60 MB (00.90%) — layout
¦ +–0.60 MB (00.90%) — all
¦ +–0.00 MB (00.00%) — (1 omitted)
+—0.20 MB (00.31%) — (1 omitted)

Other Measurements
328.78 MB — vsize
114.57 MB — resident
107.24 MB — private
67.16 MB — heap-committed
65.75 MB — heap-used
18.00 MB — js-gc-heap
10.85 MB — gfx-d2d-surfacevram
4.25 MB — heap-unused
0.24 MB — canvas-2d-pixel-bytes
0.21 MB — gfx-surface-image
0.07 MB — gfx-d2d-surfacecache
0.01 MB — heap-dirty
0.00 MB — gfx-surface-win32

without ABP and NoScript:

Main Process

Explicit Allocations
43.64 MB (100.0%) — explicit
+–23.31 MB (53.42%) — heap-unclassified
+–16.31 MB (37.38%) — js
¦ +–12.70 MB (29.09%) — compartment([System Principal])
¦ ¦ +—6.55 MB (15.01%) — gc-heap
¦ ¦ ¦ +–3.74 MB (08.56%) — objects
¦ ¦ ¦ +–2.55 MB (05.85%) — shapes
¦ ¦ ¦ +–0.26 MB (00.60%) — (5 omitted)
¦ ¦ +—2.55 MB (05.85%) — scripts
¦ ¦ +—1.65 MB (03.78%) — string-chars
¦ ¦ +—0.84 MB (01.93%) — object-slots
¦ ¦ +—0.69 MB (01.58%) — mjit-code
¦ ¦ +—0.41 MB (00.95%) — (3 omitted)
¦ +—1.55 MB (03.55%) — compartment(atoms)
¦ ¦ +–1.02 MB (02.33%) — string-chars
¦ ¦ +–0.54 MB (01.23%) — gc-heap
¦ ¦ ¦ +–0.53 MB (01.21%) — strings
¦ ¦ ¦ +–0.01 MB (00.02%) — (6 omitted)
¦ ¦ +–0.00 MB (00.00%) — (6 omitted)
¦ +—1.50 MB (03.43%) — gc-heap-chunk-unused
¦ +—0.29 MB (00.67%) — (3 omitted)
¦ +—0.27 MB (00.62%) — compartment(about:blank)
¦ +–0.25 MB (00.57%) — gc-heap
¦ ¦ +–0.25 MB (00.57%) — (7 omitted)
¦ +–0.02 MB (00.05%) — (7 omitted)
+—2.34 MB (05.37%) — storage
¦ +–2.34 MB (05.37%) — sqlite
¦ +–0.54 MB (01.25%) — cookies.sqlite
¦ ¦ +–0.54 MB (01.24%) — cache-used
¦ ¦ +–0.00 MB (00.00%) — (2 omitted)
¦ +–0.51 MB (01.18%) — other
¦ +–0.46 MB (01.06%) — extensions.sqlite
¦ ¦ +–0.35 MB (00.79%) — cache-used
¦ ¦ +–0.11 MB (00.26%) — (2 omitted)
¦ +–0.41 MB (00.94%) — places.sqlite
¦ ¦ +–0.34 MB (00.77%) — cache-used
¦ ¦ +–0.08 MB (00.18%) — (2 omitted)
¦ +–0.31 MB (00.72%) — addons.sqlite
¦ ¦ +–0.28 MB (00.65%) — cache-used
¦ ¦ +–0.03 MB (00.07%) — (2 omitted)
¦ +–0.10 MB (00.23%) — (4 omitted)
+—0.88 MB (02.02%) — xpti-working-set
+—0.59 MB (01.35%) — layout
¦ +–0.59 MB (01.35%) — all
¦ +–0.00 MB (00.00%) — (1 omitted)
+—0.20 MB (00.46%) — (1 omitted)

Other Measurements
305.23 MB — vsize
91.46 MB — resident
84.46 MB — private
43.80 MB — heap-committed
42.71 MB — heap-used
10.85 MB — gfx-d2d-surfacevram
9.00 MB — js-gc-heap
3.29 MB — heap-unused
0.24 MB — canvas-2d-pixel-bytes
0.23 MB — heap-dirty
0.21 MB — gfx-surface-image
0.09 MB — gfx-d2d-surfacecache
0.00 MB — gfx-surface-win32

I always had 2 tabs open (about:memory and the addon manager to have the option to restart FF after disabling 1 addon).

André

André: add-ons using memory isn’t a problem per se; if an add-on is providing useful functionality it’s reasonable that it takes up resources, and I think ABP and NoScript are both well written. I use ABP myself.

The problem is if an add-on starts using unreasonable amounts of memory or leaking, as CyberSearch and It’s All Text are doing. (I used to use It’s All Text myself, but I disabled it when I realized it had a leak; it was moderately useful, but not useful enough for me to tolerate leaks.)

By Post Author

The leak is in Firefox, but IAT triggers it.

I did, once, have a huge leak in it. But I’ve used the newer version for long browser sessions (many tabs, windows, and for days) with no problems. That being said, it seems like something (probably Firebug being enabled) can make it much much worse.

The large-number-of-tabs problem in Chrome has been known for a while from all I know, and what I heard about it is that it’s the process separation that causes this. A separate process for every tab has its toll.

Based on what you’ve seen, can you point out some specific things that add-ons do that cause these kinds of problems?

I’m guessing most add-on developers are clueless that they are even causing these problems.

Mike: Unfortunately I’m kind of a Gecko newbie, and I know even less about add-ons, so I don’t have any good answers, sorry. I just do low-level analysis to find problems and then let other people diagnose and fix them 🙂

By Post Author

Hey njn, I was reading your description on:

https://bugzilla.mozilla.org/show_bug.cgi?id=675132

and then I remembered something. Metrowerks’ C++ has an allocator that can return the memory to you that it would have used as padding. e.g.:

void * malloc_more(size_t *size, size_t block_alignment = 1);

Precondition: *size is positive and divisible by block_alignment.

If allocation succeeds, malloc_more will possibly adjust *size to be a larger multiple of block_alignment.

Useful for small gains that are trivial to code in string/vector/buffer classes. You may want to add something like it to jemalloc.

asdf: Thanks for the info. jemalloc has the quasi-standard malloc_usable_size. You pass it a malloc-allocated pointer and it tells you how much it actually allocated. So that gives much the same effect, AFAICT.

However, in cases where we have flexibility over how much we can request (strings, vectors, etc), it’s simpler just to ensure that all requests are power-of-two-sized, because in practice such requests never get rounded up. Then you don’t have to use malloc_usable_size, which is a win because it’s not available on all platforms.

By Post Author

It’s a real shame to hear that it’s all text has a troublesome memory leak. It was really useful from an accessibility point of view, as your editor could incorporate functionality that simply isn’t, and wouldn’t be considered for addition, to the browser’s native text widget.

Oh well, I suppose it’s back to cut and paste for me. (Oh how I wish something like the Oberon OS with it’s OS-provided text widget [hence one customisation hack became avaialable everywhere] inherited by all programs had taken off…)

Do you use firebug or anything similar? It looks like the memory problem in Firefox that IAT triggers can be made worse if Firebug is installed.

On the plus side, I’m working on version 2.0 that has minimal code in the browser and the rest in a java application that manages the editor. This will make IAT work on many more browsers and is guaranteed to remove these particular problems.

If you’re interested in doing user-testing, I’d be happy for the help. I won’t be ready for another month or so, but if you’re interested, contact me at http://docwhat.org/email

I’m particularly worried that adding Java as a requirement will be too big a burden…though Mac OS X will auto-install it for you and I’ll wrap the windows binary in an EXE that’ll download Java for you if it’s needed.

The UI (preferences, etc.) will all be done via the browser.

Ciao!

Unfortunately, Firefox’ reputation is staked to the quality of its addons, for better or worse. Maybe Jetpack will help in that area.

However, a more formal Mozilla-assisted addon QA effort could go a long way as well. Or at least looking for common memory-hogging patterns during AMO review? Or even an automated memory test on upload that reports the results to the author with a link for help on reducing footprint?

Unfortunately, I suspect JetPack by itself won’t fix these problems. If you store an element from a page into the background page, it will still hold the page’s memory hostage unless Firefox allows that element to be reaped despite a reference in the extension.

Also, Mozilla lending QA resources to the addon community is also something that would really make them stand out as other browsers continue improving their own addon support.

I have noticed a problem with ver 4.x-8.x nightly builds. I have no idea how to report the problem officially.

It involves flash video and happens with the 10.x and 11.x versions of flash.

After you close a tab with a flash movie and if the browser later on hangs you see the flash movie is still there. It looks as if it never was unloaded.

It has gotten better recently in the nightly builds but is still there.

The hangs are a result of my usage of way to many tabs.

I’m on Vista sp1 laptop with a NVidia 8600GS with up to date driver. 2GB of system memory 256mb of video mem.

You may want to compare Chrome on Linux and Opera on Windows for the 150 tab test.

Comments are closed.