Categories
add-ons Firefox MemShrink

MemShrink progress, weeks 28–29

It’s been a quiet couple of weeks, due to Christmas, New Year, and all that.

Two leaks in add-ons were fixed.

  • cyberscorpio fixed a zombie compartment in the Super Start add-on.  The problem was that the add-on was failing to remove an event observer when a page is unloaded.  The fix will be present in v3.6.2.
  • Jesse Hakanen fixed a zombie compartment in the AdBlock Plus Pop-up Addon add-on (note that this is add-on extends the blocking functionality of AdBlock Plus, it’s not AdBlock Plus itself).  The problem was that the add-on was failing to catch the page unload event, and so some references to closed pages were left in memory.  The fix is now available in v0.3.

Because both leaks had similar causes, I updated the documentation on avoiding zombie compartments in add-ons accordingly.

Patches for two other MemShrink bugs landed.

I saw two noteworthy mentions of Firefox’s recent reductions in memory consumption. The first was in webmonkey.com’s review of Firefox 9.

Alongside the faster JavaScript processing Firefox 9 continues to show improvements from Mozilla’s MemShrink project, an ongoing effort to reduce memory usage in the browser. Indeed, for the first time in a very long time my testing showed Firefox 9 using less memory than Opera (which has long been the least RAM-hungry browser I test). Opening the same dozen tabs in both Firefox and Opera used only 367MB of RAM in Firefox compared to 378MB in Opera 11.60 [Update: Note that the memory test was performed with the following Firefox add-ons running: AdBlock, Ghostery, BetterPrivacy and HTTPS-Everywhere.] There’s no longer much difference between the two, which is a testament to Firefox’s dramatic improvement over the last six months of MemShrink efforts.

Cross-browser memory comparisons are fraught with difficulties, both in the measuring and the interpreting, so I don’t give the comparison with Opera much weight.  But the fact that this author has seen Firefox’s own memory consumption drop over the last few versions is meaningful.

The second was a comment on the previous MemShrink progress report from AC about Firefox 10.

You may remember that I posted a few weeks ago saying that Firefox 9 beta was great and that I could definitely see more memory and performance improvements over Fire 7 and 8. Well I am pleased to say that after a few days of using Firefox 10 beta 1, I can clearly see more memory improvements again. There is an obvious difference between Firefox 9 and Firefox 10 on the beta channel.

I have no hard evidence as such other than saying that nothing has changed on my laptop in the last few weeks but I am using the same heavy tab load and the memory usage displayed by Windows task manager is lower than it was by about 20 MB and the responsiveness and speed feels sharp.

Whatever you guys are doing with the Memshrink project, keep doing it because the results are paying dividends.

I’ve read literally 1000s of complaints about Firefox’s memory consumption online, so it’s very encouraging to hear compliments.

Here are the current bug counts.

  • P1: 28 (-0/+2)
  • P2: 143 (-1/+5)
  • P3: 64 (-1/+1)
  • Unprioritized: 0 (-0/+0)

No big changes, unsurprisingly.  [Update: the P2 numbers were incorrect and have been fixed.]

42 replies on “MemShrink progress, weeks 28–29”

Are the numbers flipped for the P2 memshrink bug count changes? Keep up the great work and posts!

They weren’t so much flipped as completely wrong. I’ve fixed it, thanks for catching!

By Post Author

Nicholas, are you disappointed that the stagnant rate of P1 bug fixes? I realize some bugs are being fixed but new ones found however MemShrink has been running for over six months now. Did you hope for better progress, expect it? What about the next six months, do you and/or Mozilla have any goals for how many P1 bugs should be fixed? Whilst the total might not drop, would it, for example, be reasonable to expect around half (14) P1 bugs to be fixed in the next six months?

Also, the P2 total seems fairly high Is there a goal/timeframe for reducing that list markedly?

I’ve been really impressed with the progress since this started. However, I wonder if the OS X version couldn’t use it’s own team. I think the way OS X handles memory it makes memory usage much more visible than on Windows or Linux.

Nicholas, Happy New Year! Thank you for all your hard work on MemShrink! I can definitely see some improvements from Firefox 7 to Firefox 8 and then again from Firefox 8 to Firefox 9. Keep up the great work! I use the Memory Restart add-on to easily monitor how much memory Firefox is using.

Peter

I was wondering, why aren’t plugin-container.exe processes closed? This is the only page I have had open for a while now, yet I have 4 plugin containers.

Please see here: http://img1.uploadscreenshot.com/images/orig/1/307420064-orig.jpg

It is true that these processes, once opened, never get closed? Is the reason for having 4 that I use 4 plugins regularly? (Flash, Adobe Reader, Java, CDF Player) If they really never get closed, is it planned to change this in the future?

BTW Firefox 9 really is slimmer than previous versions. I had the browser open all day (though didn’t use it a lot). The memory use was a bit below 500 MB even with a single tab. After pressing the “Minimize mem usage” button, it shrinked to what you see on the screenshot. In Fx7 it would have stayed above or very close to 400.

Hi Nicholas,

I’m back with another unofficial and completely anecdotal report on the memory performance of Firefox.

My laptop is Windows 7 with 4GB RAM and a dual core 2.3GHz Intel processor.

I am on the beta channel, using 10b2. I have a British English Dictionary and Test Pilot installed as add-ons and that is it. I don’t count myself as having add-ons.

I opened the vast majority of my bookmarks by using the “Open in all tabs” function. Memory usage quickly rose to 865,000 kb as there were about 92 tabs open.

Interestingly surfing with this many tabs open was no different. They were quite slow to load, but that’s because I was opening them by the folder i.e. opening groups of 20 or 30 tabs in the same second. Once they were all open and loaded it was fine. Browsing/navigating/surfing whatever you want to call it was smooth with all these tabs open. It wasn’t sluggish at all.

I typed this post, went and made a cup of tea and talked to my friend for 5 minutes then watched some telly. The tabs were open for about 15 minutes. Memory usage hadn’t moved. It was still between 860,000 and 870,000 kb.

I then made to changes, I logged in to Facebook and Linked In and then browsed through a few pages on those sites. The memory usage rose to 900,000 to 920,000 kb. After another 5 minutes this had dropped to 866,000 kb. Impressive. With a massively heavy tab load

By this point the tabs had been open for about 20 minutes. I then got the stopwatch function ready on my phone. Finally I went to the tabs that was displaying Google.co.uk and right clicked on it.

I then simultaneously clicked “Close Other Tabs” and started the stopwatch function.

Within 45 seconds the memory usage had settled at about 225,000 kb with just one tab displaying Google UK. I waited another 5 minutes and the memory usage had stayed at 225,000 kb.

So it seems that Firefox is pretty good at releasing the memory once tabs are closed.

Another test. Very similar. Opened about 100 tabs and the memory settled comfortably at 963,000 after 5 minutes. I then closed 52 of the tabs, leaving me with 48. When I walked away from the laptop the memory was on 728,000 kb. When I came back 25 minutes later it was on 734,000 kb. An increase of less than 6 MB when idle for nearly half an hour is also pretty good.

So all in all, I’m very happy with Firefox 10 b2.

I have come to the conclusion that if people complain about memory usage it’s one of the following reasons:

Their PC/laptop is screwed, not Firefox.

They need a new profile as the current one is on it’s last legs.

The memory leak stem from add-ons, not Firefox itself.

They are spreading rumours and have no evidence to back it up.

They are bitter because their experience with Firefox 4, 5 or 6 was awful and they haven’t used Firefox since and so haven’t seen how far Firefox has come in the last 6 – 9 months.

Project Snappy now needs to return a similar level of results to Memshrink and then nobody will have anything to complain about.

Keep up the good work!

In about:memory: Is it normal for the javascript compartments for every tab I’ve opened and closed during the session to stay around until I restart Firefox?

After a bunch of experimentation it seems to be caused by Firebug. Getting rid of it, YSlow and Google Page Speed fixed the problem.

Firebug’s known to leak badly, and I’ve heard it’s developer is stumped and hoping improved memory reporters will be able to help him track it down eventually (the lack of recent activity on the bug tracker supports this).

The lack of recent activity supports the reality that Mozilla have essentially abandoned Firebug in favour of native developer tools. They will not admit it. They say they’re supporting Firebug by having one sole developer working on it full time, Jan ‘Honza’ Odvarko. This might be better than letting it fall into virtual bitrot when Hewitt moved on, but it’s not enough and regardless, the zombie compartments Firebug causes can surely be investigated by any Mozilla-familiar developer. Additionally MemShrink has been in place for over 6 months and as yet has unfortunately been unable to solve the issue.

Anecdotally one of the biggest early-to-midlife supporters of Firefox were web developers who desperately needed a better browser than IE. Now those same key evangelists are jumping to Chrome because MemShrink started years too late and hence Firefox has an awful reputation for bloat, a reputation that is still well and truly deserved amongst those same web developers who want to evangelize Firefox but need Firebug and are therefore left with an awfully degenerated Firefox experience. Is it any wonder that Chrome has grown so fast?

Mozilla can sit back and opine about not having the resources of the big three however it’s decisions like abandoning Firebug that have a big impact on a substantial proportion of their previous support base – web developers. It shouldn’t take this long to figure out why having Firebug installed – but not even in use – causes seemingly every tab to be a zombie compartment.

I have a question that has been nagging me for a bit. Should the value of ‘resident’ minus ‘explicit’ be roughly constant? My thinking here is that ‘resident’ is the whole amount of used memory and ‘explicit’ is everything that can be accounted for by various allocators, so shouldn’t the difference be things like .text and .rdata, which are of fixed size?

I ask because in my case the gap between these is nowhere near constant. I just tried an experiment where I started Fx 9.0.1 with nothing open but about:memory, and that resulted in

explicit=78.23 resident=169.03 resident-explicit=90.8

Then I went about my business using the browser for a bit. After maybe 30 minutes, and with less than a dozen tabs open, I now see

explicit=316.34 resident=475.90 resident-explicit=159.56

It seems that the ‘resident’ is increasing faster than ‘explicit’. Is this normal or is something wrong? What exactly does the difference between them include?

Good question!

What you’re describing is probably heap fragmentation. If you malloc() a million small objects and then free 95% of them at random, you won’t get 95% of your memory back. The allocator can’t give a page back to the operating system if there are any active allocations on it, and it’s likely that there are a bunch of almost-empty pages sitting around.

The number to look at is heap-committed. The difference between heap-committed and heap-allocated is heap fragmentation. (I think heap-committed shows up on Mac 10.6 and higher, Linux, and Windows.)

The difference between resident and explicit + heap fragmentation should be roughly constant. I think. 🙂

I figured something like that was probably the case. However, here are the current figures for the same browser session as the previous comment:

heap-allocated=209.24
heap-committed=228.27
explicit=322.14
resident=542.57
resident – (explicit + (heap-committed – heap-allocated)) = 201.4

201.4 is still a lot more than the 90.8 right after startup when there’s very little fragmentation.

This machine has 16GB of memory and so it really doesn’t matter too much to me how much Firefox uses, but I’m still curious why this gap widens so.

“resident” is determined by the OS. “explicit” is determined by Firefox. “heap-committed” and “heap-allocated” are determined by jemalloc, which is the allocator used by Firefox (and which is more-or-less a black box to Firefox).

I’m unconvinced that these numbers are comparable in any meaningful way, so I don’t worry about this much 🙂

By Post Author

Could you explain what is the difference between about:memory’s heap-committed measurement and Windows Task Manager’s Commit Size measurement? The latter is often twice as high as the former. I assume that Task Manager measures the amount of memory allocated by VirtualAlloc(..., MEM_COMMIT ...), but I could be wrong, the documentation is not entirely clear.

I’m running 11 right now, and with my typical several hundred tabs with a dozen or so windows, running for a week or more, memory usage is sitting at ~680MB, when I started looking at things with firefox 7 this was about 1.3G for roughly the same workload.

things are improving on the memory front (now if I could just figure out why it freezes for 10’s of seconds at a time…)

I’m glad the memory consumption has improved! Has performance improved noticeably?

As for the pauses, it could be the GC or CC. If you set javascript.options.mem.log to true in about:config you can open the error console and see when they occur and how long they take.

By Post Author

Here’s a few that really give Firefox a bad impression in terms of MemShrink/Snappy:

GC mode: full, timestamp: 1326195534724000, duration: 2797 ms.

I had another one that was over 5 seconds but the error console seems to have rotated it out already.

Is there a max limit on how many of these messages the console will accept before it deletes them?

Do the messages get dumped to a file?

Unfortunately those timestamp values are useless. What format are they?

Are these GC events ever triggered by sub routines directly? If so and those able to be logged, wouldn’t that make debugging what routines are slowing down Firefox infinitely easier?

Playing with a calculator it appears the timestamp is microseconds since 1970 (presumably an evolved unix timestamp format).

1326195534724000/1000000/3600/24/365 = 42.0533845 = 42 (365 day) years and 19.5 days. After factoring in leap days you’ll get a value representing a day or two ago.

PS all that and I still have to do a preschool arithmetic problem to prove I’m human. 🙂

I enabled this and just had a major pause, nothing showed up in the error console at that time

oops, I missed that the new log data was at the bottom, I’m seeing lots of delays is there any way to cut-n-paste the data from the error console other than one at a time?

GC(T+8284.5) Type:Glob, Total:209.9, Wait:1.3, Mark:137.9, Sweep:61.8, FinObj:11.8, FinStr:2.2, FinScr:0.4, FinShp:10.1, DisCod:2.2, DisAnl:6.0, XPCnct:13.9, Destry:0.3, End:21.6, +Chu:0, -Chu:0, Reason: API
CC(T+8285.0) collected: 6328 (6328 waiting for GC), suspected: 6629, duration: 543 ms.
GC(T+8289.2) Type:Glob, Total:173.8, Wait:0.7, Mark:112.8, Sweep:52.2, FinObj:11.6, FinStr:1.5, FinScr:0.4, FinShp:8.2, DisCod:1.5, DisAnl:3.4, XPCnct:12.1, Destry:0.2, End:18.9, +Chu:0, -Chu:0, Reason: API
CC(T+8290.2) collected: 3460 (3460 waiting for GC), suspected: 7172, duration: 210 ms.
GC(T+8294.4) Type:Glob, Total:157.5, Wait:0.6, Mark:112.7, Sweep:41.1, FinObj:4.1, FinStr:0.2, FinScr:0.7, FinShp:7.1, DisCod:1.3, DisAnl:2.7, XPCnct:11.4, Destry:0.3, End:13.2, +Chu:0, -Chu:0, Reason: API
CC(T+8299.5) collected: 112 (112 waiting for GC), suspected: 244, duration: 166 ms.
GC(T+8348.6) Type:Glob, Total:191.8, Wait:0.7, Mark:127.6, Sweep:54.8, FinObj:8.8, FinStr:2.2, FinScr:0.5, FinShp:9.7, DisCod:1.6, DisAnl:4.7, XPCnct:12.8, Destry:0.2, End:20.0, +Chu:0, -Chu:0, Reason: API
CC(T+8350.9) collected: 156 (156 waiting for GC), suspected: 3100, duration: 477 ms.
GC(T+8359.3) Type:Glob, Total:177.2, Wait:0.6, Mark:120.0, Sweep:49.6, FinObj:7.2, FinStr:0.7, FinScr:0.4, FinShp:8.0, DisCod:1.6, DisAnl:4.5, XPCnct:13.1, Destry:0.3, End:18.6, +Chu:0, -Chu:0, Reason: API
CC(T+8360.8) collected: 8493 (8493 waiting for GC), suspected: 12285, duration: 226 ms.
GC(T+8364.9) Type:Glob, Total:183.5, Wait:0.7, Mark:125.2, Sweep:50.9, FinObj:7.3, FinStr:1.7, FinScr:0.4, FinShp:9.0, DisCod:1.5, DisAnl:3.5, XPCnct:13.7, Destry:0.2, End:18.6, +Chu:0, -Chu:0, Reason: API
CC(T+8369.9) collected: 134 (134 waiting for GC), suspected: 1726, duration: 164 ms.
GC(T+8378.2) Type:Glob, Total:166.6, Wait:0.6, Mark:114.6, Sweep:46.7, FinObj:7.3, FinStr:1.4, FinScr:0.4, FinShp:7.7, DisCod:1.5, DisAnl:3.6, XPCnct:11.8, Destry:0.2, End:15.1, +Chu:0, -Chu:0, Reason: API
CC(T+8379.4) collected: 3310 (3310 waiting for GC), suspected: 5488, duration: 186 ms.
GC(T+8383.5) Type:Glob, Total:172.2, Wait:0.7, Mark:121.3, Sweep:47.2, FinObj:5.8, FinStr:0.5, FinScr:0.3, FinShp:7.5, DisCod:2.0, DisAnl:5.3, XPCnct:12.6, Destry:0.2, End:14.4, +Chu:0, -Chu:0, Reason: API
CC(T+8386.2) collected: 116 (116 waiting for GC), suspected: 2480, duration: 166 ms.
CC(T+8436.7) collected: 0 (116 waiting for GC), suspected: 1056, duration: 483 ms.

Run procdump (from sysinternals) to generate hang dumps of Firefox, open them with WinDbg, setup the symbols to get them from mozilla (srv*C:\symbols*http://symbols.mozilla.org/firefox) and run !analyze -v -hang . Do you see why Firefox hangs?

this is on linux. the fact that you talk about WinDbg makes me think you are giving instructions related to windows.

I really like seeing Firefox’s memory usage improved. Keep up the good work!

Speaking of add-ons leaking, I’ve been having really bad memory usage. When Firefox is open, it will gradually use more and more memory, until it reaches 3 GB and crashes. (Sometimes in just a couple days.) I’ve had it crash many times. Here’s the about:memory I copied when I was running Firefox 8:

2,743.75 MB (100.0%) — explicit
+–1,254.47 MB (45.72%) — heap-unclassified
+–1,236.69 MB (45.07%) — storage
¦ +–1,236.69 MB (45.07%) — sqlite
¦ +–1,204.00 MB (43.88%) — places.sqlite
¦ ¦ +–1,190.83 MB (43.40%) — cache-used [222]
¦ ¦ +—–13.18 MB (00.48%) — (2 omitted)
¦ +—–16.91 MB (00.62%) — other
¦ +—–15.77 MB (00.57%) — (14 omitted)
+—-190.17 MB (06.93%) — js
¦ +—80.66 MB (02.94%) — (65 omitted)
¦ +—69.66 MB (02.54%) — compartment([System Principal], 0x5759000)
¦ ¦ +–33.23 MB (01.21%) — gc-heap
¦ ¦ ¦ +–16.70 MB (00.61%) — objects
¦ ¦ ¦ +–16.53 MB (00.60%) — (6 omitted)
¦ ¦ +–19.84 MB (00.72%) — (7 omitted)
¦ ¦ +–16.58 MB (00.60%) — mjit-code
¦ +—20.63 MB (00.75%) — gc-heap-chunk-dirty-unused
¦ +—19.23 MB (00.70%) — compartment(http://camelegg.com/product/N82E16822148…)
¦ +–19.23 MB (00.70%) — (8 omitted)
+—–37.36 MB (01.36%) — layout
¦ +–24.03 MB (00.88%) — styledata
¦ +–13.33 MB (00.49%) — (1 omitted)
+—–25.06 MB (00.91%) — (6 omitted)

I noticed that the places.sqlite cache is always the largest problem. While reading a bug report, I found out that the number after “cache-used” is the number of open DB connections. And that each connection can have a 5 MB cache. I tried running Firefox with different add-ons disabled and found out it is Delicious Bookmarks that is leaking the connections. It leaks connections whenever I type into the awesome bar. I can type a few letters and it leaks several connections. (I have its awesome bar integration turned on; turning it off might help, but I haven’t tried.)

Unfortunately, I really want to keep using it, so I’ve been putting up with bad memory usage. (I really tempted to instead write an extension that provides better access to Firefox’s bookmarks by using their tags.) But I figured you might be interested to know that the Delicious Bookmarks add-on can leak an insane amount of memory.

BTW, which version of Firefox are you running? Hopefully it’s 9 or lower, that’ll explain why the “heap-unclassified” is so high… Firefox 10 has a fix that causes the sqlite reporters to do a better job, in 9 and earlier they could under-report significantly.

By Post Author

I’m still on version 3.6 because 4 was a memory hog and I downgraded. How does 10 compare to 3.6?

4 was pretty bad. 7 improved things a lot, and each of 8, 9 and 10 have improved things a bit more. I haven’t done any direct comparisons against 3.6, but I would expect 10 to be roughly the same or a bit better, but YMMV. Try it and let me know! 🙂

By Post Author

Webmonkey’s statement about Opera being the most memory efficient is puzzling to me. Out of all the major browser, I found Opera to be the worst by a comfortable margin in term of memory usage than Firefox and even multiprocess Chrome.

It was just a small personal sample, but that was one of the things I stood out to me when I tried out Opera. I remember complaining in the Opera forums and the posters typically replied, “memory is meant to used,” and then I just gave up. At this point I would take any personal testimony including mine with a grain of salt b/c there is just too much unexplainable discrepancy.

I know that Tomshardware isn’t necessarily the most loved website in the world in many places, but I just thought out of interest I would post the results from the memory tests on this blog for people to see, because Memshrink has taken Firefox from a browser that, let’s face it, was dreadful in version 4, to something that is very competitive in version 9 onward.

Firefox 9.0.1 won on Windows 7 because it was best all round and had more second place finishes than any other browser across all the tests, and also less weak performances. It didn’t get as many wins as the other browsers, but seems to be the most well rounded.

Memory Usage – http://www.tomshardware.com/reviews/macbook-air-chrome-16-firefox-9-benchmark,3108-13.html

Memory Management – http://www.tomshardware.com/reviews/macbook-air-chrome-16-firefox-9-benchmark,3108-14.html

I was particularly impressed with the memory management. Firefox looked like it was in last place but then within 2 minutes it managed to release 400MB and get 3rd place, breathing down Chrome’s neck. Not bad considering it’s history of being dreadful.

Without the Memshrink Project, Firefox might have some nice features, but it wouldn’t have anywhere near the stability sand performance it does now.

Hi Nick, i want to point you to a thread

http://forums.mozillazine.org/viewtopic.php?f=7&t=2399369&sid=3d8b4ac4ca3f9bbfe5160ea2063e36f9

Pointing to a memory leak site here ( Huge Mem usage )

http://www.gcforum.org/viewthread.php?tid=5721

I dont think it is really leaking anything. As about:memory shows it is using 1.3GB for images. And nothing from JS of other parts of browsers leaks.

However i wonder how it manages to get to 1.3GB of images? Surely my internet connection is no where near fast enough to have that amount of data downloaded within the time.

Wondering if it is worth you to have a look.

Regards

Here’s what I see:

├──1,323,757,989 B (78.25%) — images
│ ├──1,323,501,521 B (78.23%) — content
│ │ ├──1,323,499,275 B (78.23%) — used
│ │ │ ├──1,250,290,467 B (73.90%) — uncompressed-heap
│ │ │ ├─────73,208,808 B (04.33%) — raw
│ │ │ └──────────────0 B (00.00%) — uncompressed-nonheap

The “raw” entry is the number of bytes of downloaded images. These are compressed. Firefox must decompress them before displaying them, and when decompressed they take up 1.3GB.

The problem is that Firefox decompresses every image in the tab, even though most of them aren’t visible at any point in time. https://bugzilla.mozilla.org/show_bug.cgi?id=661304 is open to improve this situation.

By Post Author

I opened four copies of that site in separate tabs and I don’t see anything unusual in about:memory. Can you be more specific about what the problem is?

By Post Author

Ok, i think it causes Slow down , lag ( And not a memory leaks ) which is now officially a Snappy Problem. Sorry i got the two mixed up. But when opening say 5 – 10 tabs on the site it will causes slow down for some reasons.

Comments are closed.