A hodge-podge of things happened this week.
- The Mozilla QA team started running MozMill endurance tests every day. This should help identify and avoid regressions in memory usage in the future.
- Mihai Sucan fixed some memory leaks in the web console.
- Joe Siegrist, author of the LastPass add-on, fixed a leak in it that was causing zombie compartments. This fix is in version 1.75.0.
- Justin Lebar fixed a problem in the image memory reporters that could cause nonsensical results (such as negative numbers) in about:memory.
- Igor Bukanov added memory reporters to distinguish between clean and dirty unused chunk memory in about:memory.
- Sander van Veen, a new Mozilla contributor, wrote patches for two MemShrink bugs: bug 673737 and bug 673331. I’ll land the first one soon, and the second one is waiting for a review. Great work, Sander!
- I wrote about some allocations that are wasting space, and quantified Firefox 7’s memory usage improvements. The latter post’s take-home message was that “Firefox 7 uses less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.”
The MemShrink bug counts changed as follows.
- P1: 29 (-2, +4)
- P2: 62 (-5, +14)
- P3: 41 (-0, +6)
- Unprioritized: 2 (-10, +1)
We actually got through almost all the unprioritized bugs in today’s MemShrink meeting, which was good, but the counts are still going up. Fortunately, most of the new bugs are ideas for improvement that are reported by developers. My gut feeling (which I get from reading a lot of memory-related bug reports) is that the number of reports from users about high memory usage are much lower than they were a few months ago.
26 replies on “MemShrink progress, week 8”
Great work again. Today is the first day i encountered heap-unclassified with nearly 50% of my mem usage for some reason. Hopefully as your post state this will be reduce dramatically in Firefox 7. And further reduced with future release. I am currently using version 6 Beta.
The other thing which concerns me is JS memory. heap-unclassified was only a problem from time to time. But JS is usually one of the largest “memory consumer.” Or specifically gc-heap, which has very high memory usage.
Are there any plan to work on JS Engine memshrink?
Ed: heap-unclassified probably won’t drop dramatically in FF7 or any later released. But it’ll go down gradually, esp. thanks to https://bugzilla.mozilla.org/show_bug.cgi?id=676724.
MemShrink encompasses everything in the browser, including the JS engine. The two biggest improvements we’ve seen so far both improved gc-heap drastically. Hopefully we’ll get more JS improvements from GC improvements and other fixes.
Are most of these new improvements that you have blogged about in the last couple of weeks going to end up in (the already vastly improved) Firefox 7? Or only in Firefox 8?
Do you think there’ll be a significant difference (improvement) between 7 and 8’s memory usage too (if not as great as between 6 and 7)?
BTW this must be the most popular Mozilla blog these days 🙂
Curious: The release calendar is at https://wiki.mozilla.org/RapidRelease/Calendar. The FF7 development period ended on July 5th, so everything I’ve described for the past 5 weeks will only be in FF8. The FF8 development period ends on August 16.
FF8 should be a bit better than FF7. See the “average resident” and “peak resident” graphs at https://wiki.mozilla.org/RapidRelease/Calendar; and those measurements were taken a couple of weeks ago, so FF8 may have improved a bit more since then.
@Curious
check the bug entries and you see that most target Firefox 8. So get the latest nightly (I run them as portable) to see what is new and improved.
Andre how do “run them as portable”? I just had a quick look at portableapps.com and there’s no sign of nightly builds. Are you using the zip version and running a command line flag to store the profile somewhere else? I find that creating a new profile the normal way interferes with my normal build. How do you do it?
I use the loader from here:
http://ftp.hosteurope.de/mirror/stadt-bremerhaven.de/Portable_Firefox/
download the latest nightly (ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/) as zip and replace the firefox folder with the folder from the latest nightly and run the loader (FirefoxLoader.exe). the profile data must be copied to the folder “Profilordner” (sorry, this is a German Firefox loader).
With this I run the Aurora and nightly as portable and the firefox 5 as installed browser (release channel).
Thanks Andre. Sounds like a solution that works for you but which might be a bit fiddly for the average joe.
Nick is there any thoughts in Mozilla circles towards creating portable versions of the various Firefox release channels? Over time, I’ve seen people have trouble trying to run different builds on the same Windows install without screwing up their day-to-day existing install and profile.
Portable versions could finally solve that problem and perhaps contribute to an increase in the number of testers, don’t you think?
@pd
simply use on of the portable exe (like Portable_Firefox_5.0.1.exe), run the sfx exe to extarct it and replace the firefox folder with the ZIP. for me it works fine. I always test newer versions (especially when migrated from 3.6 to 4 which changed a lot) this way.
pd: I’m not the person to ask about channel stuff 🙂
Having some research and lots of news, this is a 143 Tabs using Firefox 6 beta
1,453.78 MB (100.0%) — explicit
├────751.33 MB (51.68%) — heap-unclassified
├────517.75 MB (35.61%) — js
│ ├──372.00 MB (25.59%) — gc-heap
│ ├───84.59 MB (05.82%) — mjit-code
│ ├───41.37 MB (02.85%) — tjit-data
│ │ ├──28.93 MB (01.99%) — allocators-reserve
│ │ └──12.43 MB (00.86%) — allocators-main
│ ├───12.41 MB (00.85%) — mjit-data
│ └────7.38 MB (00.51%) — tjit-code
├─────79.45 MB (05.47%) — layout
│ ├──79.45 MB (05.47%) — all
│ └───0.00 MB (00.00%) — (1 omitted)
├─────69.50 MB (04.78%) — storage
│ └──69.50 MB (04.78%) — sqlite
│ ├──38.03 MB (02.62%) — places.sqlite
│ │ ├──37.69 MB (02.59%) — cache-used
│ │ └───0.34 MB (00.02%) — (2 omitted)
│ ├──25.72 MB (01.77%) — urlclassifier3.sqlite
│ │ ├──25.64 MB (01.76%) — cache-used
│ │ └───0.08 MB (00.01%) — (2 omitted)
│ └───5.75 MB (00.40%) — (13 omitted)
└─────35.76 MB (02.46%) — images
├──35.34 MB (02.43%) — content
│ ├──35.18 MB (02.42%) — used
│ │ ├──18.08 MB (01.24%) — raw
│ │ └──17.10 MB (01.18%) — uncompressed
│ └───0.16 MB (00.01%) — (1 omitted)
└───0.41 MB (00.03%) — (1 omitted)
Other Measurements
1,582.82 MB — private
1,553.05 MB — resident
1,390.82 MB — heap-committed
1,361.81 MB — heap-used
31.18 MB — heap-unused
22.65 MB — gfx-surface-win32
3.23 MB — heap-dirty
1.75 MB — canvas-2d-pixel-bytes
0.21 MB — shmem-allocated
0.21 MB — shmem-mapped
0.00 MB — gfx-d2d-surfacecache
0.00 MB — gfx-d2d-surfacevram
0.00 MB — gfx-surface-image
To Mozilla Developers King,
I have two ides in my mind, so I thought I share with you.
1) To reduce Addons Overhead and many compartments issue (if possible) up to some extent, why not you guys build “Idle State API” (parity Chrome) so if Mozilla Windows in inactive for a time being, extension become idle and remove all compartments and reduce activity.
Limitation: Maybe it is not at all possible, I think.
2) Why not incorporate “Page Visibility API” (parity Chrome 12+ & IE 10), so pages utilizing this get less CPU usage and it will compensate up to some extent “decode on draw”, you can use its principle in decode on draw as well.
Limitation: Only site build to use Page Visibility API, can take benefit from it.
These will be really helpful especially in Fennec (2nd one), only active part will be rendered so less utilization of RAM and less cpu usage means better battery life but limitation is there.
Mozilla is incorporated Page Visibility API, it should start using it in *.mozila.com, *.mozilla.org and its other sub sites as well.
If I spitted out some thing dumb, ignore it.
I was unable to use HTML Tag well. Sorry!!
is there way to send memory usage information or dump file from withing firefox beta channel?, started my firefox with 3 tabs opened and closed like 15 tabs in last 5hours, firefox memory usage is 1048MB which quite huge. let me know if there is way to submit all the information you would need from my side, let me know.
here are the details from about:memory page,
1,044.44 MB (100.0%) — explicit
├────473.99 MB (45.38%) — heap-unclassified
├────421.93 MB (40.40%) — storage
│ └──421.93 MB (40.40%) — sqlite
│ ├──404.36 MB (38.72%) — places.sqlite
│ │ ├──399.81 MB (38.28%) — cache-used
│ │ └────4.55 MB (00.44%) — (2 omitted)
│ ├────8.16 MB (00.78%) — urlclassifier3.sqlite
│ │ ├──8.07 MB (00.77%) — cache-used
│ │ └──0.08 MB (00.01%) — (2 omitted)
│ ├────6.29 MB (00.60%) — other
│ └────3.12 MB (00.30%) — (12 omitted)
├────136.68 MB (13.09%) — js
│ ├───52.97 MB (05.07%) — compartment([System Principal])
│ │ ├──21.33 MB (02.04%) — gc-heap
│ │ │ ├───9.75 MB (00.93%) — objects
│ │ │ ├───5.91 MB (00.57%) — shapes
│ │ │ └───5.67 MB (00.54%) — (5 omitted)
│ │ ├──12.61 MB (01.21%) — string-chars
│ │ ├──11.00 MB (01.05%) — mjit-code
│ │ └───8.03 MB (00.77%) — (5 omitted)
│ ├───25.13 MB (02.41%) — compartment(https://mail.google.com/mail/?ui=2&shva=…)
│ │ ├──10.32 MB (00.99%) — gc-heap
│ │ │ └──10.32 MB (00.99%) — (7 omitted)
│ │ ├───8.68 MB (00.83%) — (6 omitted)
│ │ └───6.13 MB (00.59%) — mjit-code
│ ├───17.12 MB (01.64%) — (18 omitted)
│ ├───12.54 MB (01.20%) — gc-heap-chunk-unused
│ ├───10.75 MB (01.03%) — compartment(https://plus.google.com/u/0/_/notificati…)
│ │ └──10.75 MB (01.03%) — (8 omitted)
│ ├────9.53 MB (00.91%) — compartment(http://www.google.com/reader/view/#strea…)
│ │ └──9.53 MB (00.91%) — (8 omitted)
│ └────8.66 MB (00.83%) — compartment(atoms)
│ ├──6.21 MB (00.59%) — string-chars
│ └──2.45 MB (00.23%) — (7 omitted)
├──────6.41 MB (00.61%) — images
│ ├──6.14 MB (00.59%) — content
│ │ ├──6.14 MB (00.59%) — used
│ │ │ └──6.14 MB (00.59%) — (2 omitted)
│ │ └──0.00 MB (00.00%) — (1 omitted)
│ └──0.27 MB (00.03%) — (1 omitted)
└──────5.43 MB (00.52%) — (2 omitted)
Other Measurements
1,409.40 MB — vsize
1,104.77 MB — private
1,074.40 MB — resident
1,037.61 MB — heap-committed
1,018.69 MB — heap-used
64.00 MB — js-gc-heap
23.31 MB — heap-unused
4.02 MB — gfx-surface-win32
3.39 MB — heap-dirty
0.04 MB — gfx-surface-image
0.00 MB — gfx-d2d-surfacecache
0.00 MB — gfx-d2d-surfacevram
0.00 MB — shmem-allocated
0.00 MB — shmem-mapped
0.00 MB — canvas-2d-pixel-bytes
Your places.sqlite is absolutely anomalous. That might related to some extension. When you install Firefox Aurora you normally get the “test pilot” extension, and a prompt at first run to send some automated performance feedback.
your SQLite files are too large. When I had the large memory issues with Firefox 3.6 (over 1GB memory usage) I made a rebuild the profile (fresh profiule, installed all addons again and put all settings into the pref file) to reduce the memory usage.
@Tobu, @Andre
will recreate my profile thanks.
Hi Nick i would like to say that flash and java plugins are memory hogs in their own so i was hoping, adding a good click to play option integrated to the browser would be a good measure. disabled by default but the installation should give a quick reference and a option to activate it.
We just have the example of yet another “memory” bloat issues that is totally unrelated to Firefox itself.
420MB for SQLite is insane. I was thinking the other day that even 70MB for me is too large and we need to look for way to efficiently store our data.
Ed: High SQLite usage is an on-going problem. Add-ons are a cause in at least some of the cases — I’ve seen it go as high as 1.2GB due to add-ons. I wonder if busted profiles are also a cause. https://bugzilla.mozilla.org/show_bug.cgi?id=674210 will hopefully help.
All of a sudden i understand why reinstalling Firefox doesn’t help. And why some of my users / friends dont have the same experience as i do. I am guessing once you bloated up a profile, you will continue to use that profile ( i.e SQLite Db ) even if you reinstalled. Which means none of your and Fx’s teams hard work on optimization are beneficial to them. Arh this is Annoying.
On a side notes, Firefox loading very big page are still not very memory efficient even after Bug 678422 landed.
http://www.neowin.net/forum/topic/989780-meet-firefox-next/page__view__findpost__p__594231040
Dunno who else to mention this to, but I don’t think we should prominently mention claims about improved performance and memory in marketing for FF6.
People have just stopped believing us.
We can say it loud for FF7 though.
Caspy7: AIUI, performance won’t be mentioned in the FF6 marketing, instead it’ll focus on web development tools (like the new scratchpad).
Here the memshrink is not working so good. with the latest nightly from 22:00 to 10:00 it jumped from 500mb to 1000mb, without usage. I know it´s a know problem, firefox eating memory for midnight dinner, but I tought that was at least close to solved in firefox 7. Unfortunately to me the problem remains basically the same as with firefox 4, 5, 6.
Fernando, if you can file a bug in Bugzilla with full details — e.g. what sites were open, what add-ons were installed — that would be helpful.
I was using the latest nightly of firefox 7. After reading a little it came to my knowledge that you´ve said most of the memshirnk was not in firefox 7 and would only be in action in 8. I have just installed 8 alfa 2 and will see if the midnight dinner still happens.