MemShrink Progress, weeks 13–18

I’ve been on vacation, so this report covers (incompletely) the past six weeks’ worth of MemShrink improvements.

Big Things

Paul Biggar and Mike Hommey enabled jemalloc on Mac 10.6.  This will hopefully reduce fragmentation on that platform.  And it brings it in line with Windows, Linux and Android.  Now we just need it for Mac 10.5 and Mac 10.7.

Oleg Romashin found a way to drop some Thebes layers in inactive tabs in Fennec.  I won’t pretend to understand the details of this bug, but if I’ve understood correctly it can saves 12MB or more per background tab.

Jeff Muizelaar turned on decode-on-draw.  This means that if you open a new page in a background tab, none of its images will be decoded until you switch to that tab.  Previously any images would be decoded and then discarded after 10 to 20 seconds (unless you switched to the tab before the time-out occurred).  This change can save a lot of memory (and CPU time) for users browsing image-heavy sites.

Gian-Carlo Pascutto optimized the safe browsing database.  This hopefully has fixed our url-classifier bloat problems.  (I plan to verify this soon.)

Chris Leary and Jonathan “Duke” Leto made regexp compilation lazy.  This potentially saves 10s or even 100s of MBs of memory in some cases by not compiling some regexps, and also allowing regexps to be GC’d more quickly.  There were some possible performance regressions from this patch, it’s unclear from the bug what exactly the state of these are.

Justin Lebar converted some uses of nsTArray to nsAutoTArray (and also here).  These avoided some calls to malloc (in the latter case, around 3% of all malloc calls!) and may help reduce heap fragmentation a little.  Robert O’Callahan did a separate, similar nsAutoTArray change here.  Justin also avoided another 1% of total malloc calls in JSAutoEnterCompartment.

Chris Leary rewrote JSArena, which avoided some wasted memory, as well as replacing some hoary old C code with sleek modern C++.

SMALLER THINGS

A new contributor named Jiten (a.k.a. deLta30) fixed about:memory’s GC and CC buttons so they trigger collections in both the content and chrome process in Fennec.  Previously only the chrome process was affected.  (I’m not sure how this change will be affected by the decision to switch to the native chrome process in Fennec.)  Great work, Jiten!

I avoided some wasted space in the JS code generator, and some more in the parser.  Justin Lebar did something similar in nsTArray_base.

Jonathan Kew added a memory reporter for textruns and associated data.  Justin Lebar added the “history-links-hashtable” memory reporter.

Justin Lebar fixed some bogus “canvas-2d-pixel-bytes” values in about:memory.

Brian Bondy fixed a leak in Windows drag and drop code.

Tim Taubert wrote about finding leaks in browser-chrome mochitests.

Bug Counts

The current bug counts are as follows.  The differences are against the MemShrink week 12 counts.

  • P1: 37 (-3, +11)
  • P2: 108 (-9, +37)
  • P3: 53 (-1, +14)
  • Unprioritized: 6 (-21, +5)

They’re still going up.  The good news is that my gut feeling is that not many of these bugs are problems reported by users.  (And those that are often are caused by add-ons.)  Most of the new reports are ideas for improvements from developers.

55 Responses to MemShrink Progress, weeks 13–18

  1. While you were out I upgraded from 3.5 to 7.01, but then immediately jumped to the 8.0 beta. 8 is quite a bit better on memory (especially at startup) than 7. The bad news is, as I feared, it’s worse than 3.5; memory builds up a little bit faster than it used to. But with the exception of BarTab no longer working (gah), the browser experience overall feels much better in 8, so it’s still a trade up. I’m hoping desperately that the bug regarding Firebug’s zombie compartments will soon be fixed at either end, because I depend on that add-on.

    The improvements from 7 to 8 were so significant that I’m wondering if I should just jump to 9 as soon as possible. The Memshrink project has clearly done many good things for the browser. The only thing is, there seems to be nothing in the way of a Memshrink changelog for me to see which fixes have landed in which versions.

    • Nicholas Nethercote

      Good to hear that FF8 is better than FF7. Unfortunately we don’t have a change log for MemShrink changes, and even if we did it’d be hard to predict the impact on any individual user because browsing patterns vary so much from person to person. You’ll just have to try it (now, or by waiting) to see.

      Oh, BarTab functionality is now built-in! Under the “General” pane of the preferences dialog, check the “Don’t load tabs until selected” box :)

      And if you want to get excited by the future, check out https://bugzilla.mozilla.org/show_bug.cgi?id=637931. Brian Hackett is well on the way to reducing the size of a JSObject from 10 words to 4 words. That means all the “objects” entries in about:memory — which are generally very big — will become 2.5x smaller! And the “gc-heap-chunk-dirty-unused” entry will indirectly shrink a bit too.

      • Can you let us know which of the fixes in this blog made it into FF9 and which will be in FF10?

        • Nicholas Nethercote

          Not easily, sorry. The only way to do it is to look at the commit URL in each individual bug. Those that landed on mozilla-central before 2011-09-27 should be in FF9, those that landed after should be in FF10 (assuming they’re not backed out during aurora or beta testing).

          • The Target Milestone field in the bug should correctly show which version of Firefox a fix will ship with (i.e. mozilla10 = Firefox 10).

      • The problem is the BarTab functionality is only partly in. It did three important things: 1) It didn’t load tabs until they were needed, 2) it could unload tabs not only after a period of inactivity but also on command (and reload them later), and 3) it showed which tabes were unloaded by making them a little darker. Only #1 is implemented, not #2 or #3. I realize my use of the ColorfulTabs extension would be incompatible with #3 anyway, but the second one is still pretty important to me.

    • I bet that 9 will regress again, due to TI.

  2. I think equal that @Lummox JR, FF8 is more fasater and light

  3. It’d be interesting to see some kind of comparison between 7.0.1 and the current trunk, so see how much improvement has been made.

    Did anyone try to do some tests?

    • Nicholas Nethercote

      See http://mozmill-crowd.brasstacks.mozilla.com/#/endurance, which has endurance test results for various versions at regular intervals. The 7.0 links were blank there, but the ones for 8.0, 9.0 and 10.0 do. Looks like 9.0 improves over 8.0, and 10.0 isn’t much changed:

      8.0:
      Explicit memory (MB): Minimum: 34 / Maximum: 529 / Average: 388
      Resident memory (MB): Minimum: 94 / Maximum: 650 / Average: 498

      9.0:
      Explicit memory (MB): Minimum: 33 / Maximum: 511 / Average: 347
      Resident memory (MB): Minimum: 87 / Maximum: 621 / Average: 451

      10.0:
      Explicit memory (MB): Minimum: 53 / Maximum: 485 / Average: 346
      Resident memory (MB): Minimum: 126 / Maximum: 625 / Average: 450

      I’m not sure what happened with the minimums on 10.0, but that’s the least important measurement.

  4. I dont think Firefox 8 is better then 7 in terms of memory, as a matter of fact there are few of us who sees memory usage goes back up to pre 7 version.

    I have a few website that so far i have found to be constantly leaking memory. I will post them once i have done a few more test.

  5. Welcome back Mr Nethercote. I’ve missed my Fave Read of the Week. Given it’s been 5 weeks, do you think the progress on MemShrink has been a little slower than it could be? Still, good to read your comment about Mr Hackett’s upcoming patch.

    Unfortunately I can no longer run Aurora 9 since it broken too much behaviour, in particular tabs. Weird things like closing the far right tab, drag and dropping tabs and some tab styling going awry were enough for me. Quite a shame really.

  6. welcome back Nicholas! I hope you enjoyed your holidays :)

    I hope you can continue to make Firefox better :)

  7. Incidentally, another nsTArray->nsAutoTArray fix was recently applied to fix bug 522193, a long-standing WebGL performance problem.

  8. I’ve got to say that your posting are always nice to read. They’ve always got such a positive attitude in them. “Made progress on this”, “improved that”, “triaged and got an idea what to do next”.

    Here’s actual work being done and I like that!

  9. I always wonder how little snippet of JavaScript could use so much memory…… What are the chances that we can shrink it by 90%….

    3,238.95 MB (100.0%) — explicit
    ├──1,920.37 MB (59.29%) — js
    │ ├────353.06 MB (10.90%) — (134 omitted)
    │ ├────276.73 MB (08.54%) — compartment(http://www.facebook.com/extern/login_sta…)
    │ │ ├──121.77 MB (03.76%) — gc-heap
    │ │ │ ├───71.42 MB (02.21%) — objects
    │ │ │ ├───48.84 MB (01.51%) — shapes
    │ │ │ └────1.50 MB (00.05%) — (4 omitted)
    │ │ ├───78.98 MB (02.44%) — scripts
    │ │ ├───42.31 MB (01.31%) — mjit-code
    │ │ └───33.67 MB (01.04%) — (6 omitted)
    │ ├────126.08 MB (03.89%) — compartment(http://fileforum.betanews.com/detail/Pli…)
    │ │ ├───54.74 MB (01.69%) — gc-heap
    │ │ │ ├──31.78 MB (00.98%) — objects
    │ │ │ ├──20.44 MB (00.63%) — shapes
    │ │ │ └───2.52 MB (00.08%) — (4 omitted)
    │ │ ├───35.08 MB (01.08%) — scripts
    │ │ ├───20.38 MB (00.63%) — mjit-code
    │ │ └───15.89 MB (00.49%) — (5 omitted)
    │ ├────112.86 MB (03.48%) — compartment(https://plusone.google.com/u/0/_/+1/fast…)
    │ │ ├───52.05 MB (01.61%) — gc-heap
    │ │ │ ├──31.49 MB (00.97%) — objects
    │ │ │ ├──19.67 MB (00.61%) — shapes
    │ │ │ └───0.90 MB (00.03%) — (4 omitted)
    │ │ ├───32.47 MB (01.00%) — scripts
    │ │ └───28.34 MB (00.87%) — (6 omitted)
    │ ├────102.78 MB (03.17%) — compartment(http://www.engadget.com/2011/10/18/canon…)
    │ │ ├───41.86 MB (01.29%) — gc-heap
    │ │ │ ├──25.21 MB (00.78%) — objects
    │ │ │ └──16.65 MB (00.51%) — (5 omitted)
    │ │ ├───25.23 MB (00.78%) — scripts
    │ │ ├───22.05 MB (00.68%) — mjit-code
    │ │ └───13.64 MB (00.42%) — (5 omitted)
    │ ├─────64.78 MB (02.00%) — compartment(http://www.neowin.net/news/samsung-galax…)
    │ │ ├──34.18 MB (01.06%) — (8 omitted)
    │ │ └──30.60 MB (00.94%) — gc-heap
    │ │ └──30.60 MB (00.94%) — (6 omitted)
    │ ├─────63.65 MB (01.97%) — compartment(http://betanews.com/)
    │ │ ├──41.21 MB (01.27%) — (8 omitted)
    │ │ └──22.44 MB (00.69%) — gc-heap
    │ │ └──22.44 MB (00.69%) — (6 omitted)
    │ ├─────54.95 MB (01.70%) — compartment(http://googleads.g.doubleclick.net/pagea…)
    │ │ ├──40.66 MB (01.26%) — gc-heap
    │ │ │ ├──20.18 MB (00.62%) — objects
    │ │ │ ├──20.12 MB (00.62%) — shapes
    │ │ │ └───0.37 MB (00.01%) — (4 omitted)
    │ │ └──14.28 MB (00.44%) — (7 omitted)
    │ ├─────53.90 MB (01.66%) — compartment(http://www.appleinsider.com/articles/11/…)
    │ │ ├──33.14 MB (01.02%) — (7 omitted)
    │ │ └──20.76 MB (00.64%) — gc-heap
    │ │ └──20.76 MB (00.64%) — (6 omitted)
    │ ├─────52.93 MB (01.63%) — compartment(http://www.tomshardware.com/news/Brando-…)
    │ │ ├──27.49 MB (00.85%) — (8 omitted)
    │ │ └──25.44 MB (00.79%) — gc-heap
    │ │ └──25.44 MB (00.79%) — (6 omitted)
    │ ├─────52.73 MB (01.63%) — compartment([System Principal], 0x454a800)
    │ │ ├──35.92 MB (01.11%) — gc-heap
    │ │ │ ├──20.16 MB (00.62%) — objects
    │ │ │ └──15.75 MB (00.49%) — (6 omitted)
    │ │ └──16.81 MB (00.52%) — (8 omitted)
    │ ├─────52.50 MB (01.62%) — compartment(http://platform.twitter.com/widgets/twee…)
    │ │ ├──29.13 MB (00.90%) — gc-heap
    │ │ │ └──29.13 MB (00.90%) — (6 omitted)
    │ │ └──23.38 MB (00.72%) — (8 omitted)
    │ ├─────43.91 MB (01.36%) — compartment(http://www.xbitlabs.com/news/other/displ…)
    │ │ ├──24.51 MB (00.76%) — (8 omitted)
    │ │ └──19.40 MB (00.60%) — gc-heap
    │ │ └──19.40 MB (00.60%) — (6 omitted)
    │ ├─────41.14 MB (01.27%) — compartment(http://differentapple.blogspot.com/2011/…)
    │ │ ├──24.13 MB (00.75%) — (7 omitted)
    │ │ └──17.00 MB (00.52%) — gc-heap
    │ │ └──17.00 MB (00.52%) — (6 omitted)
    │ ├─────38.23 MB (01.18%) — compartment(http://chinese.winandmac.com/news/pccw-l…)
    │ │ ├──21.41 MB (00.66%) — (7 omitted)
    │ │ └──16.82 MB (00.52%) — gc-heap
    │ │ └──16.82 MB (00.52%) — (6 omitted)
    │ ├─────37.26 MB (01.15%) — compartment(http://www.tgdaily.com/business-and-law-…)
    │ │ ├──19.64 MB (00.61%) — (7 omitted)
    │ │ └──17.63 MB (00.54%) — gc-heap
    │ │ └──17.63 MB (00.54%) — (6 omitted)
    │ ├─────37.04 MB (01.14%) — compartment(http://www.anandtech.com/show/4959/iclou…)
    │ │ └──37.04 MB (01.14%) — (9 omitted)
    │ ├─────35.93 MB (01.11%) — compartment(http://www.macobserver.com/tmo/article/a…)
    │ │ ├──19.50 MB (00.60%) — (8 omitted)
    │ │ └──16.43 MB (00.51%) — gc-heap
    │ │ └──16.43 MB (00.51%) — (6 omitted)
    │ ├─────32.38 MB (01.00%) — compartment(http://promo.fileforum.com/)
    │ │ └──32.38 MB (01.00%) — (9 omitted)
    │ ├─────31.76 MB (00.98%) — compartment(http://www.gsmarena.com/android_ice_crea…)
    │ │ └──31.76 MB (00.98%) — (9 omitted)
    │ ├─────27.81 MB (00.86%) — compartment(http://mediacdn.disqus.com/1318904734/bu…)
    │ │ └──27.81 MB (00.86%) — (9 omitted)
    │ ├─────27.21 MB (00.84%) — compartment(https://plus.google.com/u/0/_/notificati…)
    │ │ └──27.21 MB (00.84%) — (8 omitted)
    │ ├─────26.71 MB (00.82%) — compartment(http://www.bit-tech.net/gaming/pc/2011/1…)
    │ │ └──26.71 MB (00.82%) — (9 omitted)
    │ ├─────25.44 MB (00.79%) — compartment(http://arstechnica.com/gadgets/news/2011…)
    │ │ └──25.44 MB (00.79%) — (9 omitted)
    │ ├─────24.88 MB (00.77%) — compartment(https://mail.google.com/mail/?tab=Xm#inb…)
    │ │ └──24.88 MB (00.77%) — (8 omitted)
    │ ├─────23.72 MB (00.73%) — compartment(http://whiteappleer.tw/2011/10/18/the-in…)
    │ │ └──23.72 MB (00.73%) — (9 omitted)
    │ ├─────23.64 MB (00.73%) — compartment(http://remnant.fmpub.net/zone/1545?t=htm…)
    │ │ └──23.64 MB (00.73%) — (9 omitted)
    │ ├─────20.57 MB (00.64%) — compartment(http://www.dpreview.com/news/1110/111018…)
    │ │ └──20.57 MB (00.64%) — (9 omitted)
    │ ├─────18.80 MB (00.58%) — compartment(atoms)
    │ │ └──18.80 MB (00.58%) — (2 omitted)
    │ ├─────18.76 MB (00.58%) — compartment(http://www.tuaw.com/2011/10/18/forbes-ap…)
    │ │ └──18.76 MB (00.58%) — (9 omitted)
    │ └─────18.23 MB (00.56%) — compartment(http://www.google.com/finance)
    │ └──18.23 MB (00.56%) — (9 omitted)
    ├────802.89 MB (24.79%) — heap-unclassified
    ├────321.70 MB (09.93%) — layout
    │ ├──196.58 MB (06.07%) — arenas
    │ └──125.12 MB (03.86%) — styledata
    ├─────75.30 MB (02.32%) — dom
    ├─────70.56 MB (02.18%) — images
    │ ├──70.19 MB (02.17%) — content
    │ │ ├──70.17 MB (02.17%) — used
    │ │ │ ├──49.17 MB (01.52%) — raw
    │ │ │ ├──20.99 MB (00.65%) — uncompressed-heap
    │ │ │ └───0.00 MB (00.00%) — (1 omitted)
    │ │ └───0.02 MB (00.00%) — (1 omitted)
    │ └───0.38 MB (00.01%) — (1 omitted)
    ├─────41.15 MB (01.27%) — storage
    │ └──41.15 MB (01.27%) — sqlite
    │ ├──25.01 MB (00.77%) — places.sqlite
    │ │ ├──24.76 MB (00.76%) — cache-used [4]
    │ │ └───0.25 MB (00.01%) — (2 omitted)
    │ └──16.14 MB (00.50%) — (13 omitted)
    └──────6.98 MB (00.22%) — (4 omitted)

    Other Measurements
    0.34 MB — canvas-2d-pixel-bytes
    0.20 MB — gfx-d2d-surfacecache
    9.88 MB — gfx-d2d-surfacevram
    33.04 MB — gfx-surface-image
    0.00 MB — gfx-surface-win32
    2,905.34 MB — heap-allocated
    2,960.94 MB — heap-committed
    1.97 MB — heap-dirty
    67.66 MB — heap-unallocated
    31 — js-compartments-system
    130 — js-compartments-user
    886.00 MB — js-gc-heap
    84.41 MB — js-gc-heap-arena-unused
    0.00 MB — js-gc-heap-chunk-clean-unused
    10.13 MB — js-gc-heap-chunk-dirty-unused
    10.67% — js-gc-heap-unused-fraction
    3,361.20 MB — private
    3,386.68 MB — resident
    3,796.91 MB — vsize

    • which JS? Please post the link to it.

    • Nicholas Nethercote

      They aren’t “little snippets”. Sites like Facebook use huge amounts of JS code — multiple megabytes of JS source is common. It’s not so inaccurate to think of a browser as primarily a JavaScript environment; the image/layout/etc capabilities are secondary.

    • ├──1,920.37 MB (59.29%) — js
      I am referring to the JS in total. It got me remember how C programs are tiny. While the two are totally different beast. I was wondering if something could be done to javascript output to dramatically minimize its memory usage.

      I think Facebook is just a example of heavy JS usage, others are very light.

      I just paste the post i wrote in Mozilla zine

      Lots and Lots of news today, Motorola RAZR Droid, Samsung Nexus Prime, Android IceCream Sandwich 4.0, Apple Conference Call, Intel ‘s Results………

      Which means i had opened up to 250 Tabs at one point. And Firefox was…. dead, literally. And this is on a Quad Core SandyBridge, 8GB Ram + Fast SSD Drive. And it uses the most memory out of the three.

      I tired opening the same pages with Flash Block , better this time, although it did crash in the end. Doing the same one Chrome? To my surprise it was still VERY responsive, and web page scrolling were still quick as well. And it did actually end up using less memory.

      Edit: Those that are interested I done the same one latest opera as well. It was slightly less snappy then chrome and uses about the same memory. But again no tab overflow. This is on the latest Alpha Opera.

      But, Chrome was never built for that many of tabs anyway. ( Which is contradicting since its engine was built for that ). My tabs on tab bar was insanely small that was just stupid. ( No Tab overflow )

      So yea, i hope they could speed up e10s development..

      • 250 Tabs? You should rethink your workflow. Why do you need all the tabs? I have around 15-20 at a maximum.

      • I’m sorry mate, but 250 tabs is just lazy. Close a few! Geez. There has to be a reasonable maximum number of open tabs. Mozilla should publish what is feels is a reasonable number of open tabs. All the Panorama and not-decoding-until-active and the GC running every 20 seconds … does anyone ever believe that a browser will *ever* be able to support 250 tabs?

        I guess if anything, this complaint might reasonably fall into the category of performance whilst using a lot of memory? If someone has 16GB of RAM, perhaps 250 tabs should be usable?

        • As long Firefox is 32Bit only it can’t take advantage of 16GB. I think Firefox runs into the 32Bit VA space limit and that’s why it is so slow for him.

          @Ed

          With firefox 8 you have an option to laod only tabs when you access them. Set the option and try again. It should improve the speed.

  10. Why is it necessary to do three sets of work integrating jemalloc on the crApple platform? That’s a joke surely? What a gigantic chunk of time to be spending on a platform that accounts for a teeny % of market share! Why not focus on Windows more, especially instead of old versions of crApple like the 10.6 version.

  11. I was using the stable/release channel but switched to the beta channel once 7.0 moved into beta and I haven’t looked back since. It’s been a real pleasure to use Firefox. At the moment my one and only web browser is Firefox 8.0 b3. I have 2 add-ons installed. British English Dictionary 1.19.1 for spell checking as I type on blogs like this and on Facebook and Hotmail and Testpilot 1.2 to feed data back to Mozilla. It’s on a laptop with Windows 7, Intel 2.3 GHz dual core, 4GB RAM and 280GB hard drive.

    I don’t know if it’s my low number of add-ons or my well maintained and very clean laptop but I noticed a slightly improvement all round in performance for Firefox when I moved from 7.0b6 to 8.0 b1 and was and still am very impressed with it.

    Anyway, keep up the good work. Memshrink is without a doubt critically important to the future success of Firefox.

    I can’t wait for the javascript improvements that will come with 9.0 b1 and beyond.

  12. Good to see you back Nicholas, have missed the weekly reports! :-)

  13. Hello Nicholas.

    I have a question about FF8 performance. You linked from your article “FF7 is lean and fast” a paper from Blargon7 about endurance tests:
    http://blargon7.com/2011/07/endurance-tests-demonstrate-firefoxs-memory-usage-improvements/

    Blargon7 observed FF8 has an higher average/peak explicit memory than FF7. I quote him:
    “You can see in the following charts that explicit memory usage in Firefox 7 is rough half that of Firefox 6! It appears that this has increased in Firefox 8, which will require some further investigation. The resident memory has continued to decrease in each version.”

    Do you know if it has been investigated and fixed in FF8?

    Bye.

    • Nicholas Nethercote

      The explicit measurements for FF8 were done incorrectly in the Blargon7 post. That’s why I didn’t include those graphs in my post.

  14. I saw your comment on the Firebug issue, possibly relating to code injection from chrome, and that has me very excited. I haven’t really dared to go near a nightly/aurora build yet, but if this is fixed in Firefox 10 then I’ll probably make the jump ahead as soon as 8 is released and 9 goes into beta. I’ve wondered for some time if the injection issue also impacts Greasemonkey, an add-on I can’t live without. Between the JSObject structure dropping in size (if that’s done on time) and having fewer zombie compartments, FF10 could potentially be much slimmer than previous versions.

    My hope is to get back to 3.5′s memory levels or better, without the growth over time problem; I can keep 8 up for maybe a day, whereas 3.5 could stay up usually a few days without an issue.

  15. Hi Nicholas,

    While the RAM footprint has been greatly improved with Firefox 7, it seem to still have quite a few problems with Commit Size (Windows 7 64B).

    Throughout a session, used RAM is freed up in amounts never experienced before, but the Commit Size keeps growing and growing. I rarely see any decreases in it all. After a few hours of use I often find it in the region around 3GB, while the RAM is way under 1GB.

    I definitely notice that FF7 is slowing down as this is happening, however not even remotely as much as in earlier versions.

    When I restart (using https://addons.mozilla.org/en-US/firefox/addon/restart-firefox-10749/), and are back with the same windows/tabs as before the restart, FF7 is much faster and the Commit Size is usually around the same size as the RAM usage.

    So, there are clearly a lot of improvements made in FF7 in regards om memory use, leading to a much better UX. If you also can sort out the Commit Size stuff, then its going to rock hard together with the other improvements coming in future versions.

    Thanks for awesome work sorting this out,
    /thomas

  16. It seems that the reputation of Firefox’s poor memory management is going to take a while to correct. Personally I understand this since we’ve been told (pre MemShrink days) more than once that the issue was being addressed and it has regressed back and forth like rollercoaster. It is sad however to still see articles like this:
    http://www.ghacks.net/2011/10/22/dormancy-unload-firefox-tabs-to-free-ram/
    popping up.

    • Nicholas Nethercote

      Why is it sad?

      W.r.t. bug 675539, the article says “A very similar feature is planned for upcoming Firefox versions” which is grossly overstating it. I suspect the author of the article assumes that because a feature is suggested in Bugzilla it’s certain to happen, which is far from the case.

      • Sad because the existence of both the Add-On and article suggest that people still very much think Firefox’s memory performance is a problem. The changes that went into 7 were a real milestone IMHO however I guess maybe there’s still a significant number of users who experience bad Firefox memory performance due to the inability of Firefox to deal with add-on memory management?

        • Nicholas Nethercote

          Well, there are still problems. Badly behaved add-ons are one of them. FF7 was the first big improvement, but there’s still plenty of room for improvement and that’s why MemShrink is still going :)

  17. Mr. Nethercote: thank you (and your team) for all the work that MemShrink has done. Your blog is entertaining in an epic-engineering sort of way…

    In terms of 250+ tabs: has the telemetry project given any indication of how many tabs are used by the average user (mean +/- 1 stddev)? 250 sounds large to me. Is 250+ tabs within the common case? The commenter even admits this was a burst of 250 tabs…

    Also, there are two parts of this discussion that I think are a strange:

    (1) Some people would really prefer that tabs load even if they are not visible. I usually open tabs in the background so that they can load while I am reading the current page.

    For example, go to your favorite online newspaper, open articles in new tabs, but keep reading the front page until they are ready. It’s a latency-hiding strategy, and I doubt I’m the only user who does it.

    (2) How come none of the MemShrink numbers compare against firefox 3.6? I’ve heard again and again how FF-latest is such an improvement over 4,5,6… But you know what? 4,5,6 are no longer supported, but 3.6 is still supported.

    Since these numbers are not presented, the pessimist in me assumes that FF3.6 uses less RAM under the same conditions than FF4,5,6,7,8,… Please correct me if I am wrong.

    • I agree that the not-decoding-images-in-background change and other not-loading-in-background logic is woorying. Using the QuickDrag extension, I frequently ‘flick’-load a link into a new background tab so that it will be loaded when I finish reading the page it came from. This to me is a critical feature of tabbed browsing in itself and I doubt I’m alone.

      Nicholas can you provide some more throughts on this topic? I hope it’s not the first sign of Firefox starting to cannibalize it’s core features to improve memory management.

      • Nicholas Nethercote

        AIUI, decode-on-draw didn’t show any regressions in performance so the graphics team enabled it. If you think about it, the only case where it might hurt you is if you open a tab in the background and then switch to it within 20 seconds — you might get some flickering/loading then. In FF4 that time-out was much higher, something like 4 minutes, and we had heaps of complaints. There are trade-offs all around, that’s life.

        What other signs of “cannibilization” can you give?

        • I don’t think there are other examples. That’s why I said ‘the first’. So far I’ve been very impressed with the decision-making WRT any trade-off questions.

          I don’t really understand the 20 seconds vs 4 minutes you refer to. I think a lot of users would load a page in the background and then switch to it as soon as it’s loaded. That’s a core feature of tabbed browsing. There must be a lot of websites out there that load quicker than 20 seconds. Hmmm, Might be one for the metrics people?

    • Nicholas Nethercote

      There was a test pilot study about a year ago. https://testpilot.mozillalabs.com/testcases/a-week-life-2/aggregated-data.html has the data, http://www.slate.com/articles/life/the_hive/2010/12/open_this_story_in_a_new_tab.single.html has a write-up. The number of users with 100+ tabs appears to be statistically insignificant according to the write-up, but I suspect the number is growing, and also that those 100+ tab people are power users and so disproportionately influential.

      When it comes to loading tabs in the background, that’ll always happen if you open them from a page. The lazy loading only applies to tabs when you restart Firefox.

      As for comparisons against FF3.6, it lacks much of the internal memory measuring infrastructure that FF4 and later versions have, which makes detailed comparisons harder. A while ago I tried comparing just RSS of FF3.6 to FF7 for http://gregor-wagner.com/tmp/mem, but FF3.6 crashed halfway through :( And measuring against 3.6 is not that useful — it’s going to reach it’s end-of-life pretty soon. So it’s a lack of data rather than a conspiracy of silence.

      • Figured I’d chime in as a user whose peak tab usage hits the 100ish range from a base value of 50-60 (all in a single top level window); my normal usecase for max values is #1 in Altruizm’s post above. FF’s instability under this high a load has driven me to Opera as my primary browser (halflife of a ~ 1 week instead of 1 day) despite a strong desire to use FF for its superior compatibility an extension support. A friend of mine (15-20 tabs/top level window; 1 window/task) has recently made the same switch despite even heavier dependence on extensions.

        Long term I suspect 1 process/tab or a 64 bit version will resolve the memory use related stability problems since a single tab is unlikely to ever approach the 3GB 32 bit process limit. Unfortunately the former doesn’t appear to have an definite ETA for when its expected to be completed and the latter isn’t feasible unless the catch22 with plugin support is ever resolved.

        • Nicholas Nethercote

          Can you explain more what you mean by instability? Crashes?

          Have you tried FF7?

          Do you have any add-ons installed?

      • I didn’t realize the decode-on-draw change was only when starting up. That’s not such a big deal I guess, though some might disagree.

    • I can’t speak for 3.6, but I can give you a rough comparison of 3.5 to 7 and 8. I’m a user who frequently runs many windows (at the moment 15; sometimes it goes as high as 25 or so), each of which has on average maybe 2-3 tabs open. If I had to take a ballpark guess, I run about 60 tabs on average right now, but more when I’m lazy about bookmarking stuff. I run a few add-ons, most prominently Adblock Plus, Firebug, Greasemonkey, and Forecastfox. I found out early on that 3.6 was an epic fail in every way; it took my session restore time from about 2 minutes to about 20 (not kidding); I had to regress to 3.5 and stay there until only recently.

      In a typical session, Firefox 3.5 would start out a little under 350MB, and grow slowly with usage (not idling) so that after a few days it would eventually reach numbers in the 800MB range and up. After crossing 1000MB it would tend to become completely unstable and race until it crashed.

      Firefox 7 does not have the grow-on-idle problem that was prominent in 4 and I believe all theway up through 6. However it did have higher memory usage for me. On startup it was quite high, settling at about 650MB as I recall but using virtual memory in excess of a gigabyte during the start. And this was with the BarTab extension still running, which was preventing page loads for unused tabs even though in most other respects it wasn’t working right.

      I switched to Firefox 8 right away. After startup it seems to settle in comfortably around 450MB, and then it grows over time with usage again. After about a day or two of heavy use, it crosses the 1000MB barrier. Unlike its predecessor, it seems more stable in that neighborhood but some things like Flash definitely operate slower. My investigations suggest the growth is from zombie compartments caused by Firebug (see bug 669730), an issue which may have been fixed by a related bug fix that is projected to land in Firefox 10.

      My overall anecdotal report is that every version after 3.5 has been worse than it on memory, but 7 is sort of usable due to its many fixes and 8 is significantly better. Compared to 3.5, 8 starts out a little bit higher and gains about twice as fast; that’s bad, but it’s better than 7 in both respects. On the other hand, 8 definitely feels faster and I think it performs a lot better–its Flash performance seems a lot better as well, at least until it crosses into the red on memory. (I still run a single-core machine.) My hope is that the JavaScript type inference in 9 won’t cause a significant regression, or will be balanced out by other fixes. I suspect that with the fixes currently in the pipeline, the real stellar version will be Firefox 10.

      Bear in mind a big factor in all the above is Firebug. I need to use that plugin not only for work, but also for beating obnoxious websites into submission so I can fix them with Stylish or Greasemonkey. Most add-ons don’t produce zombie compartments and so you’ll probably see memory use stay a lot more stable over time if you’re not using Firebug.

      • 450MB at startup? this is too high. I had large memory usage in Firefox 3.x compared to later versions. Here is mine of FF7.01:

        139.96 MB (100.0%) — explicit
        ├───87.46 MB (62.49%) — js
        │ ├──54.30 MB (38.80%) — gc-heap-chunk-unused
        │ ├──26.99 MB (19.28%) — compartment([System Principal])
        │ │ ├──15.64 MB (11.18%) — gc-heap
        │ │ │ ├───7.66 MB (05.47%) — objects
        │ │ │ ├───4.59 MB (03.28%) — shapes
        │ │ │ ├───2.00 MB (01.43%) — arena-unused
        │ │ │ ├───1.29 MB (00.92%) — strings
        │ │ │ └───0.11 MB (00.08%) — (3 omitted)
        │ │ ├───4.17 MB (02.98%) — string-chars
        │ │ ├───3.14 MB (02.24%) — scripts
        │ │ ├───2.43 MB (01.74%) — object-slots
        │ │ ├───1.13 MB (00.80%) — mjit-code
        │ │ └───0.48 MB (00.34%) — (3 omitted)
        │ ├───4.50 MB (03.22%) — compartment(atoms)
        │ │ ├──2.97 MB (02.12%) — string-chars
        │ │ ├──1.54 MB (01.10%) — gc-heap
        │ │ │ ├──1.52 MB (01.09%) — strings
        │ │ │ └──0.01 MB (00.01%) — (6 omitted)
        │ │ └──0.00 MB (00.00%) — (6 omitted)
        │ ├───1.19 MB (00.85%) — gc-heap-chunk-admin
        │ └───0.48 MB (00.35%) — (5 omitted)
        ├───38.53 MB (27.53%) — heap-unclassified
        ├───12.21 MB (08.72%) — storage
        │ └──12.21 MB (08.72%) — sqlite
        │ ├───8.99 MB (06.43%) — places.sqlite
        │ │ ├──8.86 MB (06.33%) — cache-used
        │ │ └──0.14 MB (00.10%) — (2 omitted)
        │ ├───2.45 MB (01.75%) — (11 omitted)
        │ └───0.76 MB (00.55%) — other
        ├────0.88 MB (00.63%) — (2 omitted)
        └────0.88 MB (00.63%) — xpti-working-set

        Other Measurements
        431.47 MB — vsize
        199.46 MB — resident
        185.12 MB — private
        142.06 MB — heap-committed
        138.58 MB — heap-used
        73.00 MB — js-gc-heap
        11.63 MB — gfx-d2d-surfacevram
        6.42 MB — heap-unused
        0.28 MB — heap-dirty
        0.23 MB — gfx-surface-image
        0.15 MB — gfx-d2d-surfacecache
        0.00 MB — gfx-surface-win32

        I think Firebug and Forecastfox cause the high mem usage. Use a portable version of Firefox and install firebug there and use this only for debugging. And firefox 3.x crashes earlier because of the 2GB VA space limit. With ff >=4 the limit is now 4GB if you use a 64Bit Windows (Large address aware flag set).

        It would be nice if Mozilla adds something like Microsoft for the kernel pool. MS added a Tag to trace which driver uses which memory. This would also help to see which addon uses to much memory.

        • This is a 32-bit system, so I’m not counting 64-bit awareness as a possible reason for improvement in that area.

          The startup memory may indeed be high because of Firebug, which I know is also causing the zombie compartments; I doubt Forecastfox is causing much of an impact. But for the number of windows and tabs I have, high usage was common in 3.5 and I’m not at all surprised to see it higher in 8.

          Are you sure you’re grabbing about:memory after session restore is finished though? If about:memory is in my saved session, it’ll erroneously report much lower usage when it first comes up because at that point not a lot has been loaded yet. However I’m not relying on it for the off-the-cuff numbers here: Those come from observations of the task manager. Since 3.5 had no about:memory feature to speak of, using that in any discussion comparing 3.5 to newer builds is apples-and-oranges.

  18. Forgive me if this is the wrong place for such questions / statements. I recently found this page after doing some searches related to FF8 and unloading tabs. I’ve seen a few comments stating that FF8 now unloads tabs if they are unused for a time to help with memory. I was extremely excited by this so i decided to try it out.

    I’m a 3.6 user with bartab installed (also TMP). At home i run anywhere between 40-80 tabs, and at work i run at least 100 constantly. This is “normal” usage for me and works quite well with bartab installed. I reboot either machine maybe once a month, and during that time FF only restarts maybe once because bartab does such a wonderful job of keeping things in check.

    So i tested FF8 tonight with the exact same tab load as my 3.6. Currently 3.6 with bartab is eating 288MB. FF8 is eating 730meg and this is after about 40 minutes of idle time . Installing memoryfox on FF8 doesn’t appear to change this at all.

    I was really hoping for an unload tab option in FF8, but all i see is the option to keep tabs from loading at startup. While this is a great step forward it seems to me it should have been the last option to implement (if were going for bartab functionality built in). I would much rather see the ability to unload tabs after a time, AND manually via a menu option. Even if my 100 tabs don’t load at startup, eventually a good portion of those are gonna get touched.

    Since we’ve now added the “don’t load on startup” option, what are the chances we’ll get a unload tab option that is user configurable similar to bartab?

    • Nicholas Nethercote

      You’re right, FF8 has a delay-tab-load feature but does not have a tab-unload feature. Implementing a tab-unload feature in a way that guarantees you never lose any important data is hard. https://bugzilla.mozilla.org/show_bug.cgi?id=675539 is open to discuss this feature, but don’t assume that just because there’s a bug filed that it’ll ever actually happen. There’s also an add-on called Dormancy (see comment 30 in the bug) that does this, but I believe it can cause data loss.

      • Yes i actually ran across the page mentioned in post 30 before i found this one. I was a little concerned about the data loss comment, but I’m curious as to what data exactly could be lost. If were talking about form data i would never leave a form filled in but not submitted to go unloaded :)

        Could you give a little more insight into this “data loss” that you say is hard to work around. With bartab I’ve never had a single issue with a missing website. So unless I’m misunderstanding exactly what kind of data loss were talking about i don’t see how the awesome devs at Mozilla couldn’t replicate a simple addon given all the amazing things they’ve done with FF over the years.

        • Nicholas Nethercote

          I don’t know about any BarTab data loss issues, not loading a page seems entirely different to unloading a page.

          You say you would never leave a form filled but unsubmitted, but plenty of other people might. As for the general challenges, I don’t know anything beyond what that bug report says, I’m not an expert on that area.

          • After doing some more reading it seems the possible data loss issue is more geared towards losing entire sessions. For example if i’ve got 100 tabs open then poof they are gone after a restart or whatever.

            For me this isn’t an issue since i use TMP for my session manager, but i can see the potential issue for others that don’t. Although if your running lots of tabs like i am and need something like Bartab i would also recommend another session manager with more features.

            Anyways Nickolas thank you for the discussion i really appreciate it. I guess I’ll just keep my fingers crossed but something tells me unless it comes in the form of a new addon I’ll be stuck with 3.6 until the end of time.