MemShrink progress, week 8

A hodge-podge of things happened this week.

The MemShrink bug counts changed as follows.

  • P1: 29 (-2, +4)
  • P2: 62 (-5, +14)
  • P3: 41 (-0, +6)
  • Unprioritized: 2 (-10, +1)

We actually got through almost all the unprioritized bugs in today’s MemShrink meeting, which was good, but the counts are still going up.  Fortunately, most of the new bugs are ideas for improvement that are reported by developers.  My gut feeling (which I get from reading a lot of memory-related bug reports) is that the number of reports from users about high memory usage are much lower than they were a few months ago.

26 Responses to MemShrink progress, week 8

  1. Great work again. Today is the first day i encountered heap-unclassified with nearly 50% of my mem usage for some reason. Hopefully as your post state this will be reduce dramatically in Firefox 7. And further reduced with future release. I am currently using version 6 Beta.

    The other thing which concerns me is JS memory. heap-unclassified was only a problem from time to time. But JS is usually one of the largest “memory consumer.” Or specifically gc-heap, which has very high memory usage.

    Are there any plan to work on JS Engine memshrink?

    • Nicholas Nethercote

      Ed: heap-unclassified probably won’t drop dramatically in FF7 or any later released. But it’ll go down gradually, esp. thanks to https://bugzilla.mozilla.org/show_bug.cgi?id=676724.

      MemShrink encompasses everything in the browser, including the JS engine. The two biggest improvements we’ve seen so far both improved gc-heap drastically. Hopefully we’ll get more JS improvements from GC improvements and other fixes.

  2. Are most of these new improvements that you have blogged about in the last couple of weeks going to end up in (the already vastly improved) Firefox 7? Or only in Firefox 8?

    Do you think there’ll be a significant difference (improvement) between 7 and 8′s memory usage too (if not as great as between 6 and 7)?

    BTW this must be the most popular Mozilla blog these days :)

    • Nicholas Nethercote

      Curious: The release calendar is at https://wiki.mozilla.org/RapidRelease/Calendar. The FF7 development period ended on July 5th, so everything I’ve described for the past 5 weeks will only be in FF8. The FF8 development period ends on August 16.

      FF8 should be a bit better than FF7. See the “average resident” and “peak resident” graphs at https://wiki.mozilla.org/RapidRelease/Calendar; and those measurements were taken a couple of weeks ago, so FF8 may have improved a bit more since then.

    • @Curious

      check the bug entries and you see that most target Firefox 8. So get the latest nightly (I run them as portable) to see what is new and improved.

      • Andre how do “run them as portable”? I just had a quick look at portableapps.com and there’s no sign of nightly builds. Are you using the zip version and running a command line flag to store the profile somewhere else? I find that creating a new profile the normal way interferes with my normal build. How do you do it?

        • I use the loader from here:

          http://ftp.hosteurope.de/mirror/stadt-bremerhaven.de/Portable_Firefox/

          download the latest nightly (ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/) as zip and replace the firefox folder with the folder from the latest nightly and run the loader (FirefoxLoader.exe). the profile data must be copied to the folder “Profilordner” (sorry, this is a German Firefox loader).

          With this I run the Aurora and nightly as portable and the firefox 5 as installed browser (release channel).

          • Thanks Andre. Sounds like a solution that works for you but which might be a bit fiddly for the average joe.

            Nick is there any thoughts in Mozilla circles towards creating portable versions of the various Firefox release channels? Over time, I’ve seen people have trouble trying to run different builds on the same Windows install without screwing up their day-to-day existing install and profile.
            Portable versions could finally solve that problem and perhaps contribute to an increase in the number of testers, don’t you think?

          • @pd

            simply use on of the portable exe (like Portable_Firefox_5.0.1.exe), run the sfx exe to extarct it and replace the firefox folder with the ZIP. for me it works fine. I always test newer versions (especially when migrated from 3.6 to 4 which changed a lot) this way.

  3. Nicholas Nethercote

    pd: I’m not the person to ask about channel stuff :)

  4. Having some research and lots of news, this is a 143 Tabs using Firefox 6 beta

    1,453.78 MB (100.0%) — explicit
    ├────751.33 MB (51.68%) — heap-unclassified
    ├────517.75 MB (35.61%) — js
    │ ├──372.00 MB (25.59%) — gc-heap
    │ ├───84.59 MB (05.82%) — mjit-code
    │ ├───41.37 MB (02.85%) — tjit-data
    │ │ ├──28.93 MB (01.99%) — allocators-reserve
    │ │ └──12.43 MB (00.86%) — allocators-main
    │ ├───12.41 MB (00.85%) — mjit-data
    │ └────7.38 MB (00.51%) — tjit-code
    ├─────79.45 MB (05.47%) — layout
    │ ├──79.45 MB (05.47%) — all
    │ └───0.00 MB (00.00%) — (1 omitted)
    ├─────69.50 MB (04.78%) — storage
    │ └──69.50 MB (04.78%) — sqlite
    │ ├──38.03 MB (02.62%) — places.sqlite
    │ │ ├──37.69 MB (02.59%) — cache-used
    │ │ └───0.34 MB (00.02%) — (2 omitted)
    │ ├──25.72 MB (01.77%) — urlclassifier3.sqlite
    │ │ ├──25.64 MB (01.76%) — cache-used
    │ │ └───0.08 MB (00.01%) — (2 omitted)
    │ └───5.75 MB (00.40%) — (13 omitted)
    └─────35.76 MB (02.46%) — images
    ├──35.34 MB (02.43%) — content
    │ ├──35.18 MB (02.42%) — used
    │ │ ├──18.08 MB (01.24%) — raw
    │ │ └──17.10 MB (01.18%) — uncompressed
    │ └───0.16 MB (00.01%) — (1 omitted)
    └───0.41 MB (00.03%) — (1 omitted)

    Other Measurements
    1,582.82 MB — private
    1,553.05 MB — resident
    1,390.82 MB — heap-committed
    1,361.81 MB — heap-used
    31.18 MB — heap-unused
    22.65 MB — gfx-surface-win32
    3.23 MB — heap-dirty
    1.75 MB — canvas-2d-pixel-bytes
    0.21 MB — shmem-allocated
    0.21 MB — shmem-mapped
    0.00 MB — gfx-d2d-surfacecache
    0.00 MB — gfx-d2d-surfacevram
    0.00 MB — gfx-surface-image

  5. I was unable to use HTML Tag well. Sorry!!

  6. is there way to send memory usage information or dump file from withing firefox beta channel?, started my firefox with 3 tabs opened and closed like 15 tabs in last 5hours, firefox memory usage is 1048MB which quite huge. let me know if there is way to submit all the information you would need from my side, let me know.

    here are the details from about:memory page,

    1,044.44 MB (100.0%) — explicit
    ├────473.99 MB (45.38%) — heap-unclassified
    ├────421.93 MB (40.40%) — storage
    │ └──421.93 MB (40.40%) — sqlite
    │ ├──404.36 MB (38.72%) — places.sqlite
    │ │ ├──399.81 MB (38.28%) — cache-used
    │ │ └────4.55 MB (00.44%) — (2 omitted)
    │ ├────8.16 MB (00.78%) — urlclassifier3.sqlite
    │ │ ├──8.07 MB (00.77%) — cache-used
    │ │ └──0.08 MB (00.01%) — (2 omitted)
    │ ├────6.29 MB (00.60%) — other
    │ └────3.12 MB (00.30%) — (12 omitted)
    ├────136.68 MB (13.09%) — js
    │ ├───52.97 MB (05.07%) — compartment([System Principal])
    │ │ ├──21.33 MB (02.04%) — gc-heap
    │ │ │ ├───9.75 MB (00.93%) — objects
    │ │ │ ├───5.91 MB (00.57%) — shapes
    │ │ │ └───5.67 MB (00.54%) — (5 omitted)
    │ │ ├──12.61 MB (01.21%) — string-chars
    │ │ ├──11.00 MB (01.05%) — mjit-code
    │ │ └───8.03 MB (00.77%) — (5 omitted)
    │ ├───25.13 MB (02.41%) — compartment(https://mail.google.com/mail/?ui=2&shva=…)
    │ │ ├──10.32 MB (00.99%) — gc-heap
    │ │ │ └──10.32 MB (00.99%) — (7 omitted)
    │ │ ├───8.68 MB (00.83%) — (6 omitted)
    │ │ └───6.13 MB (00.59%) — mjit-code
    │ ├───17.12 MB (01.64%) — (18 omitted)
    │ ├───12.54 MB (01.20%) — gc-heap-chunk-unused
    │ ├───10.75 MB (01.03%) — compartment(https://plus.google.com/u/0/_/notificati…)
    │ │ └──10.75 MB (01.03%) — (8 omitted)
    │ ├────9.53 MB (00.91%) — compartment(http://www.google.com/reader/view/#strea…)
    │ │ └──9.53 MB (00.91%) — (8 omitted)
    │ └────8.66 MB (00.83%) — compartment(atoms)
    │ ├──6.21 MB (00.59%) — string-chars
    │ └──2.45 MB (00.23%) — (7 omitted)
    ├──────6.41 MB (00.61%) — images
    │ ├──6.14 MB (00.59%) — content
    │ │ ├──6.14 MB (00.59%) — used
    │ │ │ └──6.14 MB (00.59%) — (2 omitted)
    │ │ └──0.00 MB (00.00%) — (1 omitted)
    │ └──0.27 MB (00.03%) — (1 omitted)
    └──────5.43 MB (00.52%) — (2 omitted)

    Other Measurements
    1,409.40 MB — vsize
    1,104.77 MB — private
    1,074.40 MB — resident
    1,037.61 MB — heap-committed
    1,018.69 MB — heap-used
    64.00 MB — js-gc-heap
    23.31 MB — heap-unused
    4.02 MB — gfx-surface-win32
    3.39 MB — heap-dirty
    0.04 MB — gfx-surface-image
    0.00 MB — gfx-d2d-surfacecache
    0.00 MB — gfx-d2d-surfacevram
    0.00 MB — shmem-allocated
    0.00 MB — shmem-mapped
    0.00 MB — canvas-2d-pixel-bytes

    • Your places.sqlite is absolutely anomalous. That might related to some extension. When you install Firefox Aurora you normally get the “test pilot” extension, and a prompt at first run to send some automated performance feedback.

    • your SQLite files are too large. When I had the large memory issues with Firefox 3.6 (over 1GB memory usage) I made a rebuild the profile (fresh profiule, installed all addons again and put all settings into the pref file) to reduce the memory usage.

  7. Benjamin Llie

    Hi Nick i would like to say that flash and java plugins are memory hogs in their own so i was hoping, adding a good click to play option integrated to the browser would be a good measure. disabled by default but the installation should give a quick reference and a option to activate it.

  8. We just have the example of yet another “memory” bloat issues that is totally unrelated to Firefox itself.

    420MB for SQLite is insane. I was thinking the other day that even 70MB for me is too large and we need to look for way to efficiently store our data.

  9. Dunno who else to mention this to, but I don’t think we should prominently mention claims about improved performance and memory in marketing for FF6.
    People have just stopped believing us.
    We can say it loud for FF7 though.

    • Nicholas Nethercote

      Caspy7: AIUI, performance won’t be mentioned in the FF6 marketing, instead it’ll focus on web development tools (like the new scratchpad).

  10. Here the memshrink is not working so good. with the latest nightly from 22:00 to 10:00 it jumped from 500mb to 1000mb, without usage. I know it´s a know problem, firefox eating memory for midnight dinner, but I tought that was at least close to solved in firefox 7. Unfortunately to me the problem remains basically the same as with firefox 4, 5, 6.

    • Nicholas Nethercote

      Fernando, if you can file a bug in Bugzilla with full details — e.g. what sites were open, what add-ons were installed — that would be helpful.

      • I was using the latest nightly of firefox 7. After reading a little it came to my knowledge that you´ve said most of the memshirnk was not in firefox 7 and would only be in action in 8. I have just installed 8 alfa 2 and will see if the midnight dinner still happens.