MemShrink progress, week 20

Surprise of the week

[Update: This analysis about livemarks may be wrong.  Talos results from early September show that the MaxHeaps number increased, and the reduction when the “Latest Headlines” livemark was removed has undone that increase.  So livemarks may not be responsible at all, it could just be a coincidence.  More investigation is needed!]

Jeff Muizelaar removed the “Latest Headlines” live bookmark from new profiles.  This was in the Bookmarks Toolbar, and so hasn’t been visible since Firefox 4.0, and most people don’t use it.  And it uses a non-zero amount of memory and CPU.  Just how non-zero was unclear until Marco Bonardo noticed some big performance improvements.  First, in the Talos “MaxHeap” results on WinNT5.2:

Talos MaxHeap graph

And secondly in the Talos “Allocs” results on WinNT5.2 and Mac10.5.2:

Talos Allocs graph

In the WinNT5.2 cases, it looks like we had a bi-modal distribution previously, and this patch changed things so that the higher of the two cases never occurred.  In the Mac10.5.2 case we just had a simple reduction in the number of allocations.  On Linux the results were less conclusive, but there may have been a similar if smaller effect.

This surprised me greatly.  I’ve done a lot of memory profiling of Firefox and never seen anything that pointed to the feed reader as using a lot of memory.  This may be because the feed reader’s usage is falling into a larger, innocuous bucket, such as JS or generic strings.  Or maybe I just missed the signs altogether.

Some conclusions and questions:

  • If you have live bookmarks in your bookmarks toolbar, remove them! [Update: I meant to say “unused live bookmarks”.]
  • We need to work out what is going on with the feed reader, and optimize its memory usage.
  • Can we disable unused live bookmarks for existing users?

Apparently nobody really owns the feed reader, because previous contributors to it have all moved on.  So I’m planning to investigate, but I don’t know the first thing about it.  Any help would be welcome!

 Other stuff

There was a huge memory leak in the Video DownloadHelper add-on v4.9.5, and possibly earlier versions.  This has been fixed in v4.9.6a3 and the fix will make it into the final version v4.9.6 when it is released.  That’s one more add-on leak down, I wonder how many more there are to go.

TraceMonkey, the trace JIT, is no longer built by default.  This means it’s no longer used, and this saves both code and data space.  The old combination of TraceMonkey and JaegerMonkey is slower than the new combination of JaegerMonkey with type inference, and TraceMonkey is also preventing various clean-ups and simplifications, so it’ll be removed entirely soon.

I refined the JS memory reporters in about:memory to give more information about objects, shapes and property tables.

I avoided creating small property tables, removed KidHashes when possible, and reduced the size of KidHashes with few entries.

I wrote about various upcoming memory optimizations in the JS engine.

Justin Lebar enabled jemalloc on MacOS 10.5 builds.  This was expected to be a space improvement, but it also reduced the “Tp5 MozAfterPaint” page loading benchmark by 8%.

Robert O’Callahan avoided excessive memory usage in certain DOM animations on Windows.

Drew Willcoxon avoided excessive memory usage in context menu items created by the add-on SDK.

Bug Counts

  • P1: 35 (-1, +1)
  • P2: 116 (-2, +5)
  • P3: 55 (-2, +3)
  • Unprioritized: 5 (-4, +5)

At this week’s MemShrink meeting we only had 9 bugs to triage, which is the lowest we’ve had in a long time.  It feels like the MemShrink bug list is growing slower than in the past.

25 Responses to MemShrink progress, week 20

  1. <rant>How hiding a problem makes it fixed? The fact that you don’t use LiveMarks (at least that’s how it was called back in the day) does not make them any less useful. LiveMarks is the built-in feed reader without a need for 3rd party tools. Expand a folder with all your LiveMarks in the Bookmarks side bar (ctrl+shift+b) and it’s a full-blown feed reader – right there and ready to use. What can be simpler?

    I just don’t get the urge to fix everything that ain’t broken. </rant>

    • I’m confused.
      At no point was it suggested that LiveMarks be removed. It was simply discovered that the feature is consuming more resources than it should, therefore because almost no one is using the default, complementary Latest Headlines folder which come in the default profile (which is now hidden by default), remove it.
      And if you are concerned about the memory used by this feature (perhaps if you have unused LiveMark folders) then consider removing them.
      In the mean time they are working to resolve the memory issues with LiveMarks so it will become a non-issue.

      • > In the mean time they are working to resolve the memory issues with LiveMarks so it will become a non-issue.

        It’s good to hear that LiveMarks are not being removed and their memory use will be looked at. That’s what I wanted to hear. The reason I made my comment in the first place was to make sure that LiveMakrs would not end up being dropped because they are not visible in the main UI by default.

        I’m aware and fully appreciative of all the great work that goes into making Firefox a leaner software that continues to outperform the competition.

  2. How can we avoid being surprised by things like livemarks in the future? I have a few ideas, but I bet you have better ones!

    * Split frontend code into multiple compartments.

    * Disable frontend features one at a time and subtract to find out how much memory each one uses.

    * Profile JS memory use based on JS allocation sites, or based on the JS data structure graph, rather than at the C++ class level.

  3. Having raised the bug that led to this analysis, I’ve some experience of RSS in bookmarks – and I suggest you modify your recommendation to remove all live RSS feeds from the bookmarks toolbar.

    I removed the default Latest Headlines from the bookmarks folder (I think one was there but not 100% sure) and replaced it with about 8-10 feeds which I use regularly. I can live with the memory growth and close FF about twice per week for other reasons.

    My suggestion is that you recommend people to remove Latest Headlines and other live feeds they they do not use

  4. Way way back in the day (2007) I found a memory leak from simply loading an RSS feed via the RSS discovery icon:

    https://bugzilla.mozilla.org/show_bug.cgi?id=398270

    I wonder if the same methodology is still applicable today?

  5. @WalterK – Go back and read it again properly! The feed reader is broken (uses excessive memory and CPU) and they’re looking into ways of making it more efficient :)

  6. With the bookmark toolbar hidden, how do existing users go about getting rid of live bookmarks if we don’t use them?

    • Nicholas Nethercote

      On a Mac: click the “View”/”Toolbars”/”Bookmarks Toolbar” menu item to show the bookmarks toolbar. Then right-click on the “Latest Headlines” button (and any other live bookmarks) and choose “delete”.

      On Windows the menu location will be slightly different, but shouldn’t be too hard to find.

  7. Since I don’t know where else to post it, I have noticed huge memory usage (going up to nearly 3 GB) in the nightly-ux channel, while the memory usage is under 1 GB in the nightly builds, with the same tabs open, the same add-ons and approximately the same time of usage.

  8. Thank you for all the great work on reducing Firefox memory. I removed the live bookmarks for the “Latest Headlines” from the Firefox toolbar. You also wrote that we should remove other live bookmarks if we use any. Is there an easy way to disable them for now so that we can add them back when the memory leak issues are solved?

    Also, I noticed that the title for today’s report at https://wiki.mozilla.org/Performance/MemShrink is incorrect. It shows a date of 2011-10-25 but I think it should be 2011-11-02.

    Thank you,
    Peter

    • Nicholas Nethercote

      I think the only way to disable them is to delete them from the bookmarks toolbar or sidebar.

      I fixed the date, thanks!

      • I use live bookmarks, and have not yet tried comparisons in profiles with and without them. Again not tried (I will probably not have free time now until the weekend ) but does and setting a preference help I am wondering if creating and changing the value of that preference affects to any extent, when and how much of a performance hit firefox suffers.

        I note the comment preview now displays :-)
        But it does not display how html anchors display :-(
        I will repeat the link, this does display in preview: http://kb.mozillazine.org/Browser.bookmarks.livemark_refresh_seconds

        Thanks again great work, and it is nice that you write about it.

  9. Thanks for all the grate work and the weekly updates. I’m a heavy RSS feed user with 30+ feeds! I can’t live without it! I have them along the menu bar for quick access. It’s the best thing since tab browsing! Please Do Not Remove Feed reader. It would be nice if there was a way to disable it for further testing on the nightly channel without having to delete all the feeds. I miss the orange icon for RSS feeds in the Address Bar . Would anyone elese like this back?

    • Nicholas Nethercote

      Don’t worry, the feed reader isn’t going to be removed. All we’ve done is remove the “Latest Headlines” feed from new profiles.

      I wrote “if you have live bookmarks in your bookmarks toolbar, remove them!”, but I meant to say “unused live bookmarks”. I’ve added a note to the post about this.

      As for the orange icon, I believe it was removed from the default configuration because user studies showed it was barely used. You can add it back in by right-clicking on the navigation toolbar (e.g. in the grey selection just below the URL bar), selecting “customize” and then dragging the icon into the navigation toolbar.

  10. I’m not great at tracking down regressions, but maybe you should look at what happened on September 8th.

    http://graphs-new.mozilla.org/graph.html#tests=29,63,8&sel=none&displayrange=90&datatype=running

  11. Nick, your updates are phenomenal and give me greater hope for the future of Firefox and Mozilla.

  12. Nick, if you’re ever up in Brisbane, I’d love to buy you a drink as thanks for all your hard work minimising Firefox’s memory footprint. :)

  13. I don’t really understand the graphs to interpret how much memory is used over time by having live bookmarks. At what rate is memory lost by having the “Latest Headlines” live bookmark? How much memory would be lost for each hour that Firefox is running with this live bookmark. Before yesterday I had 23 live bookmarks for Firefox. How much memory would be lost per day in the 10 hours that Firefox is running? Does the amount of memory lost depend on the amount of articles found in each live bookmark? Thank you.

    • Nicholas Nethercote

      The answer to almost all your questions is “I don’t know”. Note that each dot on the graph shows a single measurement, which is the average memory usage when running a particular series of pageloads. The drop corresponds to the commit where the default livemark was removed. It’s not even clear if removing the livemark was truly the cause for the drop, or if that triggered something else.

  14. Are you sure that these measurements are tracking a series of pageloads?

  15. Nicholas, I just want to apologize for me constantly posting about:memory usage and blaming things are slow because it uses too much memory ( Although it is true ), I *think* i found the culprit to all the slowness.

    I am heavy on Tabs, On a 2GB memory machines, everytime i see Firefox using 1.2 – 1.4GB of Memory I know it has used it all up and are paging to VM for more memory. CPU Spike was high and I have no idea what was going on. I have always assume that memory is low and GC was running like mad and causes problem.

    One day i just thought i haven’t defrag my HDD for a long time. So i decide to do a full defrag ( Offline + Online ), after that even when i am on the same set of pages, everything was way more responsive. And i discover there seems to be lots of file inside profile that gets easily fragmented.

  16. I would also like to know if this
    https://bugzilla.mozilla.org/show_bug.cgi?id=669034

    Belongs to Memshrink. ( Large session restore )

    • Nicholas Nethercote

      Randell (the bug filer) attends MemShrink meetings, I think we discussed this one at one point but decided it didn’t need to be a MemShrink bug.