MemShrink progress, week 3

This was the final week of development for Firefox 7, and lots of exciting MemShrink stuff happened.

Per-compartment memory reporters

I landed per-compartment memory reporters, using the new memory multi-reporter interface.  I’ve written previously about per-compartment reporters;  the number of things measured has increased since then, and each compartment now looks like this:

Output of a per-compartment memory reporter

One nice thing about this feature is that it gives technically-oriented users a way to tell which web sites are causing high memory usage.  This may help with perception, too;  people might think “geez, Facebook is using a lot of memory” instead of “geez, Firefox is using a lot of memory”.

Another nice thing is that it can help find leaks.  If you close all your tabs and some compartments are still around, that’s a leak, right?  Actually, it’s more complicated than that:  compartments are destroyed by the garbage collector, so there can be a delay.  But there are buttons at the bottom of about:memory which force GC to occur, so if you press them and some compartments are still around, that’s a leak right?  Well, it appears that sites that use timers can stick around while the timers are still live.  For example, TBPL appears to have a refresh timer that is 2 or 3 minutes long, so it’s compartment stays alive for that long after the page is closed.  (Actually, it’s not entirely clear if or why that should happen.)  However, once you’ve accounted for these complications, any remaining compartments are likely to have leaked.

Another thing to note is that the JS heap space used by each compartment is now reported.  Also, the fraction of the JS heap that isn’t currently used by any particular compartment is also reported:

Output of non-compartment GC memory reporters

Once this change landed, it didn’t take long to see that the “gc-heap-chunk-unused” number could get very large.  In particular, if you browsed for a while, then closed all tabs except about:memory, then triggered heap minimization (via the buttons at the bottom of about:memory), you’d often see very large “gc-heap-chunk-unused” numbers.  It turns out this is caused by fragmentation.  The JS heap is broken into 1MB chunks, and often you’d end up with with a lot of almost-empty chunks, but they’d have a small number of long-lived objects in them keeping them alive, and thus preventing them from being deallocated.

Reduced JavaScript heap fragmentation

Fortunately, Gregor Wagner had already anticipated this, and he tightened his grasp on the 2011 MemShrink MVP award by greatly reducing this fragmentation in the JavaScript heap.  Parts of Firefox itself are implemented in JavaScript, and many objects belonging to Firefox itself (more specifically, objects in the “system principal” and “atoms” compartments) are long-lived.  So Gregor added simple chunk segregation;  objects belonging to Firefox get put in one group of chunks, and all other objects (i.e. those from websites) get put in a second group of chunks.  This made a huge difference.  See this comment and subsequent comments for some detailed measurements of a short browsing session;  in short, the size of the heap was over 5x smaller (21MB vs. 108MB) after closing a number of tabs and forcing garbage collection.  Even if you don’t force garbage collection, it still helps greatly, because garbage collection happens periodically anyway, and longer browsing sessions will benefit more than shorter sessions.

This change will help everyday browsing a lot.  It will also help with the perception of Firefox’s memory usage — once you learn about about:memory, an obvious thing to try is to browse for a while, close all tabs, and see what the memory usage looks like.  Prior to this patch, the memory usage was often much higher than it is on start-up.  With this patch, the usage is much closer to the usage seen at start-up.  Ideally, it would be the same, and Justin Lebar opened a bug to track progress on exactly this issue.

There’s still room for reducing fragmentation in the JavaScript heap further.  Kyle Huey has some good ideas in this bug.

Other bits and pieces

  • Chris Pearce finished a major refactoring of media code.  Previously, every media (<audio> and <video>) element in a page required three threads (four on Linux).  And each thread has a stack.  On Linux, the stack is 8MB, on Windows it’s 1MB, on Mac it’s 64KB.  (Edit: Robert O’Callahan pointed out in the comments that this space is not necessarily committed, which means they’re not really costing anything other than virtual memory space.)  Webpages can have dozens, even hundreds of media elements, so these stacks can really add up.  Chris changed things so that each media element has a single thread.  This really helps on complicated web sites/apps like The Wilderness Downtown, ROME, and WebGL Quake.  (There is also a bug open to reduce the thread stack size on Linux;  8MB is excessive.)  Chris only just landed these patches on mozilla-inbound, he deliberately waited until after the FF7 deadline to maximize testing time.  Assuming there are no problems, this improvement will be in FF8.
  • Jesse Ruderman tweaked some of his fuzzing tools to look for leaks and found four new ones.
  • Mounir Lamouri landed some memory reporters for the DOM.  They’re currently disabled while he extends them and makes that they’re reporting sensible numbers.
  • I created mlk-fx8, a bug for tracking leaks and quasi-leaks reported against Firefox 8.

Quantifying progress

I’m going to attempt to quantify MemShrink’s progress in these weekly reports.  It’s not obvious what’s the best way to do this.  The MemShrink wiki page discusses this a bit.  I’m going to do something crude:  just count the number of bugs annotated with “MemShrink”, and track this each week.  Here are the current counts:

  • P1: 18
  • P2: 44
  • P3: 25
  • Unprioritized: 2

Obviously, this is a flawed way to measure progress;  for example, if we improve our leak-detection efforts, which is a good thing, we’ll get more bug reports.  But hopefully these numbers will trend downwards in the long term.

I’m also wondering if my choice to have three priority levels was a mistake, if only for the reason that people like to pick the middle item, so P2 will always dominate.  If we had four priority levels, we’d be forced to prioritize all those middling bugs up or down.

The minutes of today’s meeting are available here.  Thanks to Jesse for taking them.

42 Responses to MemShrink progress, week 3

  1. Robert O'Callahan

    You might want to clarify that those thread stack sizes are VM space only, not necessarily committed memory.

  2. Have i misunderstood ( again ) the dev process? Shouldn’t it be final week of development for Firefox 6? Since Firefox 7 continue to live in Nightly?

  3. Nicholas Nethercote

    Ed, Firefox 7 graduated from Nightly to Aurora just a few hours ago. See https://wiki.mozilla.org/RapidRelease/Calendar.

    And when I say “development” I mean “active anything goes development”, ie. what happens on the Nightly channel, as opposed to the very few and conservative changes made on the Aurora and Beta channels.

  4. Sorry i was not having enough coffee. Damn it Does that mean we wont even get this bugfix in Firefox 7?

  5. Nicholas Nethercote

    Ed: which bugfix? Bug 666058 made it in.

  6. OMFGawd. FINALLY resolution to the chrome vs content JS dilemma! This is insanely brilliant. Gregor and Nicholas you guys are dual MVP’s IMHO – sensational. In the past I’d never think of going to a nightly unless using a test machine. Now I’m going to risk it, using Add-On Compat Reporter, for the benefits of these incredibly super memory management improvements.

    Mozilla marketing department? Crank the truth/spin dial towards truth for the Firefox 7 campaign! Finally ‘improved memory management’ will mean something!

    Nicholas do you think it is feasible to get Add-On memory consumption reporting in the Firefox 8 cycle?

    • Nicholas Nethercote

      pd: measuring add-on memory consumption is hard, unfortunately. Firefox lets add-ons hook their claws in so deeply it’s difficult to know where Firefox ends and the add-on starts. I wish I knew a way to do it, because it would render the “are you running add-ons? have you disabled them?” routine moot.

  7. Thanks for the release info.

    Real shame about Add-On memory exposure. Is that a problem that can be improved with JetPack? I know not every Add-On is (or perhaps will ever be) a JetPack-base Add-On however when I looked at JetPack earlier in the year, they had some memory-analysis tools in the SDK. I’m not C++ guru but does that suggest that the JetPack API’s are more memory reporting friendly? Do you think that memory reporting, on at-least JetPack-based Add-Ons, would be a good start?

    If the memory usage of specific Add-Ons cannot (yet) be identified (even JetPack-based Add-ons), is it possible to at least drill-down to memory usage of JavaScript modules? AFAIK all JetPack APIs are built on JSMs aren’t they? Maybe the existing reporting in about:memory covers this?

    I’m just throwing up some strands of vague knowledge. I hope they are useful but understand if they aren’t :)

    Thanks for your time.

    Keep up the great work. Magnificent.

    pd

  8. Thanks for this follow-up and all the work on memory. Progresses are impressive and fast, that’s good to see.

    “…Well, it appears that sites that use timers can stick around while the timers are still live. For example, TBPL appears to have a refresh timer that is 2 or 3 minutes long, so it’s compartment stays alive for that long after the page is closed. (Actually, it’s not entirely clear if or why that should happen.)”
    => I’m not familiar with Firefox code, but there is no reason for this to happen. When the page is closed, timers won’t fire (at least, it won’t be observable, which is basically the same thing), so remaining timers should be disabled as soon as page close occur, I think. Is there a bug open on that?

  9. I may be speaking out of place here, but you said that addons claw their way deep, but isn’t their a way to make sure they leave a mark? I was thinking of maybe having addons be forced to run on a separate dependent process similar to the plug-in container. This one for addons may be depended on the core firefox process, but if you get core-firefox (the vanilla process) stable and see you addons rising in memory usage you can narrow it down quite a bit I would say, without having to switch profiles or run separate builds.

    • Nicholas Nethercote

      pd, DeathShot: the best info I have about add-ons is from http://groups.google.com/group/mozilla.dev.platform/msg/b3107a003d96d88f:

      “We are actively working on moving at least jetpack-style addons into a separate process. This was part of the original goal for the Jetpack SDK 1.0 but was not feasible this time. But even there, the addon will also be able to run code in the context of content (content scripts), so we’ll have to do some combination of measurements. It probably isn’t feasible to do this work for old-style addons because they are basically chrome and much of their code runs in the main browser window global.”

      So, there’s hope there, but I don’t know how to measure even jetpack-style addons right now, without process separations.

  10. nice to hear this. I had 700MB memory usage today on Firefox 5.0 and Firefox was extremely slow. I hope it is better in Firefox 6 and 7.

    Btw, when do we officially get a 64Bit Firefox under Windows? This fixes some memory issues (fragmentation)

  11. David Illsley

    I thought each jetpack addon was getting its own compartment… no?

    • Nicholas Nethercote

      David: I think I said each jetpack addon was getting its own compartment in a previous blog post, but I may have been wrong about that. I’m going to investigate more today.

  12. I hope any changes to how add-ons are run and compartmentalized don’t interfere with the ability of add-ons like DownloadHelper and AdBlock Plus to catch subrequests from Flash. The fact that Chrome apparently doesn’t allow that (I suspect it’s endemic to the model they use) is one of my Giant List of Gripes about that browser and one of the reasons I’ll never switch. Firefox’s add-on model is far superior when it comes to the capabilities allowed.

    • Nicholas Nethercote

      Lummox JR: AIUI the people working on process separation are working very hard to ensure that add-ons can continue doing all the things they currently do. I can’t guarantee they’ll succeed, but AdBlock Plus is the #1 add-on so I expect they’ll ensure it can continue to work as well as it currently does.

  13. I mean the 32Bit VA 2GB fragmentation due to alloc/freeing memory. With firefox 3.5 the fragentation view of VMMap looked horrible. I only had very small pieces of free VM and Firefox was extremely slow allocating memory. But I see that you enabled the large address aware flag, so I have 4GB VA space on a 64Bit Windows. When was this changed? The last time i checked this, it was not enabled. So this

    For the 64Bit:

    “Risks

    - Could accidentally give 64 bits builds to users who can’t run them
    - 64bit user experience could be worse (with plugins, etc) and reflect poorly on the overall product ”

    With this we never get a 64bit version. If no 64bit version is released, software vendors will never build 64bit plugins. Make it available as an optional download. Link to 32bit by default and advanced users can download the 64bit version.

  14. I think Nicholas is a pretty cool guy. eh kills laeks and doesn’t afraid of anything…

  15. Alexander Nolting

    Hello,

    I recognized that FF8 with the same number of open tabs uses the double amount of Mem on Ubuntu 11.04 64bit compared to FF7. Is there a way for Non-Developers to see what happen?

    Alex

    • Nicholas Nethercote

      Alexander: thanks for the data point. That sounds like this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=669815, which I’m monitoring *very* closely, and will investigate further today. If you look at about:memory (or better, about:memory?verbose) you’ll be able to see a comparison between FF7 and FF8.

    • Nicholas Nethercote

      Alexander: can you tell me what the workload was where you saw the 2x regression? I.e. what tabs did you have open? And how are you measuring memory usage? If you can comment directly in https://bugzilla.mozilla.org/show_bug.cgi?id=669815 that would be very helpful.

    • Nicholas Nethercote

      Alexander: actually, you should see bug 669815 only if you are using FF8 builds from the tracemonkey repository. If you are using standard FF8 builds from mozilla-central, that must be another regression. Knowing the workload would be *really* useful in that case.

  16. I have a strange scenario from early Aurora 7.0a2 use. Today I left my browser running for at least 10 hours with just 12 tabs open. 6 tabs were from the same site. Anyway it appears the memory usage climbed whilst I was not using the browser to 444MB or 960MB. Then I click on to the browser again and read one tab, close it, and the browser drops to 300MB! This is the tab:

    http://news.bbc.co.uk/sport2/hi/formula_one/14076892.stm

    as you can see, minimal content really.

    Then I load this site and the memory crept up to 400MB again whilst it’s now sitting at 351MB.

    I might be misreading things but I thought the new garbage collection patch(es) were supposed to reduce, if not eliminate, the Firefox habit of increased memory use even when it’s not active. Is this true? FWIW none of the sites I have loaded are contain Flash or other rich media objects and I’ve AdBlocked all ads.

    about:memory says the biggest memory users ATM are:

    40.14 MB (15.89%) — gc-heap-chunk-unused
    103.42 MB (40.93%) — heap-unclassified
    114.27 MB (45.22%) — js

    All compartments attribute to specific URLs actually make up much less than the 45% taken by the entire js tree.

    Is this information helpful? If not, how can I make it more useful and if this is not the right forum, what is? I have ‘submit perf data’ ticked. Maybe that’s enough?

    • Nicholas Nethercote

      Today I left my browser running for at least 10 hours with just 12 tabs open. 6 tabs were from the same site. Anyway it appears the memory usage climbed whilst I was not using the browser to 444MB or 960MB. Then I click on to the browser again and read one tab, close it, and the browser drops to 300MB!

      pd, that’s not quite enough data for me to tell if it’s unexpected. First, how did you measure the memory usage — with about:memory, or some external program? Second, what were the sequences you saw? You say “444MB or 960MB”, that’s a big difference. Do you know what the memory usage was before you left the machine idle, what it was at the end of that idle period, and what it dropped to? Any extra data would be helpful. Also, if you could say what websites you had open while the machine was idle, that would be helpful. Thanks.

      An aside: you say that BBC website has “minimal content”. Although it doesn’t have a great deal of content that is visible, you don’t know what’s happening beneath the surface. Many sites have huge amounts of JavaScript doing all sorts of complicated stuff, much of it relating to advertising. So don’t be misled by the simple appearance.

  17. Hi Nicholas

    I first would like to thank you for your memshrink work that you have been doing and driving. I would like to say upfront that I am just Computer enthusiast with no programing knowledge. Firefox have been my default browser since I version 1 and I have mostly been enjoying using it. I was curious about Firefox 4 and sometime during the development I started following the development by visiting the Mozilla blog. I also run Firefox 4 betas and was really pleased with the progress and potential. I cannot tell you how disappointed I was with the final product and some of the decisions comments and the management of the rapid release process.

    I thought the new release cycle would be great news for users. The user experience of Firefox have a lot to be desired. When opening multiple tabs they froze for several seconds until they are loaded. When closing tabs you get errors in scripts and the user experience have a lot of inconsistency in its design. I was really hoping that with the rapid release cycle some of these issues will be a priority and by fixing these issues in Firefox 5 and 6 would be closer to the browser that Firefox 4 should have been!

    Sadly, the initial drive have been to just lump more features in and it did not add much in the way of spit and polish. That is really why I am writing, to thank you for memshrink, because memory usage is a long and old problem of Firefox and thanks to your work it is getting to attention that is needed.

    In that spirit I would like to draw your attention to flash games in Facebook and the Firefox plugin container memory usage. It is not a strange occurrence to see 1.5gb of memory being used and once all tabs have been closed to still see 300mb left. If you would like to use my Facebook gaming account to witness some of these behaviour first hand I would be more than willing to help.

    Please keep on driving memory leaks down, it is the first step in making Firefox more appealing to people…

    • Nicholas Nethercote

      thinus: thanks for the encouragement. Firefox 4 was an important but flawed release. You identified some valid problems (memory usage, freezing, script problems, etc) but you seem to attribute those problems to the rapid release cycle, which I think is not fair. Firefox 5 was the first rapid release, it didn’t have many changes in it because the process was new and still being worked out. I think Firefox 6 and particularly 7 will start to make the spit and polish improvements you mentioned. And things should just continue to get better, with a new release every 6 weeks, after that.

  18. Thanks for this amazing MemShrink work! Recent nightlies show fantastic improvement in mem usage patterns.

    I’d like to second pd’s request for any tool that’ll allow tracking addon’s resources usage.

    As an addon developer (SmoothWheel) and as a user of many addons myself, understanding which addons affect mem usage negatively can go a long way IMO towards improving many users’ overall Firefox experience.

    I’m not proficient with Firefox internals, but wouldn’t it be possible to show resource usage by chrome paths/files? or even specific objects? Maybe expand Venkman and/or DOM Inspector to include such info?

  19. Hi Nichaolas. That was meant to be 444 of 960MB. My mistakes, typo.

    Anyway that pales into insignificance now. Aurora 7.0a2 just ate all my memory to the point of Not Responding! I only had 19 tabs open, one of which was a mere server not found. One was gmail. the rest are fairly non-intensive pages. I’m using a task manager replacement to determine overall memory usage. This output from it might help:

    Process name: firefox.exe,
    Status: Not Responding
    Private bytes: 936.09 MB
    Peak Working Set: 618.89
    Peak Private Bytes: 988.65
    Page faults: > 8.5 million

    Perhaps some good news is I rode our the Not Responding (despite fearing the effect on my hard drive of letting it get thrashed) hang and closed the Sage extension, no difference. Closed an automatically updating (very minimal data though) router status tab. Closed a server not found tab. Closed a torrent tails tab and for some reason the above now is:

    Process name: firefox.exe,
    Status: Running
    Private bytes: 638.72 MB
    Peak Working Set: 618.89 MB
    Peak Private Bytes: 988.65
    Page faults: > 9.1 million

    Additional the process’s Virtual Size is 1.04 GB.

    Remember I’ve only got 1GB of memory on this machine, less the video card shared memory and I’m left with 960MB.

    I just unclosed all the tabs I referred to above (so I could remember what they were in order to write this comment) and added this site in another tab. I’ve got about 20 tabs and the memory according to the Task manager replacement I’m using (Process Hacker) is still low 600 MBs.

    What explains this eating memory until there’s none left, when I’m not even using the browser and when the browser still operates @ low 600s with almost the same tabs open, after the hang recovery?

    I’ll now look at your response to my earlier comment (which feels inconsequential when I was pointing out just a mere 144MB ‘idle’ increase in use).

  20. Hi Nicholas

    I’m a web developer so I have some perspective on what a heavy and light site is. I block ads and Flash content using AdBlock and run NoScript so the content I load on any given site is fairly well controlled. This BBC site was a singular story page, not a front page or sub-section page and they have geolocation censorship anyway where they do not serve up any rich media to non-British viewers.

    If there was a simple yet reasonably anonymous way I could report the exact tabs and extensions I have open and their memory use when I’ve experienced significant idle memory use growth or when the browser has eaten all my available memory, I’d do that. With Telemetry – like crash data – is there any way to list the data in an about: page, from which I can choose to make notes against certain entries and perhaps upload or send that data to a particular place or person?

    Gmail compartment is only accounting for 22.6MB at the moment yet I have 19 tabs open and the browser is using 659 MB Private Bytes of 960 MB physical available memory (according to Process Hacker). Windows Task Manager says I’m using 474 MB though, so who to trust?

  21. Hmm, I forgot to mention that in addition to the 20 sh tabs I had open, another 11 were open within 3 different tab groups, hidden by Panorama. Not sure if that is relevant. FWIW the tabs do not even have thumbnails when I switch to Panorama. Does this mean they are inactive/idle?

    • Nicholas Nethercote

      pd: you’ve given a mixture points there, but it’s almost impossible to know what’s going on without clear steps to reproduce. This includes listing what add-ons you have installed, what sites you’re accessing, and details about the sequence that you’re visiting them in. Also, the output of about:memory at key points would be enormously helpful. If you can narrow the bad behaviour down to a smaller number of sites, that also helps greatly. If you’re happy to provide that information, it would be best to file a bug in Bugzilla; that’s a better place to report problems than the comments of my blog.

      Telemetry doesn’t provide anything as sophisticated as you suggested. It’s not really designed to aid in single bug reports, it’s aimed instead at aggregating data from many users (eg. what’s the average start-up time) and trends over time (has the average start-up time increased/decreased in version N+1 vs. version N).

  22. They banned me from Bugzilla. What I described is hardly sophisticated. It’s typical of Mozilla that all of these simple but really helpful ideas take stupid amounts of time to develop. Look at Feedback/Firefox Input; about:memory itself; Add-On Compat Reporter; better crash reporting (Talkback?); the nightly unit testing code, etc. The list is long. These good ideas slowly get done but they take forever. It kinda makes a mockery of the ‘rapid-release’ concept.

    I’d like to haul all that data together but it’s a PITA when the Add-On Manager has just been re-done without any facility to dump all installed extensions to a list. The extension that actually did that was broken by the flipping change to a tab-based Add-On manager!

    It’s all fairly crazy the Mozilla world and it’s great when the attitude from Mozilla is usually: don’t whinge, write an Add-On. Unfortunately not all of us have time to do that and for those who do, they get the rug swept out from under them when those employed full time by Mozilla are able to achieve so much more by working on Mozilla code full time.

    *sigh*

    FWIW left browser open all day, 13 tabs open plus 11 in Panorama, one tab is gmail, browser currently sitting on reasonable 350 MB :) Nice to finish on a good note.

  23. Hi Nicholas,

    I’ve used the 64Bit nightly build since my last post and I can’t see any real disadvantages except that the 64Bit Flash is missing. But i can live with that. If several 64bt browsers are released adobe will release a compatible plugin.

    The memshrink is fine, but the cause of the firefox slowness is a different part. The SQLite 3 storage. I used procdump to automatically create dumps if the firefox hangs and I always get this:

    Firefox 5 Release:
    *** Process exceeded 20% CPU for 5 seconds. Thread consuming CPU: 884 (0×374)’

    0:018> k
    ChildEBP RetAddr
    1052fb30 739fdd4c mozsqlite3!sqlite3VdbeExec+0xc9 [e:\builds\moz2_slave\rel-rel-w32-bld\build\db\sqlite3\src\sqlite3.c @ 61194]
    1052fc64 6b511f46 mozsqlite3!sqlite3_step+0xbc [e:\builds\moz2_slave\rel-rel-w32-bld\build\db\sqlite3\src\sqlite3.c @ 59103]
    1052fc88 6b568e2f xul!mozilla::storage::stepStmt+0×26 [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\mozstorageprivatehelpers.cpp @ 283]
    1052fc9c 6b658225 xul!mozilla::storage::Statement::ExecuteStep+0xaf [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\mozstoragestatement.cpp @ 617]
    1052fd30 6b645d12 xul!`anonymous namespace’::FetchPageInfo+0×205 [e:\builds\moz2_slave\rel-rel-w32-bld\build\toolkit\components\places\asyncfaviconhelpers.cpp @ 136]
    1052fd94 6b58489a xul!mozilla::places::AsyncAssociateIconToPage::Run+0×26 [e:\builds\moz2_slave\rel-rel-w32-bld\build\toolkit\components\places\asyncfaviconhelpers.cpp @ 707]
    1052fdc4 6b530c48 xul!nsThread::ProcessNextEvent+0x15a [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\nsthread.cpp @ 624]
    1052fde4 7391bdd9 xul!nsThread::ThreadFunc+0×88 [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\nsthread.cpp @ 273]
    00000000 00000000 nspr4!_PR_NativeRunThread+0×169 [e:\builds\moz2_slave\rel-rel-w32-bld\build\nsprpub\pr\src\threads\combined\pruthr.c @ 448]

    64Bit nightly:
    *** Process exceeded 20% CPU for 5 seconds. Thread consuming CPU: 3384 (0xd38)’

    0:020> k
    Child-SP RetAddr Call Site
    00000000`146bf640 000007fe`f0d5da4f mozsqlite3!sqlite3VdbeExec+0×384 [e:\builds\moz2_slave\m-cen-w64-ntly\build\db\sqlite3\src\sqlite3.c @ 61256]
    *** WARNING: Unable to verify checksum for xul.dll
    00000000`146bfad0 000007fe`ebeaf1c6 mozsqlite3!sqlite3_step+0xdf [e:\builds\moz2_slave\m-cen-w64-ntly\build\db\sqlite3\src\sqlite3.c @ 59103]
    00000000`146bfb20 000007fe`ebe0348e xul!mozilla::storage::Statement::ExecuteStep+0xd6 [e:\builds\moz2_slave\m-cen-w64-ntly\build\storage\src\mozstoragestatement.cpp @ 617]
    00000000`146bfb80 000007fe`ebe0ba4f xul!`anonymous namespace’::FetchPageInfo+0×272 [e:\builds\moz2_slave\m-cen-w64-ntly\build\toolkit\components\places\asyncfaviconhelpers.cpp @ 136]
    00000000`146bfcc0 000007fe`ebf1f080 xul!mozilla::places::AsyncAssociateIconToPage::Run+0×23 [e:\builds\moz2_slave\m-cen-w64-ntly\build\toolkit\components\places\asyncfaviconhelpers.cpp @ 741]
    00000000`146bfd60 000007fe`ebf8a4df xul!nsThread::ProcessNextEvent+0×260 [e:\builds\moz2_slave\m-cen-w64-ntly\build\xpcom\threads\nsthread.cpp @ 623]
    *** WARNING: Unable to verify checksum for nspr4.dll
    00000000`146bfe00 000007fe`fa433298 xul!nsThread::ThreadFunc+0x8f [e:\builds\moz2_slave\m-cen-w64-ntly\build\xpcom\threads\nsthread.cpp @ 272]
    00000000`146bfe30 000007fe`fa433a2d nspr4!_PR_NativeRunThread+0×158 [e:\builds\moz2_slave\m-cen-w64-ntly\build\nsprpub\pr\src\threads\combined\pruthr.c @ 448]
    00000000`146bfea0 00000000`6fcd2cc7 nspr4!pr_root+0xd [e:\builds\moz2_slave\m-cen-w64-ntly\build\nsprpub\pr\src\md\windows\w95thred.c @ 123]
    00000000`146bfed0 00000000`6fcd2d64 mozcrt19!_callthreadstartex+0×17 [e:\builds\moz2_slave\m-cen-w64-ntly\build\obj-firefox\memory\jemalloc\crtsrc\threadex.c @ 348]
    00000000`146bff00 00000000`76e1651d mozcrt19!_threadstartex+0×84 [e:\builds\moz2_slave\m-cen-w64-ntly\build\obj-firefox\memory\jemalloc\crtsrc\threadex.c @ 326]
    00000000`146bff30 00000000`76f4c2e1 kernel32!BaseThreadInitThunk+0xd
    00000000`146bff60 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

    After the memshrink improve the performance of the SQLite dbs, please!

    André

  24. I created a new dump when firefox hang and it shows the same sqlite issue:

    *** procdump -h -o firefox D:\Daten\Downloads\_dumps\
    *** Hung window detected: 860c82′

    STACK_TEXT:
    0026d2d0 7713b4d4 000006e4 00000000 00000000 ntdll!NtWaitForSingleObject+0×15
    0026d334 7713b3b8 00000000 00000000 056fd580 ntdll!RtlpWaitOnCriticalSection+0x13e
    0026d35c 73a46f3b 063132e8 73a46be3 063132e8 ntdll!RtlEnterCriticalSection+0×150
    0026d364 73a46be3 063132e8 056fd598 6b4ef64f mozsqlite3!winMutexEnter+0xb [e:\builds\moz2_slave\rel-rel-w32-bld\build\db\sqlite3\src\sqlite3.c @ 17445]
    0026d370 6b4ef64f 07e9d408 00000001 056fd598 mozsqlite3!sqlite3_extended_result_codes+0×13 [e:\builds\moz2_slave\rel-rel-w32-bld\build\db\sqlite3\src\sqlite3.c @ 107339]
    0026d390 6b51878d 07e9d408 056fd598 0a5a5b20 xul!mozilla::storage::prepareStmt+0×14 [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\mozstorageprivatehelpers.cpp @ 315]
    0026d3b8 6b66e0a3 056fd580 07e9d408 0575d3dc xul!mozilla::storage::Statement::initialize+0x5d [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\mozstoragestatement.cpp @ 178]
    0026d3d4 6b96a8b6 0a5a5b20 0026d42c 056fd580 xul!mozilla::storage::Connection::CreateStatement+0x4e [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\mozstorageconnection.cpp @ 940]
    0026d514 739490e9 00270040 09a11730 00000020 xul!mozilla::storage::`anonymous namespace’::Vacuumer::execute+0x27cac2
    0026d52c 739499ed 00270040 6b76be31 06392790 mozcrt19!arena_malloc_small+0×149 [e:\builds\moz2_slave\rel-rel-w32-bld\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 3789]
    0026d53c 6b780245 57bd391a 0026d578 6b729d7a mozcrt19!malloc+0x3d [e:\builds\moz2_slave\rel-rel-w32-bld\build\obj-firefox\memory\jemalloc\crtsrc\jemalloc.c @ 5876]
    0026d578 6b728b6d 05380ca0 00000000 00000000 xul!mozilla::storage::`anonymous namespace’::Vacuumer::Vacuumer+0x1f [e:\builds\moz2_slave\rel-rel-w32-bld\build\storage\src\vacuummanager.cpp @ 195]
    0026d5ac 6b6e7b8b 00000001 09a10c20 6c10e7a4 xul!nsIdleServiceDaily::Observe+0×68 [e:\builds\moz2_slave\rel-rel-w32-bld\build\widget\src\xpwidgets\nsidleservice.cpp @ 98]
    0026d67c 6bece3eb 00000000 0a58f4c0 6b609710 xul!nsIdleService::CheckAwayState+0x16c [e:\builds\moz2_slave\rel-rel-w32-bld\build\widget\src\xpwidgets\nsidleservice.cpp @ 426]
    0026d688 6b609710 0a58f4c0 09a10c20 00837550 xul!nsIdleService::IdleTimerCallback+0xc [e:\builds\moz2_slave\rel-rel-w32-bld\build\widget\src\xpwidgets\nsidleservice.cpp @ 311]
    0026d6b0 6b60995c 00837594 6b584a22 043b8f60 xul!nsTimerImpl::Fire+0xe0 [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\nstimerimpl.cpp @ 424]
    0026d6b8 6b584a22 043b8f60 00816770 00851240 xul!nsTimerEvent::Run+0x1c [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\nstimerimpl.cpp @ 522]
    0026d6dc 6b554a2e 008561a0 00816760 6b55db80 xul!nsThread::ProcessNextEvent+0x2e2 [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\nsthread.cpp @ 618]
    0026d71c 6b777c7b 00000000 00851240 6b777c64 xul!TimerThread::RemoveTimer+0x2e [e:\builds\moz2_slave\rel-rel-w32-bld\build\xpcom\threads\timerthread.cpp @ 417]
    0026d728 6b777c64 57bd3aaa 00837550 008876a0 xul!MessageLoop::RunInternal+0×11 [e:\builds\moz2_slave\rel-rel-w32-bld\build\ipc\chromium\src\base\message_loop.cc @ 219]
    0026d740 7395808b 0026d730 00001f00 0026faf0 xul!MessageLoop::RunHandler+0x1d [e:\builds\moz2_slave\rel-rel-w32-bld\build\ipc\chromium\src\base\message_loop.cc @ 203]
    0026d760 6b777c10 0026d7a0 00000001 6b524100 mozcrt19!_VEC_memzero+0×36
    0026d780 6b75c12e 050ce660 008876a0 6b77875e xul!MessageLoop::Run+0×15 [e:\builds\moz2_slave\rel-rel-w32-bld\build\ipc\chromium\src\base\message_loop.cc @ 177]
    0026d78c 6b77875e 008876a0 03a52880 00da0000 xul!nsBaseAppShell::Run+0×34 [e:\builds\moz2_slave\rel-rel-w32-bld\build\widget\src\xpwidgets\nsbaseappshell.cpp @ 191]
    0026f6e0 6b778792 008876a0 00000000 6b6a9975 xul!nsAppShell::Run+0×42 [e:\builds\moz2_slave\rel-rel-w32-bld\build\widget\src\windows\nsappshell.cpp @ 270]
    0026f6ec 6b6a9975 03a52880 008290a8 00000001 xul!nsAppStartup::Run+0x1e [e:\builds\moz2_slave\rel-rel-w32-bld\build\toolkit\components\startup\nsappstartup.cpp @ 219]
    0026fa74 00da134c 00000001 008290a8 00817480 xul!XRE_main+0xdcf [e:\builds\moz2_slave\rel-rel-w32-bld\build\toolkit\xre\nsapprunner.cpp @ 3758]
    0026fac0 00da16f2 00000001 0081f680 00812280 firefox!wmain+0x34c [e:\builds\moz2_slave\rel-rel-w32-bld\build\toolkit\xre\nswindowswmain.cpp @ 128]
    0026fb00 7673337a fffde000 0026fb4c 7713bf52 firefox!__tmainCRTStartup+0×152 [e:\builds\moz2_slave\rel-rel-w32-bld\build\obj-firefox\memory\jemalloc\crtsrc\crtexe.c @ 591]
    0026fb0c 7713bf52 fffde000 679a2b3d 00000000 kernel32!BaseThreadInitThunk+0xe
    0026fb4c 7713bf25 00da1870 fffde000 00000000 ntdll!__RtlUserThreadStart+0×70
    0026fb64 00000000 00da1870 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b

    Please improve this in future versions.

    • Nicholas Nethercote

      André, that last stack trace shows that the SQLite places database is being “vacuumed”, which is an operation that repacks the data to be more compact. My understanding is that it can be expensive — eg. takes 10s of seconds — but that it’s only supposed to run occasionally, and when the browser is idle (note the nsIdleService::CheckAwayState call in the stack). Did you see this pause while Firefox wasn’t idle? Maybe you got unlucky and it had just started when you came back to using Firefox?

  25. Hi Nicholas,

    I see this action during normal usage with Firefox when I browse the web. The Firefox is not idle at this time. I submitted it yesterday to this bug entry:

    https://bugzilla.mozilla.org/show_bug.cgi?id=671558

    I have this issue for a long time.

    André

  26. Hi Nicholas,

    I hope my second attempt to write this answer works.

    I get this during normal usage of Firefox when I browse the www.

    I submitted it yesterday:

    https://bugzilla.mozilla.org/show_bug.cgi?id=671558

    I have this issue for a longer time. I also tried to optimize the DB files with SpeedyFox, but it makes no difference.

    André