{"id":1531,"date":"2011-12-21T15:38:58","date_gmt":"2011-12-21T04:38:58","guid":{"rendered":"http:\/\/blog.mozilla.org\/nnethercote\/?p=1531"},"modified":"2011-12-21T15:38:58","modified_gmt":"2011-12-21T04:38:58","slug":"memshrink-progress-week-27","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nnethercote\/2011\/12\/21\/memshrink-progress-week-27\/","title":{"rendered":"MemShrink progress, week 27"},"content":{"rendered":"<h3>Adaptive Memory Behaviour<\/h3>\n<p>The biggest news this week was that Justin Lebar landed code to <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=670967\">make Firefox&#8217;s memory consumption adaptive on Windows<\/a>.\u00a0 The code works by wrapping VirtualAlloc() and some similar OS-level allocation functions.\u00a0 Each time memory is allocated, the amount of available virtual and physical memory is measured by the wrapper.\u00a0 If either of those amounts are below a threshold, a memory pressure event is triggered, which causes numerous components within Firefox to take steps to reduce their memory consumption.<\/p>\n<p>It might not sound like it, but this is a big deal, and it was a MemShrink:P1 bug.\u00a0 Firefox runs on a huge variety of machines and devices, and trying to find one-size-fits-all heuristics for discarding regenerable data is a recipe for unhappiness somewhere.\u00a0 Adaptive behaviour is much better.<\/p>\n<p>The initial settings for this feature are conservative, and some tuning may be needed.<\/p>\n<ul>\n<li>The threshold used for virtual memory is 128MB, i.e. if the amount of available virtual memory drops below 128MB a memory pressure event is triggered.\u00a0 (This number can be changed in about:config via the &#8220;memory.low_virtual_memory_threshold_mb&#8221; setting.) Hopefully this will reduce the number of OOM crashes that occur on Windows.<\/li>\n<li>The threshold used for physical memory is 0, which means that it is currently ignored.\u00a0 (This number can be changed in about:config via the &#8220;memory.low_physical_mem_threshold_mb&#8221; setting.) The reason for ignoring it currently is that low physical memory may be caused by a program other than Firefox &#8212; we don&#8217;t want to start discarding data if Firefox is only using a small fraction of physical memory.\u00a0 In the future we can probably do better here by triggering memory pressure events if the fraction of physical memory used by Firefox exceeds some threshold;\u00a0 hopefully this will avoid paging when memory usage is high.<\/li>\n<li>Memory pressure events are throttled so they aren&#8217;t sent more than once every 10 seconds.\u00a0 (This number can be changed in about:config via the &#8220;memory.low_physical_memory_notification_interval_ms&#8221; setting.)\u00a0 This prevents thrashing in cases where the memory pressure events don&#8217;t help reduce memory consumption.<\/li>\n<\/ul>\n<p>Memory pressure events are also currently unevenly observed &#8212; plenty of data could be dropped that currently isn&#8217;t &#8212; so there is <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=710501\">room for improvement<\/a> there.<\/p>\n<p>Finally, there&#8217;s <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=664291\">another bug<\/a> still open to add similar adaptive behaviour on Mac, Linux and Android.\u00a0 Note also that memory pressure events are also triggered on Android when Firefox goes into the background.<\/p>\n<h3>Add-ons<\/h3>\n<p>There has been good news for add-ons:\u00a0 the #1, #6 and #8 most popular add-ons on AMO have all recently seen some MemShrink-related improvements.<\/p>\n<ul>\n<li>Wladimir Palant fixed <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=672111\">an intermittent zombie compartment in AdBlock Plu<\/a>s.\u00a0 This fix is available in AdBlock Plus 2.0.1.<\/li>\n<li>Jan Honza Odvarko fixed <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=699801\">a bug in Firebug that was causing it to use and hold onto excessive amounts of memory when websites chunked files through XMLHTTPRequest<\/a>.\u00a0 This fix is available in Firebug 1.9b3.<\/li>\n<li>Nils Maier added some memory reporters to DownThemAll!\u00a0 These show up in the &#8220;Other Measurements&#8221; list in about:memory, and may help the DownThemAll! developers diagnose leaks and excessive memory consumption caused by the add-on.\u00a0 These are available in the <a href=\"https:\/\/code.downthemall.net\/nightly\/\">DownThemAll! nightly builds<\/a>.<\/li>\n<\/ul>\n<p>Adding memory reporters to add-ons is a really good thing to do, and I plan to write some documentation to encourage other add-on developers to follow suit.\u00a0 However, if you are an add-on developer and are thinking of doing this, I need to stress one thing:\u00a0 the reports must have KIND_OTHER so that they end up in the &#8220;Other Measurements&#8221; list in about:memory.\u00a0 If they are put in the &#8220;explicit&#8221; tree, then some memory bytes may be counted twice (once by Firefox, once by the add-on) which will completely screw up about:memory&#8217;s results.<\/p>\n<h3>Memory Reporters<\/h3>\n<p>I landed several improvements to memory reporters.<\/p>\n<ul>\n<li>I f<a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=709653\">ixed two bugs in the JS memory reporter<\/a> that caused the size of the JS heap to be over-estimated by about 5%.\u00a0 Prior to this fix, &#8220;explicit&#8221;, &#8220;gc-heap-chunk-dirty-unused&#8221;, and &#8220;heap-unclassified&#8221; were all inflated slightly.\u00a0 I also added an assertion to prevent similar bugs happening again.<\/li>\n<li>I added <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=697335\">another memory reporter for the startup cache<\/a>, called &#8220;explicit\/startup-cache\/data&#8221;.\u00a0 This is often several MB shortly after start-up, but it tends to drop to almost zero after a while.<\/li>\n<li>I updated the <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=709470\">xpti<\/a> and <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=705602\">prefixset<\/a> memory reporters to the <a href=\"https:\/\/wiki.mozilla.org\/Platform\/Memory_Reporting\">new style<\/a>.\u00a0 This means they now measure slop bytes and are integrated with DMD.\u00a0 I also <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=707865\">added SizeOfExcludingThis() functions to nsTArray and nsTHashtable<\/a>, which gives the same benefits to all memory reporters that measure these basic types.<\/li>\n<\/ul>\n<h3>Bug Counts<\/h3>\n<p>Here are the current bug counts.<\/p>\n<ul>\n<li>P1: 26 (-2\/+1)<\/li>\n<li>P2: 139 (-6\/+6)<\/li>\n<li>P3: 64 (-2\/+2)<\/li>\n<li>Unprioritized: 0 (-1\/+0)<\/li>\n<\/ul>\n<p>Nice to see the total count drop, even if only by one.<\/p>\n<p>Due to the Christmas break there won&#8217;t be a MemShrink meeting nor a MemShrink progress report next week.\u00a0 I&#8217;ll be back in two weeks, enjoy the break!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Adaptive Memory Behaviour The biggest news this week was that Justin Lebar landed code to make Firefox&#8217;s memory consumption adaptive on Windows.\u00a0 The code works by wrapping VirtualAlloc() and some similar OS-level allocation functions.\u00a0 Each time memory is allocated, the amount of available virtual and physical memory is measured by the wrapper.\u00a0 If either of [&hellip;]<\/p>\n","protected":false},"author":139,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4550,4559,30,4544,4546],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/1531"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/users\/139"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/comments?post=1531"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/1531\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/media?parent=1531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/categories?post=1531"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/tags?post=1531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}