{"id":866,"date":"2011-06-09T13:24:05","date_gmt":"2011-06-09T02:24:05","guid":{"rendered":"http:\/\/blog.mozilla.org\/nnethercote\/?p=866"},"modified":"2011-06-09T15:37:32","modified_gmt":"2011-06-09T04:37:32","slug":"asynchronous-database-connections-must-be-closed-with-asyncclose","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nnethercote\/2011\/06\/09\/asynchronous-database-connections-must-be-closed-with-asyncclose\/","title":{"rendered":"Asynchronous database connections must be closed with asyncClose()"},"content":{"rendered":"<p>TL;DR:\u00a0 if you&#8217;re familiar with any Mozilla (C++ or JS) code that opens an async database  connection, please go and check that it calls asyncClose() to close the connection;\u00a0 not doing so can lead to crashes and\/or memory leaks.<\/p>\n<p>Specifically, in Firefox 6, when such a connection is destroyed (because it&#8217;s explicitly deallocated or its refcount drops to zero in C++, or it&#8217;s garbage collected in JS) it&#8217;ll currently cause sporadic crashes in about:memory or telemetry code or Test Pilot code.\u00a0 This is because memory reporters end up pointing to connections that have been freed, and so when they are used they end up (innocently) accessing memory they shouldn&#8217;t.<\/p>\n<p>I&#8217;ve opened <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=662989\">bug 662989<\/a> to avoid the crashes, but even once it&#8217;s implemented, I think that if you fail to call asyncClose() it will still cause a memory leak, because sqlite3_close() is never called on the connection and so SQLite won&#8217;t free up memory used by the connection.\u00a0 Also, connections that needlessly remain open can have active locks that can starve out other connections or  cause extreme growth of the write-ahead-log.<\/p>\n<p>As I write this, thanks to Marco Bonardo, I&#8217;m aware of three places in JS code that fail to call asyncClose() when they should:<\/p>\n<ul>\n<li>browser\/components\/preferences\/aboutPermissions.js.\u00a0 <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=654573#c48\">Bug 654573 <\/a>covers this, I&#8217;m about to land a patch to fix it.<\/li>\n<li>toolkit\/components\/contentprefs\/nsContentPrefService.js.\u00a0 <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=662986\">Bug 662986<\/a> covers this.<\/li>\n<li>browser\/app\/profile\/extensions\/testpilot@labs.mozilla.com\/modules\/experiment_data_store.js.\u00a0 <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=662985\">Bug 662985<\/a> covers this.\u00a0 Note that Test Pilot seems to fail to use asyncClose() when it should, and it also uses memory reporters.\u00a0 So it&#8217;s unclear which of these two things is responsible for the Test Pilot crashes of this sort seen so far;\u00a0 in other words, Test Pilot may be a culprit or an innocent bystander, or both.<\/li>\n<\/ul>\n<p id=\"comment_text_60\">These were found via a crude <a href=\"http:\/\/mxr.mozilla.org\/\">MXR<\/a> search.\u00a0 I haven&#8217;t looked for C++ code that makes this mistake.<\/p>\n<p>If you only do synchronous transactions over your database connection, the connection will be cleaned up properly, even if you don&#8217;t call close(), when it is deallocated or garbage collected.\u00a0 However, it&#8217;s better to close the connection as soon as you can so that the resources (memory, locks) are freed immediately.<\/p>\n<p>See <a href=\"https:\/\/developer.mozilla.org\/en\/storage#Closing_a_connection\">this MDC page<\/a> for more details about database connections and the relevant APIs.\u00a0 It appears that these APIs are somewhat error-prone.\u00a0 As it happens, an NS_ENSURE_TRUE macro will fail when an async connection is destroyed without having been asyncClose&#8217;d first, and this causes a warning message to be printed to stderr.\u00a0 Unfortunately, stderr is spammed with all sorts of warnings (something that was discussed in the comments of a <a href=\"http:\/\/blog.mozilla.org\/nnethercote\/2011\/06\/07\/what-is-the-point-of-non-fatal-assertions\/\">previous post<\/a> of mine), and so this warning gets lost among the noise.\u00a0 Addressing this problem is something I&#8217;ll write about shortly.<\/p>\n<p>Thanks to Andrew Sutherland and Marco Bonardo for helping me greatly with the bugs mentioned above.\u00a0 Any errors in this post are mine!<\/p>\n<p><strong>UPDATE:<\/strong> I did some <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=662989#c1\">investigation<\/a> and found that about:permissions leaked about 180KB of memory on my machine every time it failed to asyncClose a DB connection.\u00a0\u00a0 This showed up under the explicit\/storage\/sqlite\/other reporter in about:memory.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR:\u00a0 if you&#8217;re familiar with any Mozilla (C++ or JS) code that opens an async database connection, please go and check that it calls asyncClose() to close the connection;\u00a0 not doing so can lead to crashes and\/or memory leaks. Specifically, in Firefox 6, when such a connection is destroyed (because it&#8217;s explicitly deallocated or its [&hellip;]<\/p>\n","protected":false},"author":139,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4550,528,30,4551],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/866"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/users\/139"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/comments?post=866"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/866\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/media?parent=866"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/categories?post=866"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/tags?post=866"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}