{"id":61,"date":"2011-01-11T02:58:15","date_gmt":"2011-01-11T01:58:15","guid":{"rendered":"http:\/\/blog.mozilla.org\/jseward\/?p=61"},"modified":"2011-03-05T00:40:35","modified_gmt":"2011-03-04T23:40:35","slug":"finding-data-races-in-the-js-engine","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/jseward\/2011\/01\/11\/finding-data-races-in-the-js-engine\/","title":{"rendered":"Finding data races in the JS engine"},"content":{"rendered":"<h3>Some background<\/h3>\n<p>Back in March last year I spent some time using Valgrind&#8217;s Helgrind tool<br \/>\nto look for data races in the browser.\u00a0 Data races happen when two<br \/>\nthreads access the same piece of memory without any form of<br \/>\nsynchronisation (either locking or the use of atomic operations), and at<br \/>\nleast one of the accesses is a write.\u00a0 That can lead to all manner of<br \/>\ndata structure corruption and crashes.\u00a0 What&#8217;s really bad is that such<br \/>\nbugs are often nearly impossible to reproduce, because they are timing<br \/>\ndependent.\u00a0 They therefore fall into the Very Scary Bugs category, and<br \/>\nare something we really want to get rid of by any means possible.<br \/>\nBefore release!<\/p>\n<p>Helgrind is a runtime analysis tool that looks for such races.\u00a0 It is far<br \/>\nfrom perfect, but better than nothing.\u00a0 For those familiar with these<br \/>\nthings, it&#8217;s a pure happens-before race detector capable of showing full<br \/>\nstacks for both memory accesses involved in a race.\u00a0 It also checks for<br \/>\nlock ordering inconsistencies (a.k.a. potential deadlocks) and various<br \/>\nmisuses of the POSIX pthreads API.<\/p>\n<p>So what happened with Helgrinding the browser?\u00a0 In short, I didn&#8217;t get<br \/>\nfar.\u00a0 Mostly I just got greyer hair.\u00a0 I had hoped to get the browser<br \/>\nHelgrind-clean, in the same way it is now pretty much Memcheck-clean,<br \/>\nbut the effort got mired in difficulties:<\/p>\n<ul>\n<li> Race-checking is resource-intensive, more so than the memory checking<br \/>\nthat Valgrind (Memcheck) is normally used for.\u00a0 A critical data<br \/>\nstructure in Helgrind (vector timestamps) turned out not to work well<br \/>\nat the scale demanded for full browser runs, and they became<br \/>\ninfeasibly slow and memory hungry.<\/li>\n<\/ul>\n<ul>\n<li> Happens-before race detection has the advantage of not giving false<br \/>\npositives.  But the downside is it is scheduling-sensitive, so<br \/>\nidentical runs sometimes report different subsets of the races that<br \/>\nare really present.  Add to that the nondeterminism of the browser and<br \/>\nthe slowness of Helgrind, and I had a major repeatability problem.<\/li>\n<\/ul>\n<ul>\n<li> The browser has a number of deliberate or apparently-harmless races.<br \/>\nMaking sense of the race reports required examining bits of source<br \/>\ncode all over the tree that I&#8217;d never seen before and didn&#8217;t<br \/>\nunderstand.\u00a0 This proved to be difficult and time consuming.<\/li>\n<\/ul>\n<p>So I left it at that, resolving one day to come back and fix the vector<br \/>\ntimestamp representations, so as to at least avoid the resource<br \/>\nproblems.<\/p>\n<p>Nothing happened for some months.\u00a0 Then, in December, Paul Biggar wrote<br \/>\na nice self-contained threaded Javascript test case, <a href=\"http:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=619595\">bug 619595<\/a>.<\/p>\n<p>And I thought: hmm, maybe I should Helgrindify just the threaded jsshell<br \/>\nrunning Paul&#8217;s test.\u00a0 The standalone JS engine is smaller and more<br \/>\ntractable than the browser, and I knew that Jason Orendorff had<br \/>\nsuccessfully used Helgrind on it earlier in the year.\u00a0 Also, there&#8217;s<br \/>\nbeen a vast amount of JS engine hackery in the past year, with<br \/>\nparticular emphasis on the threading aspects.\u00a0 And 4.0 is coming up<br \/>\nfast.\u00a0 So, I thought, now might be a good time to give it a spin.<\/p>\n<h3>Preparation<\/h3>\n<p>To work properly, Helgrind needs to see all inter-thread synchronisation<br \/>\nevents in the program under test.\u00a0 It can do that without help for<br \/>\nprograms which use only the POSIX pthreads API.\u00a0 But many programs roll<br \/>\ntheir own synchronisation primitives, and since it can&#8217;t see those,<br \/>\nHelgrind reports huge numbers of races which don&#8217;t exist.\u00a0 Both NSPR and<br \/>\nthe JS engine do this (eg ThinLocks).\u00a0 A small amount of markup using<br \/>\nclient requests (<a href=\"http:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=551155\">bug 551155<\/a>) provides Helgrind with the information it needs.<\/p>\n<p>Paul&#8217;s test runs, or, at least, tries to run, all the Sunspider tests in<br \/>\nparallel.\u00a0 I modified it trivially to make it run up to 10 copies of<br \/>\neach test in parallel.\u00a0 This stresses Helgrind to the limit of<br \/>\nfeasibility on my machine, but shakes out more races.<\/p>\n<p>Then we&#8217;re ready to go.<\/p>\n<h3>Results<\/h3>\n<p>I found a number of races &#8212; some expected, some not.\u00a0 The unintended<br \/>\nand dangerous-looking ones are:<\/p>\n<ul>\n<li> allocator for YARR-generated code is not thread safe (<a href=\"http:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=587288\">bug 587288<\/a>)<br \/>\n&#8212; potential crasher<\/li>\n<\/ul>\n<ul>\n<li> race on JSContext::defaultCompartmentIsLocked (<a href=\"http:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=622691\">bug 622691<\/a>)<br \/>\n&#8212; consequences unknown to me, but doesn&#8217;t look correct<\/li>\n<\/ul>\n<ul>\n<li>various races on parts of the property tree, eg kids[] array<br \/>\nelements (<a href=\"http:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=609104#c3\">bug 609104 comment 3<\/a>)<\/li>\n<\/ul>\n<p>Then there are two which are unintended but probably harmless.\u00a0 In both<br \/>\ncases each thread initialises a shared data structure to some value<br \/>\nwhich never changes after that, so multiple initialisations are harmless:<\/p>\n<ul>\n<li>jsdate.cpp: global &#8220;static jsdouble LocalTZA;&#8221; is raced<\/li>\n<\/ul>\n<ul>\n<li>nanojit::Assembler::nHints[] is raced.<\/li>\n<\/ul>\n<p>Then there are races which are intended and, so, presumably harmless, on<br \/>\nthe following fields:<\/p>\n<ul>\n<li>JSRuntime::gcMallocBytes (also gcBytes, I think)<\/li>\n<\/ul>\n<ul>\n<li> JSRuntime::gcPoke<\/li>\n<\/ul>\n<ul>\n<li> JSRuntime::protoHazardShape<\/li>\n<\/ul>\n<ul>\n<li> JSThreadData::requestDepth<\/li>\n<\/ul>\n<ul>\n<li> JSRuntime::gcIsNeeded<\/li>\n<\/ul>\n<p>Finally, there&#8217;s one I can&#8217;t decide about:<\/p>\n<ul>\n<li> The GC&#8217;s stack scanner races against other functions that touch the<br \/>\nstack &#8212; that is, just about everything.\u00a0 I don&#8217;t know if I expect<br \/>\nthat or not.\u00a0 I would have thought that if one thread is scanning the<br \/>\nstack, all the other threads are blocked waiting for it, hence there<br \/>\nis no race.\u00a0 So either (1) my understanding is wrong, (2) helgrind<br \/>\ndoesn&#8217;t see the inter-thread sync events causing other threads to<br \/>\nwait, or (3) the stack scanner is borked.\u00a0 I suspect (1) or (2).<\/li>\n<\/ul>\n<h3>Comments<\/h3>\n<p>It&#8217;s pleasing to have a list of at least some of the observable races in<br \/>\nthe JS engine, since it provides something to cross-check assumed racey<br \/>\nbehaviour against.\u00a0 It&#8217;s also good to have found some unintended races<br \/>\nbefore release.<\/p>\n<p>I&#8217;m a little concerned about the intended races, eg JSRuntime::gcPoke.<br \/>\nMy sketchy understanding of the C++0x draft standard is that accesses to<br \/>\nshared locations must be mediated either by locking or by machine-level<br \/>\natomic operations.\u00a0 All other shared accesses count as races.\u00a0 C++0x, in<br \/>\n<a href=\"http:\/\/www.hpl.hp.com\/personal\/Hans_Boehm\/c++mm\/threadsintro.html#c++0x\">the words of Hans J Boehm<\/a>, &#8216;guarantees nothing in the event of a data<br \/>\nrace.\u00a0 Any program allowing a data race produces &#8220;undefined behavior&#8221;.&#8217;<\/p>\n<p>Boehm has good presentation which summarises the proposed C++0x memory<br \/>\nmodel, at <a href=\"http:\/\/www.hpl.hp.com\/personal\/Hans_Boehm\/misc_slides\/c++mm.pdf\">http:\/\/www.hpl.hp.com\/personal\/Hans_Boehm\/misc_slides\/c++mm.pdf<\/a>.<br \/>\nSee in particular slides 7, 9 and 11.<\/p>\n<p>From a Helgrind-usage point of view, these races are easily suppressed<br \/>\nby adding client requests to specify that the fields in question should<br \/>\nnot be race-checked.\u00a0 But, overall, I still don&#8217;t like them: deliberate<br \/>\nraces are a hindrance to understandability and to automated checking.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Some background Back in March last year I spent some time using Valgrind&#8217;s Helgrind tool to look for data races in the browser.\u00a0 Data races happen when two threads access the same piece of memory without any form of synchronisation &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/jseward\/2011\/01\/11\/finding-data-races-in-the-js-engine\/\">Continue reading<\/a><\/p>\n","protected":false},"author":240,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/posts\/61"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/users\/240"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/jseward\/wp-json\/wp\/v2\/tags?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}