{"id":114,"date":"2012-11-01T02:10:20","date_gmt":"2012-11-01T02:10:20","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=114"},"modified":"2012-11-01T02:10:20","modified_gmt":"2012-11-01T02:10:20","slug":"running-js-tests-faster","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2012\/11\/01\/running-js-tests-faster\/","title":{"rendered":"running js tests faster"},"content":{"rendered":"

I burned the tree<\/a> today.\u00a0 To verify my proposed fix, I had to run make check<\/tt>, which I don’t normally do.\u00a0 Excuse me, make -srj20 check<\/tt>, because that’s what my development machine can handle.\u00a0 Which is all fine until I got to check-jit-tests<\/tt> in js\/src<\/tt><\/a>.\u00a0 Which requires running nearly 30 thousand<\/em> tests.\u00a0 (N tests x 10 different JS engine option combinations starts to add up rather quickly…)<\/p>\n

In serial<\/em>.\u00a0 A load average of 1 has never looked so pitiful.<\/p>\n

Surely there must be a better way.\u00a0 While my machine churned through these tests, one at a time, I had a chance to take a look around, and found bug 686240<\/a>.\u00a0 Said bug is mostly about reducing overhead by cutting down the number of processes spawned, but it does give a nod to running the tests in parallel.\u00a0 But I wanted to improve the way the test harness runs first; fixing the innards of the test harness on top of that would just be gravy.<\/p>\n

I considered modifying jit_test.py<\/tt> to handle spawning multiple processes and managing them.\u00a0 But that’s just silly; we already have a tool for doing that: it’s called make<\/tt>.\u00a0 So let’s use it:<\/p>\n

diff --git a\/js\/src\/Makefile.in b\/js\/src\/Makefile.in\r\nindex 31dc5fb..7264cc5 100644\r\n--- a\/js\/src\/Makefile.in\r\n+++ b\/js\/src\/Makefile.in\r\n@@ -602,7 +602,7 @@ endif\r\n endif\r\n\r\n check-jit-test::\r\n-\t$(wildcard $(RUN_TEST_PROGRAM)) $(PYTHON) -u $(srcdir)\/jit-test\/jit_test.py \\\r\n+\t+$(wildcard $(RUN_TEST_PROGRAM)) $(PYTHON) -u $(srcdir)\/jit-test\/jit_test.py \\\r\n \t        --no-slow --no-progress --tinderbox --tbpl $(JITTEST_VALGRIND_FLAG) \\\r\n \t        $(DIST)\/bin\/js$(BIN_SUFFIX)\r\n\r\ndiff --git a\/js\/src\/jit-test\/jit_test.py b\/js\/src\/jit-test\/jit_test.py\r\nindex 7690548..e7d323b 100755\r\n--- a\/js\/src\/jit-test\/jit_test.py\r\n+++ b\/js\/src\/jit-test\/jit_test.py\r\n@@ -502,6 +502,33 @@ def main(argv):\r\n     if not OPTIONS.run_slow:\r\n         test_list = [ _ for _ in test_list if not _.slow ]\r\n\r\n+    if 'JS_PARALLELIZE' in os.environ and len(test_list) > 1:\r\n+        makefile_path = tmppath('jsmakefile')\r\n+        with open(makefile_path, 'w') as f:\r\n+            makefile_targets = []\r\n+            for (i, test) in enumerate(test_list):\r\n+                target = \"test%d\" % i\r\n+                subcmd = [sys.executable, sys.argv[0]]\r\n+                if OPTIONS.tinderbox:\r\n+                    subcmd.append('--tinderbox')\r\n+                if OPTIONS.tbpl:\r\n+                    subcmd.append('--tbpl')\r\n+                if OPTIONS.ion:\r\n+                    subcmd.append('--ion')\r\n+                if OPTIONS.hide_progress:\r\n+                    subcmd.append('--no-progress')\r\n+                subcmd.append(JS)\r\n+                subcmd.append(os.path.basename(test.path))\r\n+                makefile_targets.append((target, subcmd))\r\n+            f.write(\".PHONY: %s\\n\" % ' '.join([x for (x, y) in makefile_targets]))\r\n+            f.write(\"check: %s\\n\" % ' '.join([x for (x, y) in makefile_targets]))\r\n+            for (target, cmdline) in makefile_targets:\r\n+                f.write(\"%s:\\n\\t@%s\\n\" % (target, subprocess.list2cmdline(cmdline)))\r\n+        make_cmd = ['make', '-f', makefile_path, 'check']\r\n+        env = os.environ.copy()\r\n+        del env['JS_PARALLELIZE']\r\n+        os.execvpe(make_cmd[0], make_cmd, env)\r\n+\r\n     # The full test list is ready. Now create copies for each JIT configuration.\r\n     job_list = []\r\n     if OPTIONS.tbpl:<\/pre>\n
What the above code is doing is writing out a makefile that handles executing all the tests found by the test harness in parallel. There’s one target for each test, and then the check<\/tt> target depends on all of them being run. The script then runs a copy of make that executes all the commands found in this new makefile. The modification to Makefile.in<\/tt> is so make<\/tt> knows that you’re going to be executing recursive make<\/tt>s with that command. Credit where credit is due: I cribbed this idea from GCC<\/a>, which uses it to run bits of LTO compilation in parallel.<\/p>\n
Anyway, how fast does it go? JS_PARALLELIZE=1 time -p make -srj20 check-jit-test<\/tt> says:<\/p>\n
real 539.33\r\nuser 5492.89\r\nsys 653.84<\/pre>\n
Those times are in seconds. That’s for a --disable-optimize --enable-debug<\/tt> build, by the way.\u00a0 An --enable-optimize --enable-debug<\/tt> build looks like:<\/p>\n
real 144.20\r\nuser 1389.44\r\nsys 308.25<\/pre>\n
So about a 10x speedup either way. Not too shabby for 30 extra lines of code!<\/p>\n
The above patch is obviously only a proof-of-concept:<\/p>\n
    \n
  • We leave our temporary makefile lying around;<\/li>\n
  • Propagation of options to the sub-make needs to be expanded and cleaned up;<\/li>\n
  • Output from the sub-make may need to be handled sanely;<\/li>\n
  • I have no idea whether this works properly with pymake<\/tt>;<\/li>\n
  • …and probably a few other things I haven’t thought about.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
    I burned the tree today.\u00a0 To verify my proposed fix, I had to run make check, which I don’t normally do.\u00a0 Excuse me, make -srj20 check, because that’s what my development machine can handle.\u00a0 Which is all fine until I got to check-jit-tests in js\/src.\u00a0 Which requires running nearly 30 thousand tests.\u00a0 (N tests x […]<\/p>\n","protected":false},"author":320,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/114"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/users\/320"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/114\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/media?parent=114"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/categories?post=114"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/tags?post=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}