{"id":2219,"date":"2012-08-24T15:17:16","date_gmt":"2012-08-24T04:17:16","guid":{"rendered":"http:\/\/blog.mozilla.org\/nnethercote\/?p=2219"},"modified":"2012-08-26T14:12:00","modified_gmt":"2012-08-26T03:12:00","slug":"octane-minus-v8","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nnethercote\/2012\/08\/24\/octane-minus-v8\/","title":{"rendered":"Octane minus V8"},"content":{"rendered":"<p>The Google Chrome team recently <a href=\"http:\/\/blog.chromium.org\/2012\/08\/octane-javascript-benchmark-suite-for.html\">extended their V8 benchmark suite with five new benchmarks<\/a>, and renamed it &#8220;<a href=\"https:\/\/developers.google.com\/octane\/\">Octane<\/a>&#8220;.\u00a0 The <a href=\"https:\/\/developers.google.com\/octane\/benchmark\">descriptions page<\/a> says the following about the new benchmarks.<\/p>\n<ul>\n<li><strong>pdf.js.<\/strong> Mozilla&#8217;s <a href=\"http:\/\/mozilla.github.com\/pdf.js\/\">PDF Reader<\/a> implemented in JavaScript. It measures decoding and interpretation time (33,056 lines).<\/li>\n<li><strong>Mandreel.<\/strong> Runs the <a href=\"http:\/\/bulletphysics.org\/wordpress\/\">3D Bullet Physics Engine<\/a> ported from C++ to JavaScript via <a href=\"http:\/\/www.mandreel.com\/\">Mandreel<\/a> (277,377 lines).<\/li>\n<li><strong>GB Emulator.<\/strong> Emulate the portable console&#8217;s architecture and runs a demanding 3D simulation, all in JavaScript (11,097 lines).<\/li>\n<li><strong>Code loading.<\/strong> Measures how quickly a JavaScript engine can start executing code after loading a large JavaScript program, social widget being a common example. The source for test is derived from open source libraries (<a href=\"https:\/\/developers.google.com\/closure\/\">Closure<\/a>, <a href=\"http:\/\/jquery.com\/\">jQuery<\/a>) (1,530 lines).<\/li>\n<li><strong>Box2DWeb.<\/strong> Based on <a href=\"http:\/\/code.google.com\/p\/box2dweb\/\">Box2DWeb<\/a>, the popular 2D physics engine originally written by Erin Catto, ported to JavaScript. (560 lines, 9000+ de-minified).<\/li>\n<\/ul>\n<p>I haven&#8217;t looked closely at these benchmarks, but <strong><\/strong>the descriptions are very promising.\u00a0 Hennessy and Patterson&#8217;s classic <em>Computer Architecture<\/em> lists the following five categories of benchmarks, from best to worst.<\/p>\n<ol>\n<li>Real applications.<\/li>\n<li>Modified applications (e.g. with I\/O removed to make it CPU-bound).<\/li>\n<li>Kernels (key fragments of real applications).<\/li>\n<li>Toy benchmarks (e.g. sieve of Erastosthenes).<\/li>\n<li>Synthetic benchmarks (code created artificially to fit a profile of particular operations, e.g. Dhrystone).<\/li>\n<\/ol>\n<p>Four of the five new Octane benchmarks are category 1 or perhaps 2 (some have minor modifications to make them benchmarkable).\u00a0 &#8220;Code loading&#8221; is the only exception;\u00a0 it sounds like a kernel.\u00a0 Furthermore, most of these benchmarks are <strong><\/strong>large (look at those line counts!) and represent cutting-edge JavaScript code that real websites and browsers are using today (pdf.js!\u00a0 jQuery!\u00a0 Game engines!)<\/p>\n<p>I&#8217;m not saying these benchmarks are perfect &#8212; for example, there&#8217;s arguably too much focus on games, and <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=785079#c1\">the use of the proprietary Mandreel instead of the open source Emscripten is unfortunate<\/a> &#8212; but they certainly pass the initial &#8220;sniff test&#8221;.\u00a0 <strong>[Update:\u00a0 Alon Zakai has written a <a href=\"https:\/\/hacks.mozilla.org\/2012\/08\/no-single-benchmark-for-the-web\/\">more detailed critique<\/a> of the new benchmarks, particularly Mandreel and Box2DWeb.]<\/strong>\u00a0 Much more so, in fact, than the existing eight benchmarks that have been carried over from V8.\u00a0 I&#8217;ve listed their descriptions below;\u00a0 my annotations are in square brackets.<\/p>\n<ul>\n<li><strong>Richards.\u00a0 <\/strong>OS kernel simulation benchmark, originally written in BCPL by <a href=\"http:\/\/www.cl.cam.ac.uk\/%7Emr10\/\">Martin Richards<\/a> (539 lines).\u00a0 <em>[BCPL is a programming language that predates C.\u00a0 Martin Richards is a former colleague of mine, and I remember him saying in 2004 that its main use these days is running the control systems for some ancient car factories in South America!]<\/em><\/li>\n<li><strong>Deltablue<\/strong>.\u00a0 One-way <a href=\"http:\/\/www.cs.washington.edu\/research\/constraints\/deltablue\/\">constraint solver<\/a>, originally written in Smalltalk by John Maloney and Mario Wolczko (880 lines).\u00a0<em> [Ported SmallTalk code?!]<\/em><\/li>\n<li><strong>Raytrace. <\/strong>Ray tracer benchmark based on code by <a href=\"http:\/\/flog.co.nz\/\">Adam Burmister<\/a> (904 lines).<\/li>\n<li><strong>Regexp.\u00a0<\/strong>Regular expression benchmark generated by extracting regular expression operations from 50 of the most popular web pages (1761 lines). <em>[A kernel, but at least it comes from real websites.\u00a0 However, the results of the regexp invocations are not used which makes it easy to game.]<\/em><\/li>\n<li><strong>NavierStokes. <\/strong>2D NavierStokes equations solver, heavily manipulates double precision arrays. Based on Oliver Hunt&#8217;s code (387 lines).\u00a0<em> [This was added to V8 only <a href=\"http:\/\/blog.chromium.org\/2012\/03\/v8-benchmark-suite-extended-with.html\">a few months ago<\/a>.]<\/em><\/li>\n<li><strong>Crypto<\/strong>. Encryption and decryption benchmark based on code by Tom Wu (1698 lines).<\/li>\n<li><strong>Splay.<\/strong> Data manipulation benchmark that deals with splay trees and exercises the automatic memory management subsystem (394 lines).\u00a0 <em>[The data inserted into this splay tree is completely synthetic, which greatly limits it usefulness as a benchmark.]<\/em><\/li>\n<li><strong>EarleyBoyer.\u00a0 <\/strong>Classic Scheme benchmarks, translated to JavaScript by Florian Loitsch&#8217;s Scheme2Js compiler (4684 lines).\u00a0 <em>[&#8220;Classic&#8221; here means &#8220;old&#8221;.\u00a0 Also, auto-compiled Scheme code?!]<\/em><\/li>\n<\/ul>\n<p>These are all much smaller.\u00a0 Also, <strong>Regexp<\/strong> is the only one that is clearly based on code commonly run in web browsers.<\/p>\n<p>In fact, these new benchmarks are so much better than the old benchmarks that I wish the Google Chrome team had instead released them as a separate benchmark suite.\u00a0 That would have allowed the old benchmarks to gradually move into a well-earned retirement (along with the equally venerable and flawed <a href=\"http:\/\/www.webkit.org\/perf\/sunspider\/sunspider.html\">SunSpider<\/a>).\u00a0 I guess there&#8217;s nothing stopping people from effectively doing this by running just the new benchmarks.\u00a0 Perhaps this could be called the &#8220;Octane minus V8&#8221; benchmark suite&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Google Chrome team recently extended their V8 benchmark suite with five new benchmarks, and renamed it &#8220;Octane&#8220;.\u00a0 The descriptions page says the following about the new benchmarks. pdf.js. Mozilla&#8217;s PDF Reader implemented in JavaScript. It measures decoding and interpretation time (33,056 lines). Mandreel. Runs the 3D Bullet Physics Engine ported from C++ to JavaScript [&hellip;]<\/p>\n","protected":false},"author":139,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15269],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/2219"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/users\/139"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/comments?post=2219"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/2219\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/media?parent=2219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/categories?post=2219"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/tags?post=2219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}