{"id":2924,"date":"2014-06-16T16:08:44","date_gmt":"2014-06-16T05:08:44","guid":{"rendered":"http:\/\/blog.mozilla.org\/nnethercote\/?p=2924"},"modified":"2014-06-16T16:08:44","modified_gmt":"2014-06-16T05:08:44","slug":"a-browser-benchmarking-manifesto","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nnethercote\/2014\/06\/16\/a-browser-benchmarking-manifesto\/","title":{"rendered":"A browser benchmarking manifesto"},"content":{"rendered":"

This post represents my own opinion, not the official position of Mozilla.<\/strong><\/em><\/p>\n

I’ve been working on Firefox for over five years and I’ve been unhappy with the state of browser benchmarking the entire time.<\/p>\n

The current problems<\/h3>\n

The existing benchmark suites are bad<\/h4>\n

Most of the benchmark<\/span> suites are terrible. Consider at Hennessy and Patterson’s categorization of benchmarks<\/span>, from best to worst.<\/p>\n

    \n
  1. Real applications.<\/li>\n
  2. Modified applications (e.g. with I\/O removed to make it CPU-bound).<\/li>\n
  3. Kernels (key fragments of real applications).<\/li>\n
  4. Toy benchmarks<\/span> (e.g. sieve of Erastosthenes).<\/li>\n
  5. Synthetic benchmarks<\/span> (code created artificially to fit a profile of particular operations, e.g. Dhrystone).<\/li>\n<\/ol>\n

    In my opinion, benchmark suites should only contain benchmarks in categories 1 and 2. There are certainly shades of grey in these categories, but I personally have a high bar, and I’m likely to consider most programs whose line count is in the hundreds as closer to a “toy benchmark” than a “real application”.<\/p>\n

    Very few browser benchmark<\/span> suites contain benchmarks from category 1 or 2. I think six or seven of the benchmarks in Octane<\/a> meet this standard. I’m having trouble thinking of any other existing benchmark suite that contains any<\/em> that reach this standard. (This includes Mozilla’s own Kraken suite.)<\/p>\n

    Bad benchmark suites hurt the web, because every hour that browser vendors spend optimizing bad benchmarks is an hour they don’t spend improving browsers in ways that actually help users. I’ve seen first-hand how much time this wastes.<\/p>\n

    Conflicts of interest<\/h4>\n

    Some of the benchmark<\/span> suites — including the best-known ones — come from browser vendors. The conflict of interest is so obvious I won’t bother discussing it further.<\/p>\n

    Opaque scoring systems<\/h4>\n

    Some of them have opaque scoring systems. Octane (derived from V8bench) is my least favourite here. It also suffers from much higher levels of variance between runs than the benchmarks with time-based scoring systems.<\/p>\n

    A better way<\/h3>\n

    Here’s how I think a new browser benchmark<\/span> suite should be created. \u00a0Some of these ideas are taken from the SPEC benchmark suites<\/a>which have been used for over 20 years to benchmark the performance of C, C++, Fortran and Java programs, among other things.<\/p>\n

    A collaborative and ongoing process<\/h4>\n

    The suite shouldn’t be owned by a single browser vendor. Ideally, all the major browser vendors (Apple, Google, Microsoft, and Mozilla) would be involved. Involvement of experts from outside those companies would also be desirable.<\/p>\n

    This is a major political and organizational challenge, but it would avoid bias (both in appearance and reality), and would likely lead to a higher-quality suite, because low-quality benchmarks are likely to be shot down.<\/p>\n

    There should be a public submissions process, where people submit web
    \napps\/sites for consideration. \u00a0And obviously there needs to be a selection
    \nprocess to go with that.<\/p>\n

    The benchmark<\/span> suite should be open source and access should be free of
    \ncharge. \u00a0It should be hosted on a public site not owned by a browser vendor.<\/p>\n

    There should also be an official process for modifying the suite: modifying benchmarks<\/span>, removing benchmark<\/span>, and adding new benchmarks<\/span>. \u00a0This process
    \nshouldn’t be too fluid. \u00a0Perhaps allowing changes once per year would be reasonable. The current trend of continually augmenting old suites should be resisted.<\/p>\n

    High-quality<\/h4>\n

    All benchmarks<\/span> should be challenging, and based on real website\/apps; \u00a0they should all belong to category 1 or 2 on the above scale.<\/p>\n

    Each benchmark<\/span> should have two or three inputs of different sizes. \u00a0Only the
    \nlargest input would be used in official runs, but the smaller inputs
    \nwould be useful for browser developers for testing and profiling purposes.<\/p>\n

    All benchmarks<\/span> should have side-effects of some kind that depend on most or
    \nall of the computation. \u00a0There should be no large chunks of computation whose
    \nresults are unused, because such benchmarks<\/span> are subject to silly results due
    \nto certain optimizations such as dead code elimination.<\/p>\n

    Categorization<\/h4>\n

    Any categorization of benchmarks<\/span> should not be done by browser subsystems. E.g. there shouldn’t be groups like “DOM”, “CSS”, “hardware acceleration”, “code loading” because that encourages artificial benchmarks<\/span> that stress a single
    \nsubsystem. Instead, any categories should be based on application domain:
    \n“games”, “productivity”, “multimedia”, “social”, etc. If a grouping like
    \nthat ends up stressing some browser subsystems more than others, well that’s
    \nfine, because it means that’s what real websites\/apps are doing (assuming the
    \nbenchmark<\/span> selection is balanced).<\/p>\n

    There is one possible exception to this rule, which is the JavaScript engine. At least some of the benchmarks<\/span> will be dominated by JS execution time, and because JS engines can be built standalone, the relevant JS engine
    \nteams will inevitably want to separate JS code from everything else so that
    \nthe code can be run in JS shells. It may be worthwhile to pre-emptively do that, where appropriate. If that were done, you wouldn’t run these JS-only sub-applications as part of an official suite run, but they would be present in the repository.<\/p>\n

    Scoring<\/h4>\n

    Scoring should be as straightforward as possible. Times are best because their meaning is more obvious than anything else, *especially* when multiple
    \ntimes are combined. Opaque points systems are awful.<\/p>\n

    Using times for everything makes most sense when benchmarks<\/span> are all
    \nthroughput-oriented. \u00a0If some benchmarks<\/span> are latency-oriented (or FPS, or
    \nsomething else) this becomes trickier. \u00a0Even still, a simple system that humans can intuitively understand is best.<\/p>\n

    Conclusion<\/h3>\n

    It’s time the browser world paid attention to the performance and benchmarking lessons that the CPU and C\/C++\/Fortran worlds learned years ago. It’s time to grow up.<\/p>\n

    If you want to learn more, I highly recommend getting your hands on a copy of Hennessy & Patterson’s Computer Architecture<\/em> and reading the section on measuring performance. This is section 1.5 (“Measuring and Reporting Performance”) in the 3rd edition, but I expect later editions have similar sections.<\/p>\n","protected":false},"excerpt":{"rendered":"

    This post represents my own opinion, not the official position of Mozilla. I’ve been working on Firefox for over five years and I’ve been unhappy with the state of browser benchmarking the entire time. The current problems The existing benchmark suites are bad Most of the benchmark suites are terrible. Consider at Hennessy and Patterson’s […]<\/p>\n","protected":false},"author":139,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15269],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/2924"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/users\/139"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/comments?post=2924"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/posts\/2924\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/media?parent=2924"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/categories?post=2924"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nnethercote\/wp-json\/wp\/v2\/tags?post=2924"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}