{"id":103,"date":"2012-10-05T21:21:07","date_gmt":"2012-10-05T21:21:07","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=103"},"modified":"2012-10-05T21:21:07","modified_gmt":"2012-10-05T21:21:07","slug":"looking-at-talos-differently-part-2","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2012\/10\/05\/looking-at-talos-differently-part-2\/","title":{"rendered":"looking at talos differently, part 2"},"content":{"rendered":"<p>Over the past couple of weeks, I&#8217;ve been working on refining <a title=\"looking at talos differently\" href=\"http:\/\/blog.mozilla.org\/nfroyd\/2012\/09\/19\/looking-at-talos-differently\/\">the Talos summary script<\/a> I blogged about earlier.\u00a0 The fruits of the labor can be seen in summary pages for the last three release cycles:<\/p>\n<ul>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff18-inbound\/\">FF18 inbound Talos summary<\/a><\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff17-inbound\/\">FF17 inbound Talos summary<\/a><\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff16-inbound\/\">FF16 inbound Talos summary<\/a><\/li>\n<\/ul>\n<p>(It&#8217;s worth pointing out that the coloring for the JS-related tests is wrong; I think those tests are &#8220;bigger is better&#8221; tests.)<\/p>\n<p>One meta-point before diving into suggestions for various tests: emails don&#8217;t always get triggered for the correct changesets.\u00a0 To see what I mean, take a look at the following examples:<\/p>\n<ul>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff17-inbound\/#ts-med-dirty-profile\">FF17 Ts, MED Dirty Profile<\/a>: We were quite fortunate during the FF17 cycle; our Talos emails identified <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=a9502620734e&amp;tochange=3b312c31d1cd\">a single changeset<\/a> as causing significant regressions in several areas, notably startup.\u00a0 But if you look at the above linked chart, you&#8217;ll see that on x86-64 Linux, the regression is linked to changesets occuring <strong>after<\/strong> the regressing changeset.\u00a0 I&#8217;m not sure how this happens, but it&#8217;s clearly a problem for identifying regressions.<\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff16-inbound\/#svg-opacity-row-major\">FF16 SVG, Row Opacity Major<\/a>: <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=de483b66eb94&amp;tochange=0a009343d59d\">The (first?) DLBI backout<\/a> shows a significant improvement on a good number of platforms, but Win XP-PGO&#8217;s improvement is attributed <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=5c30c255b490&amp;tochange=2e9813915a1b\">to changesets after the backout<\/a> that have nothing to do with SVG.<\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff18-inbound\/#trace-malloc-allocs\">FF18 Trace Malloc Allocs<\/a>: <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=095c446daab7&amp;tochange=a5eaeae97650\">This changeset<\/a> shows a signficant improvement in the number of allocations on Linux.\u00a0 Problem is, that changeset touched non-Linux related code.<\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff16-inbound\/#number-of-constructors\">FF16 Number of Constructors<\/a>: Mike Hommey did heroic work to significantly reduce the number of static constructors in the tree.\u00a0 Problem is that <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=0aa408fd0505&amp;tochange=b344d7df7b1a\">Bobby Holley&#8217;s bugfixing push<\/a> got all the credit.<\/li>\n<li><a href=\"http:\/\/method-combination.net\/talos\/ff18-inbound\/#dhtml-row-major-mozafterpaint\">FF18 DHTML Row Major MozAfterPaint<\/a>: XP shows an improvement that&#8217;s almost certainly related to DLBI landing, except that the improvement is attributed to the changesets <strong>before<\/strong> the landing, which is bizarre.<\/li>\n<\/ul>\n<p>All this suggests that there&#8217;s a bug in how we&#8217;re benchmarking our trees and generating our results; I haven&#8217;t investigated any further.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Over the past couple of weeks, I&#8217;ve been working on refining the Talos summary script I blogged about earlier.\u00a0 The fruits of the labor can be seen in summary pages for the last three release cycles: FF18 inbound Talos summary FF17 inbound Talos summary FF16 inbound Talos summary (It&#8217;s worth pointing out that the coloring [&hellip;]<\/p>\n","protected":false},"author":320,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/103"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/users\/320"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/comments?post=103"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/103\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/media?parent=103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/categories?post=103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/tags?post=103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}