{"id":105,"date":"2012-10-06T13:21:17","date_gmt":"2012-10-06T13:21:17","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=105"},"modified":"2012-10-06T02:59:18","modified_gmt":"2012-10-06T02:59:18","slug":"looking-at-talos-differently-part-3","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2012\/10\/06\/looking-at-talos-differently-part-3\/","title":{"rendered":"looking at talos differently, part 3"},"content":{"rendered":"<p>A few thoughts from several days of <a title=\"looking at talos differently, part 2\" href=\"http:\/\/blog.mozilla.org\/nfroyd\/2012\/10\/05\/looking-at-talos-differently-part-2\/\">staring at these charts<\/a>; I&#8217;m going to focus on the tests that generate the most email.\u00a0 Because Talos generates so much email, developers are prone to ignore it.\u00a0 And that&#8217;s not what we want them to do: getting a Talos email should be cause for consternation or celebration, not callous indifference.\u00a0 By focusing on the tests that generate the most email at first, we&#8217;ll weed out the bulk of the redundancy in Talos emails.<\/p>\n<p>Having said that:<\/p>\n<ul>\n<li>The Dromaeo tests are very noisy. I understand they&#8217;re important, and we ought to have some way of testing JS\/DOM\/CSS performance. But consider the <a href=\"http:\/\/method-combination.net\/talos\/ff17-inbound\/#dromaeo-dom\">FF17 Dromaeo DOM tests<\/a>: we can see <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=9215610e05a1&amp;tochange=5439489dc320\">three<\/a> <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=6be6808c2a4a&amp;tochange=6a732d5ea1e8\">different<\/a> <a href=\"http:\/\/hg.mozilla.org\/integration\/mozilla-inbound\/pushloghtml?fromchange=6a732d5ea1e8&amp;tochange=a468dc4bfaf4\">changesets<\/a> causing 6-7% swings in these tests. And those changesets don&#8217;t touch code anywhere near the JS\/DOM bits being tested! If we&#8217;re going to keep these tests, we need to find some way of making them more stable.<\/li>\n<li>Trace Malloc Allocs\/MaxHeap\/Leaks all send too much email for trivial changes: they&#8217;re often telling you about changes of tenths or even hundredths of a percent (to four significant digits!). I understand that many small changes eventually accumulate into big ones, but this seems a little excessive. There should be some sort of threshold, say a ~2% change either way, before warning emails get sent.<\/li>\n<li>The Number of Constructors&#8230;numbers have been identical across x86 and x86-64 Linux the last three release cycles and <a href=\"http:\/\/method-combination.net\/talos\/ff18-inbound\/#number-of-constructors\">the numbers keep going up<\/a>. In the abstract, sure, x86 and x86-64 can have different behavior, but in practice, we just don&#8217;t add static constructors dependent on the word size of the platform. We should cut this back to one platform at the very least, and consider setting a threshold here as well.<\/li>\n<li>The Tp5 No Network Row Major MozAfterPaint tests all generate a lot of email; it&#8217;s a bit of a tossup as to which one is going to stand out. Some of the numbers may be skewed due to DLBI&#8217;s cycles of landing and backouts, too. I will say that the more detailed tests, measuring Private Bytes, Main RSS, Content RSS, and %CPU, don&#8217;t identify regression candidates that our other tests don&#8217;t catch and are therefore not that useful.<\/li>\n<li>a11y Row Major MozAfterPaint has the same problem: it&#8217;s meant to identify issues in the a11y implementation, but more often than not, winds up complaining about regressions that other tests have already caught for us.<\/li>\n<\/ul>\n<p>All that is to say we could cut down the amount of email significantly with a couple of simple changes:<\/p>\n<ul>\n<li>Set thresholds before email alerts are sent for the Trace Malloc tests;<\/li>\n<li>Pick x86-64 or x86 Linux for Number of Constructors, possibly set a threshold here too;<\/li>\n<li>Remove the specific measurements in the Tp5 test.<\/li>\n<\/ul>\n<p>Other ideas:<\/p>\n<ul>\n<li>The above analysis was only for Mozilla-Inbound; there are of course statistics from other trees that are sent to dev-tree-management. Maybe it&#8217;s worth splitting dev-tree-management up? Must compute statistics on what trees generate the most mail to the list.<\/li>\n<li>The graphserver links sent in the emails are helpful; it would be even better if they featured multiple platforms. That way developers would have an easy(ier) way of assessing the usefulness of pursuing a given regression.<\/li>\n<li>It&#8217;d be even better if regression emails weren&#8217;t sent unless there were regressions on multiple platforms. This would be a little tricky.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>A few thoughts from several days of staring at these charts; I&#8217;m going to focus on the tests that generate the most email.\u00a0 Because Talos generates so much email, developers are prone to ignore it.\u00a0 And that&#8217;s not what we want them to do: getting a Talos email should be cause for consternation or celebration, [&hellip;]<\/p>\n","protected":false},"author":320,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/105"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/users\/320"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/comments?post=105"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/105\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/media?parent=105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/categories?post=105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/tags?post=105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}