{"id":152,"date":"2021-04-28T03:22:31","date_gmt":"2021-04-28T10:22:31","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=152"},"modified":"2021-04-28T12:30:53","modified_gmt":"2021-04-28T19:30:53","slug":"performance-sheriff-newsletter-march-2021","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2021\/04\/28\/performance-sheriff-newsletter-march-2021\/","title":{"rendered":"Performance Sheriff Newsletter (March 2021)"},"content":{"rendered":"<p>In March there were <strong>288 alerts<\/strong> generated, resulting in <strong>28 regression bugs<\/strong> being filed on average <strong>4 days<\/strong> after the regressing change landed.<\/p>\n<p>Welcome to the March 2021 edition of the performance sheriffing newsletter. Here you&#8217;ll find the usual summary of our sheriffing efficiency metrics, followed by some analysis on the data footprint of our performance metrics. If you&#8217;re interested (and if you have access) you can view the <a href=\"https:\/\/sql.telemetry.mozilla.org\/dashboard\/perf-sheriff-newsletter\">full dashboard<\/a>.<\/p>\n<h2>Sheriffing efficiency<\/h2>\n<ul>\n<li>All alerts were triaged in an average of <strong>1.2 days<\/strong><\/li>\n<li><strong>92%<\/strong> of alerts were triaged within 3 days<\/li>\n<li>Valid regressions were associated with bugs in an average of <strong>1.5 days<\/strong><\/li>\n<li><strong>96%<\/strong> of valid regressions were associated with bugs within 5 days<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-158\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/se-march-2021.png\" alt=\"Sheriffing Efficiency (Mar 2021)\" width=\"832\" height=\"234\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/se-march-2021.png 832w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/se-march-2021-300x84.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/se-march-2021-600x169.png 600w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/se-march-2021-768x216.png 768w\" sizes=\"(max-width: 832px) 100vw, 832px\" \/><\/p>\n<p>Interestingly, the close correlation we&#8217;ve seen between alerts and time to bug did not continue into March. It&#8217;s not clear why this might be, however there were some temporary adjustments to the sheriffing team during this time. We also saw an increase in the percentage of alert summaries that marked as invalid, which might have an impact of our sheriffing efficiency.<\/p>\n<h2>What&#8217;s new in Perfherder?<\/h2>\n<p>I last provided an update on Perfherder <a href=\"https:\/\/blog.mozilla.org\/performance\/2020\/07\/15\/whats-new-in-perfherder\/\">in July 2020<\/a> so felt it was about time to revisit.<\/p>\n<h3>Compact Bugzilla summaries &amp; descriptions<\/h3>\n<p>Until recently, Perfherder would simply try to include all affected tests and platforms in the summary and description for all regression bugs. Not only does this make the bugs difficult to read, it also meant we hit the maximum field size for regressions that impacted a large number of tests.<\/p>\n<p><a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1697112\">Bug 1697112<\/a> is an example of how this looked before the recent change. The description contained 24 regression alerts, and 22 improvement alerts. The summary was edited by a performance sheriff to fit within the maximum field size:<\/p>\n<blockquote><p>4.55 &#8211; 18.83% apple ContentfulSpeedIndex &#8230; tumblr SpeedIndex (windows10-64-shippable) regression on push 6ea4d69aa5c6c7064d3b4a195bf96617baa3aebf (Thu March 4 2021)<\/p><\/blockquote>\n<p>With the recent improvements we limit how many tests are named in the summary and show a count of the omitted tests. We now list common names for the affected platforms, and no longer include the suspected commit hash. For the description, when we have many alerts we now show the most\/least affected and indicate that one or more have been omitted for display purposes. <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1706333\">Bug 1706333<\/a> is an example ofthe improved description and summary:<\/p>\n<blockquote><p>122.22 &#8211; 2.73% cnn-ampstories FirstVisualChange \/ ebay ContentfulSpeedIndex + 22 more (Windows) regression on Fri April 16 2021<\/p><\/blockquote>\n<h3>Compare view sorting<\/h3>\n<p>We&#8217;ve added the ability to sort columns in <a href=\"https:\/\/treeherder.mozilla.org\/perfherder\/comparechooser\">compare view<\/a>. This is useful when you&#8217;re comparing many tests and you&#8217;d like to quickly sort the results by confidence, delta, or magnitude.<\/p>\n<div id=\"attachment_156\" style=\"width: 610px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57.png\"><img aria-describedby=\"caption-attachment-156\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-156 size-large\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57-600x238.png\" alt=\"Compare view sorted by confidence\" width=\"600\" height=\"238\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57-600x238.png 600w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57-300x119.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57-768x305.png 768w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57-1000x397.png 1000w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.37.57.png 1237w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><p id=\"caption-attachment-156\" class=\"wp-caption-text\">Compare view sorted by confidence<\/p><\/div>\n<h3>Infrastructure changelog<\/h3>\n<p>Last year we created a unified changelog consolidating commits from repositories related to our automation infrastructure. Changes to infrastructure can impact our performance results and time can be wasted investigating regressions in our products that aren&#8217;t there. To help with this, we now annotate Perfherder graphs with data from the infrastructure changelog. When one of these markers correlates to an alert it can provide a valuable clue for our sheriffs. The repositories monitored for changes can be found here <a href=\"https:\/\/github.com\/mozilla\/treeherder\/blob\/master\/treeherder\/changelog\/repositories.json\">here<\/a>.<\/p>\n<div id=\"attachment_157\" style=\"width: 610px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08.png\"><img aria-describedby=\"caption-attachment-157\" decoding=\"async\" loading=\"lazy\" class=\"size-large wp-image-157\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08-600x179.png\" alt=\"Perfherder graph showing infrastructure changelog\" width=\"600\" height=\"179\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08-600x179.png 600w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08-300x89.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08-768x229.png 768w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/04\/Screenshot-2021-04-26-at-15.47.08.png 802w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><p id=\"caption-attachment-157\" class=\"wp-caption-text\">Graph showing infrastructure changelog markers<\/p><\/div>\n<h3>Stop alerting on tier 3 jobs<\/h3>\n<p>After updating our <a href=\"https:\/\/www.mozilla.org\/en-US\/about\/governance\/policies\/regressions\/\">Performance Regressions Policy<\/a> to explicitly mention that the sheriffs do not monitor tier 3 jobs, we fixed Perfherder to prevent these from alerting. Anything running below tier 2 is considered unstable, and not a valuable performance indicator.<\/p>\n<h3>Reduced data footprint<\/h3>\n<p>We have also spent <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1665857\">a lot of effort<\/a> reducing the data footprint of our performance data by updating and enforcing our <a href=\"https:\/\/treeherder.readthedocs.io\/data_cycling.html\">data retention policy<\/a>. You can read more about our data footprint in <a href=\"https:\/\/blog.mozilla.org\/performance\/2021\/03\/08\/performance-sheriff-newsletter-february-2021\/\">last month&#8217;s newsletter<\/a>.<\/p>\n<h3>Email reports<\/h3>\n<p>When working on our data retention policy we wanted some way of reporting the signatures that were being deleted, and so we introduced email reports. We&#8217;re also now sending reports for automated backfills, and in the future we&#8217;d like to generate more reports. If you&#8217;re curious, these are being sent to <a href=\"https:\/\/groups.google.com\/a\/mozilla.com\/g\/perftest-alerts\">perftest-alerts<\/a>.<\/p>\n<h3>Bug fixes<\/h3>\n<p>The following bug fixes are also worth highlighting:<\/p>\n<ul>\n<li><a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1632730\">Graph y-axis labels were sometimes being truncated<\/a><\/li>\n<li><a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1676498\">Dates were appearing in the future<\/a><\/li>\n<\/ul>\n<h3>Acknowledgements<\/h3>\n<p>These updates would not have been possible without <a href=\"https:\/\/people.mozilla.org\/p\/r--rN2QCNVkXD2z_ljQSFT47g==\">Ionu\u021b Goldan<\/a>, <a href=\"https:\/\/people.mozilla.org\/p\/r--QKqQcEIDWmqjBwgGQgH1Yw==\">Alexandru Irimovici<\/a>, <a href=\"https:\/\/people.mozilla.org\/p\/alexandrui\">Alexandru Ionescu<\/a>, <a href=\"https:\/\/people.mozilla.org\/p\/andraesanu\">Andra Esanu<\/a>, <a href=\"https:\/\/people.mozilla.org\/p\/r--AnqQZ3EbWIm9w9Y4GpTDGQ==\">Beatrice Acasandrei<\/a> and <a href=\"https:\/\/people.mozilla.org\/p\/bebe\">Florin Strugariu<\/a>. Thanks also to the Treeherder team for reviewing patches and supporting these contributions to the project. Finally, thank you to all of the Firefox engineers for all of your bug reports and feedback on Perfherder and the performance workflow. Keep it coming, and we look forward to sharing more updates with you all soon.<\/p>\n<h2>Summary of alerts<\/h2>\n<p>Each month I&#8217;ll highlight the regressions and improvements found.<\/p>\n<ul>\n<li>\ud83d\ude0d <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1694840,1694248,1682194,1526970,1694909,1690842,1687022,1693483,1689413,1696709,1696709,1689413,1695917,1696447,1664881,1697904,1697803,1697479,1687977,1691464,1699088,1675614\">20 bugs were associated with improvements<\/a><\/li>\n<li>\ud83e\udd10 <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1697455,1698023,1698072,1699054,1699775,1700295\">6 regressions were accepted<\/a><\/li>\n<li>\ud83e\udd29 <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1694248,1695879,1694248,1694248,1694248,1695879,1694248,1697803,1689413,1700313,1701146,1701648,1701806,1698156\">9 regressions were fixed (or backed out)<\/a><\/li>\n<li>\ud83e\udd25\u00a00 regressions were invalid<\/li>\n<li>\ud83e\udd17 0 regressions are assigned<\/li>\n<li>\ud83d\ude28\u00a0<a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1653659,1701147,1653659,1701866\">3 regressions are still open<\/a><\/li>\n<li>\ud83d\ude35\u00a0<a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1669840\">1 regression was reopened<\/a><\/li>\n<\/ul>\n<p>Note that whilst I usually allow one week to pass before generating the report, there are still alerts under investigation for the period covered in this article. This means that whilst I believe these metrics to be accurate at the time of writing, some of them may change over time.<\/p>\n<p>I would love to hear your feedback on this article, the queries, the dashboard, or anything else related to performance sheriffing or performance testing. You can comment here, or find the team on Matrix in <a href=\"https:\/\/chat.mozilla.org\/#\/room\/#perftest:mozilla.org\">#perftest<\/a> or <a href=\"https:\/\/chat.mozilla.org\/#\/room\/#perfsheriffs:mozilla.org\">#perfsheriffs<\/a>.<\/p>\n<p>The dashboard for March can be <a href=\"https:\/\/sql.telemetry.mozilla.org\/dashboard\/perf-sheriff-newsletter?p_range=2021-03-01--2021-03-31\">found here<\/a> (for those with access).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In March there were 288 alerts generated, resulting in 28 regression bugs being filed on average 4 days after the regressing change landed. Welcome to the March 2021 edition of &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/performance\/2021\/04\/28\/performance-sheriff-newsletter-march-2021\/\">Read more<\/a><\/p>\n","protected":false},"author":513,"featured_media":153,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[281231],"tags":[457005,117,812,281231],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/152"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/users\/513"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/comments?post=152"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/152\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/media\/153"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/media?parent=152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/categories?post=152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/tags?post=152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}