{"id":135,"date":"2021-02-09T04:55:24","date_gmt":"2021-02-09T12:55:24","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=135"},"modified":"2021-03-01T07:22:39","modified_gmt":"2021-03-01T15:22:39","slug":"performance-sheriff-newsletter-january-2021","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2021\/02\/09\/performance-sheriff-newsletter-january-2021\/","title":{"rendered":"Performance Sheriff Newsletter (January 2021)"},"content":{"rendered":"<p>In January there were <strong>106 alerts<\/strong> generated, resulting in <strong>15 regression bugs<\/strong> being filed on average <strong>4.3 days<\/strong> after the regressing change landed.<\/p>\n<p>Welcome to the January 2021 edition of the performance sheriffing newsletter. Here you&#8217;ll find the usual summary of our sheriffing efficiency metrics, followed by some analysis of the bug products and components that were identified as the cause of regressions in 2020. If you&#8217;re interested (and if you have access) you can view the <a href=\"https:\/\/sql.telemetry.mozilla.org\/dashboard\/perf-sheriff-newsletter\">full dashboard<\/a>.<\/p>\n<h2><a href=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/sheriffing-efficiency-jan21.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignright wp-image-136 size-medium\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/sheriffing-efficiency-jan21-300x175.png\" alt=\"Sheriffing Efficiency (Jan 2021)\" width=\"300\" height=\"175\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/sheriffing-efficiency-jan21-300x175.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/sheriffing-efficiency-jan21.png 466w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a>Sheriffing efficiency<\/h2>\n<ul>\n<li>All alerts were triaged in an average of <strong>1.2 days<\/strong><\/li>\n<li><strong>90%<\/strong> of alerts were triaged within 3 days<\/li>\n<li>Valid regressions were associated with bugs in <strong>1.7 days<\/strong><\/li>\n<li><strong>100%<\/strong> of valid regressions were associated with bugs within 5 days<\/li>\n<\/ul>\n<h2>Regression Bug Analysis<\/h2>\n<p>January was a quiet month for alerts, so I thought I&#8217;d share some analysis I performed recently on the performance regression bugs identified in 2020. Note that this analysis is biased towards areas we have test coverage, areas of active development, and areas that are sensitive to performance.<\/p>\n<h3>Products<\/h3>\n<p>To create the following chart, I collected the product\/component of all bugs indicated as a regressor for regression bugs. Where no regressor bugs were given, I report the product\/component of the regression bug itself.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter wp-image-138 size-full\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814.png\" alt=\"Regression Bugs by Product (2020)\" width=\"1070\" height=\"569\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814.png 1070w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814-300x160.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814-600x319.png 600w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814-768x408.png 768w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-product-e1612378242814-1000x532.png 1000w\" sizes=\"(max-width: 1070px) 100vw, 1070px\" \/><\/p>\n<p>Perhaps unsurprisingly, the majority of the regression bugs were opened against the Core product. The <strong>16%<\/strong> in the Testing product is likely caused by <strong>Testing::Performance<\/strong> being the default product\/component for regression bugs.<\/p>\n<h3>Components<\/h3>\n<p>First, let&#8217;s take a look over the components within the <strong>Core<\/strong> product:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-139\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-core-component.png\" alt=\"Regression Bugs by Core Component 2020)\" width=\"748\" height=\"666\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-core-component.png 748w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-core-component-300x267.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-core-component-600x534.png 600w\" sizes=\"(max-width: 748px) 100vw, 748px\" \/><\/p>\n<p>To conserve space, and ensure the above chart is readable, I&#8217;ve grouped 29 components with 2 or fewer regression bugs into an &#8220;Other&#8221; category. Now let&#8217;s look at the components that fall outside of the <strong>Core<\/strong> product:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-140\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-non-core-components.png\" alt=\"Regression Bugs by Component 2020)\" width=\"748\" height=\"754\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-non-core-components.png 748w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-non-core-components-300x302.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-non-core-components-600x605.png 600w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bugs-by-non-core-components-150x150.png 150w\" sizes=\"(max-width: 748px) 100vw, 748px\" \/><\/p>\n<p>Similar to the previous chart, I&#8217;ve grouped products\/components with just one regression bug into an &#8220;Other&#8221; category.<\/p>\n<h3>Priority<\/h3>\n<p>Finally, let&#8217;s take a look at the priority of the bugs. Although <strong>37%<\/strong> have no priority set, all but one of these had an assignee, and most of them have been resolved.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter wp-image-142 size-full\" src=\"http:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bug-priorities-1.png\" alt=\"Regression Bugs by Priorities (2020)\" width=\"538\" height=\"538\" srcset=\"https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bug-priorities-1.png 538w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bug-priorities-1-300x300.png 300w, https:\/\/blog.mozilla.org\/performance\/files\/2021\/02\/bug-priorities-1-150x150.png 150w\" sizes=\"(max-width: 538px) 100vw, 538px\" \/><\/p>\n<h2>Summary of alerts<\/h2>\n<p>Each month I&#8217;ll highlight the regressions and improvements found.<\/p>\n<ul>\n<li>\ud83d\ude0d <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1679361,1684982,1681043,1684003,1685563,1640607,1675590\">7 bugs were associated with improvements<\/a><\/li>\n<li>\ud83e\udd10 <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1685539,1686963,1686963,1686963,1687234,1687238,1688819,1689536\">6 regressions were accepted<\/a><\/li>\n<li>\ud83e\udd29 <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1686018,1685703,1686035,1685703,1686190,1686964,1688820\">6 regressions were fixed (or backed out)<\/a><\/li>\n<li>\ud83e\udd25\u00a00 regressions were invalid<\/li>\n<li>\ud83e\udd17\u00a0<a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1683897\">1 regression is assigned<\/a><\/li>\n<li>\ud83d\ude28 <a href=\"https:\/\/bugzilla.mozilla.org\/buglist.cgi?bug_id=1686220\">1 regression is still open<\/a><\/li>\n<\/ul>\n<p>Note that whilst I usually allow one week to pass before generating the report, there are still alerts under investigation for the period covered in this article. This means that whilst I believe these metrics to be accurate at the time of writing, some of them may change over time.<\/p>\n<p>I would love to hear your feedback on this article, the queries, the dashboard, or anything else related to performance sheriffing or performance testing. You can comment here, or find the team on Matrix in <a href=\"https:\/\/chat.mozilla.org\/#\/room\/#perftest:mozilla.org\">#perftest<\/a> or <a href=\"https:\/\/chat.mozilla.org\/#\/room\/#perfsheriffs:mozilla.org\">#perfsheriffs<\/a>.<\/p>\n<p>The dashboard for January can be <a href=\"https:\/\/sql.telemetry.mozilla.org\/dashboard\/perf-sheriff-newsletter?p_range=2021-01-01--2021-01-31\">found here<\/a> (for those with access).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In January there were 106 alerts generated, resulting in 15 regression bugs being filed on average 4.3 days after the regressing change landed. Welcome to the January 2021 edition of &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/performance\/2021\/02\/09\/performance-sheriff-newsletter-january-2021\/\">Read more<\/a><\/p>\n","protected":false},"author":513,"featured_media":137,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[281231],"tags":[457005,117,812,281231],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/135"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/users\/513"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/comments?post=135"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/135\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/media\/137"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/media?parent=135"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/categories?post=135"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/tags?post=135"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}