{"id":485,"date":"2023-12-13T02:01:21","date_gmt":"2023-12-13T10:01:21","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=485"},"modified":"2023-12-13T04:01:35","modified_gmt":"2023-12-13T12:01:35","slug":"new-sheriffing-feature-and-significant-updates-to-kpi-reporting-queries","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2023\/12\/13\/new-sheriffing-feature-and-significant-updates-to-kpi-reporting-queries\/","title":{"rendered":"New Sheriffing feature and significant updates to KPI reporting queries"},"content":{"rendered":"

A year ago I was sharing<\/a> how a Mozilla Performance Sheriff catches performance regressions, the entire Workflow<\/a> they go through, and the incoming improvements. Since I joined the Performance Tools Team<\/a> (formerly Performance Test), almost five years ago, a whole lot of improvements<\/a> have been made, and features have been added.<\/p>\n

In this article, I want to focus on a special set of features, that give the Performance Sheriffs more control over the Sheriffing Workflow (from when an alert is triggered, triaged to when the regression bug is filed and linked to the alert). We call them time-to-triage<\/b> (from alert to triage) and time-to-bug<\/b> (from alert to bug). They are actually the object of our Sheriffing Team’s KPIs<\/b>, the KPIs that measure the performance of the Performance Sheriffs team (I like puns).<\/p>\n

The time-to-triage<\/b> KPI measures the time since an alert was triggered by a performance change to when it was triaged (basically first-time analysis). It is at most 3 days<\/i>, and at least 80%<\/i> of the sheriffed alerts have to meet this deadline (or 20% is allowed not to). However, our team does not work weekends and they have to be excluded. For example, if an alert was created on a Friday (any), the three-day-triage time ends on Monday instead of Wednesday when the three business days actually expire. This means we basically only get a single day to triage it. So every time something like this happens, we have to manually exclude those alerts from the old queries of the KPI report that do not exclude the weekends from those times. The new queries do this exclusion automatically.<\/p>\n

 <\/p>\n

\"Triage

Triage Response Times (time-to-triage)
Year To Date<\/p><\/div>\n

\"Triage

Triage Response Times (New Query)
Year To Date<\/p><\/div>\n

\"Alerts

Alerts Exceeding Triage Target
Year To Date<\/p><\/div>\n

The same thing is true for an alert created on a weekend, where a part of the alert-to-triage time falls on the weekend. Actually, the only alerts that can not capture weekends are the ones created Monday and Tuesday.<\/p>\n

The time-to-bug<\/b> KPI measures the time since an alert was triggered by a performance change to when a bug was linked to the alert. It is at most 5 days<\/i>, and at least 80%<\/i> of the valid regression alerts must meet this deadline (or 20% is allowed not to). The only alerts that can not capture weekends within this KPI are the ones created on Monday, the first hour in the morning, whose KPI ends Friday in the last hour of the day.<\/p>\n

\"Regression

Regression Bug Response Times
Year To Date<\/p><\/div>\n

\"Regression

Regression Bug Response Times (New Query)
Year To Date<\/p><\/div>\n

\"Regressions

Regressions Exceeding Bug Target
Year To Date<\/p><\/div>\n

In the images above, you can see a difference in the percentages of time-to-triage<\/b> (86.9%<\/span> vs. 97.9%<\/span> old query vs. new query<\/i>) and time-to-bug<\/b> (75.7%<\/span> vs. 97%<\/span> old query vs. new query<\/i>). This is not because the Sheriffing Team is doing a better job, they were doing this the whole time. It is because the feature we developed helps measure the percentages accurately by excluding the weekends from the calculated times. According strictly to the percentages, the impact of this feature is significant, taking us from an average – maybe struggling – performance, to a really good one. Of course, the inclusion of weekends in the report of the KPIs was known a while ago, but having a bigger picture and concrete metrics is more revealing.<\/p>\n

The development of these time-to-triage\/time-to-bug features is full-stack and involved:<\/p>\n