In January there were 161 alerts generated, resulting in 20 regression bugs being filed on average 13.4 days after the regressing change landed.
Welcome to the January 2022 edition of the performance sheriffing newsletter. Here you’ll find the usual summary of our sheriffing efficiency metrics, followed by a review of the year. If you’re interested (and if you have access) you can view the full dashboard.
- All alerts were triaged in an average of 0.7 days
- 99% of alerts were triaged within 3 days
- Valid regressions were associated with bugs in an average of 8.8 days
- 81% of valid regressions were associated with bugs within 5 days
As you can see there has been a huge increase in the average number of days before valid regressions were associated with bugs. We have identified a number of regression alerts from January that were handled incorrectly. For most of these, a comment was left on the culprit bug instead of a new regression bug being opened. We have been taking corrective measures, including revisiting all recent regression alerts and reviewing our sheriffing workflow and training material. We’re also exploring ways to improve our tools to reduce the risk of this recurring, and looking into how we can detect such issues more expediently in the future.
Summary of alerts
Each month I’ll highlight the regressions and improvements found.
- 😍 1 bug was associated with an improvement
- 🤐 2 regressions were accepted
- 🤩 6 regressions were fixed (or backed out)
- 🤥 0 regressions were invalid
- 🤗 1 regression is assigned
- 😨 6 regressions are unassigned
- 😵 1 regression was reopened
Note that whilst I usually allow one week to pass before generating the report, there are still alerts under investigation for the period covered in this article. This means that whilst I believe these metrics to be accurate at the time of writing, some of them may change over time.
I would love to hear your feedback on this article, the queries, the dashboard, or anything else related to performance sheriffing or performance testing. You can comment here, or find the team on Matrix in #perftest or #perfsheriffs.
The dashboard for January can be found here (for those with access).