In July there were 105 alerts generated, resulting in 20 regression bugs being filed on average 6.6 days after the regressing change landed.
Welcome to the July 2021 edition of the performance sheriffing newsletter. Here you’ll find the usual summary of our sheriffing efficiency metrics, followed by some details on how we’re growing the test engineering team. If you’re interested (and if you have access) you can view the full dashboard.
- All alerts were triaged in an average of 1.4 days
- 88% of alerts were triaged within 3 days
- Valid regressions were associated with bugs in an average of 2.7 days
- 89% of valid regressions were associated with bugs within 5 days
It’s initially disappointing to see the alert to bug increase last month, however after some investigation it appears that a single alert has thrown this out. With fewer alerts (as we’ve seen over the last two months), any that exceed our targets have an increased impact on our average response times. In this case, it was this alert for a 2% glvideo regresion. The backfill bot did trigger backfills for this job, however the culprit commit still wasn’t clear. Even after over 250 retriggers, the sheriff was unable to determine a culprit. Perhaps a better way to measure the effectiveness of the auto backfills is to look at the average time from alert to bug where we meet the threshold, to filter out alerts that are particularly challenging for our sheriffs.
Join the team!
I’m excited to share that after the performance test engineering team is currently hiring! We have ambitious plans to modernise and unify our performance test frameworks, automate more of our regression sheriffing workflows, increase the accuracy and sensitivity of our regression detection, and support the culture of performance at Mozilla. By growing the team we hope to accelerate these efforts and to ensure every Firefox release performs better than the last.
If you’re interested in helping us to build the future of performance testing at Mozilla, and to have a direct impact on the performance of Firefox, then please take a look over the following job descriptions:
- Software Development Engineer in Test (Performance)
- Senior Software Development Engineer in Test (Performance)
Note that the first role is based in Toronto as we have a number of team members in this location, and currently feel that hiring in this location would provide a better opportunity for the successful candidate. The senior role is remote US and Canada.
You can learn more about the team on our wiki page, and if you’re interested in joining us, you can apply directly from the job descriptions above. If these positions don’t interest you, but you like the idea of working to keep the internet open and accessible to all, take a look over our other open positions.
Summary of alerts
Each month I’ll highlight the regressions and improvements found.
- 😍 5 bugs were associated with improvements
- 🤐 7 regressions were accepted
- 🤩 3 regressions were fixed (or backed out)
- 🤥 0 regressions were invalid
- 🤗 0 regressions are assigned
- 😨 0 regressions are unassigned
- 😵 0 regressions were reopened
Note that whilst I usually allow one week to pass before generating the report, there are still alerts under investigation for the period covered in this article. This means that whilst I believe these metrics to be accurate at the time of writing, some of them may change over time.
I would love to hear your feedback on this article, the queries, the dashboard, or anything else related to performance sheriffing or performance testing. You can comment here, or find the team on Matrix in #perftest or #perfsheriffs.
The dashboard for July can be found here (for those with access).