Understanding Performance Impact

A few years ago, a small group of engineers at Mozilla introduced a process to identify the tasks that would have the greatest impact on the performance of Firefox. They would gather each week to look over user submitted profiles, and discuss bugs reports. Each bug would then be assigned a category and a score to reflect its impact on performance. This would help teams to prioritise their work for performance, and proved crucial in delivering the significant speed improvements that were present in Firefox Quantum.

Fast forward to today, and this performance triage process has continued, but you’d be forgiven for not knowing about it. This year we have been making improvements to the way bugs are nominated for triage, how the impact and keywords are determined, and getting more people involved. I’d like to share some of these changes with you, starting with how to request a performance impact review for a bug.

How performance impact reviews are requested

If you believe a bug may have an impact on the performance of our products, you can nominate it for triage. You can do this by editing the bug and setting the “Performance Impact” flag to “?”.

Performance Impact in Bugzilla

This will cause the bug to show up in the queries used by our triage sheriffs, who will attempt to determine the performance impact at the next meeting. If you have any additional details that may help, we encourage you to mention these as a comment in the bug. If you’re interested in joining the discussion, you can reach out to the lead of the next triage meeting published here.

How performance impact is determined

In the past we relied on the knowledge and experience of the triage sheriffs attending the meeting to determine the performance impact. Whilst efficient, this approach wasn’t great for onboarding new members to the triage rotation, and could also be inconsistent. We resolved this by building a tool to calculate the impact from a series of simple prompts.

Performance Impact Calculator

Starting with a base impact score of zero, each answer in the calculator either increases the base impact score or applies a multiplier. For example:

• Causes noticeable jank: +2
• Severe page load impact: +10
• Affects major websites: ×5
• Reproduces in Chrome: ×0.3
• Total impact score = (2+10)×5×0.3 = 18

For insight into how each of the options affects the score, you can check the “Debug” button to display a breakdown next to the labels.

Calculator with debug enabled

If the score is greater than zero then the bug is considered to have an impact on performance. The scores relate to the impact flag values as follows:

• 0: none
• 0-40: medium
• 40+: high

Note that the calculator is being adjusted regularly to ensure it gives an accurate reflection of the performance impact, therefore the values above may at some point be out of date. If you disagree with a performance impact result, please add a comment on the bug with your concerns and either needinfo the triage sheriff or re-nominate the bug for triage. Feel free to try the calculator out for yourself, and if you have any suggestions for improvements, please open an issue.

How performance impact affects you

It’s important to note that the performance team does not directly interfere with the priorities of bugs outside of their own components. During triage, bugs will be reassigned wherever possible to the most appropriate component, and it is the responsibility of the triage owners for those components to set a priority. The goal of the performance impact flag is to provide additional context that may assist the triage owners in their role. That said, a bug with a high performance impact on the product quality and experience could reasonably be expected to cause a Firefox user to switch browsers, and therefore should be considered S2 in severity.

How to get help

The performance triage wiki serves as a guide for running the triage, and goes into some detail on the Bugzilla fields and queries used. Please note that the impact calculator is constantly being tweaked, and your feedback is essential for helping us to improve the formula. If you notice a bug with a performance impact that doesn’t feel quite right, please let us know. You can do this by changing the “performance impact” flag to “?” and adding a comment, or by reaching out in #perf-help on Matrix or Slack.