What’s new in Perfherder?

Since last “What’s new in Perfherder” article a lot has changed. Our development team is making progresses towards automating the regression detection process. This post will cover the various improvements that have been made to Perfherder since July 2020.

Alerts view

We added tags for tests. They are meant to describe what type of test it is. For example, the alert below is the PerceptualSpeedIndex visual metric for the cold variant of reddit.

The “Tags” column is next to “Test and platform”

We improved the checkbox of the alert summaries so can all alert items be selected by a specific status.

Check alerts menu

Talos tests now have links to documentation for every alert item, so if one isn’t very familiar to the regressed/improved test, this documentation can help for a better understanding of it. The alert items can be sorted by the various columns present in the alerts view. We split the Test and platform column into Test and Platform and we are now able to sort by platform also.

The Previous Value, New Value, Absolute Difference, and Magnitude of Change were joined together into a single Magnitude of Change column as they were showing basically the same information. Last but not least, the graph link at the end of each test name was moved under the star as a graph icon.

The documentation link is present at the end of every test name and platform The sorting buttons are available next to each column

Regression template

We’ve almost automated the filing of the regression bugs. We don’t have to copy-paste anymore the details from the regressor bug but just to input its number in the dialog below and the new bug screen the fields will auto-populate. The only thing that’s left to be automated is setting the Version of the bug, which should be the latest release of Firefox. It is currently set to unspecified.

File Regression Bug Modal

The autofilled fields, Screen 1

The autofilled fields, Screen 2

The autofilled fields, Screen 3

Another cool thing that we improved is a link to the visual recordings of a browsertime pageload test. In the comment 0/description of the bug, the old and new (regressed) values are linked to a tgz archive that contains the video recording of the pageload test for each page cycle.

The before and after links are under “Absolute values” column

Compare view

We added pagination for compare view when the number of results is higher than 10 and now we don’t have the problem of loading too much results in one page anymore.

Compare view with pagination

Backfill tool

The backfill tool is probably the biggest surprise. We call it Sherlock. It runs every hour and checks that there are data points for the revision the alert is created on and the previous one. It basically makes sure that there’s no data points gap so the culprit is identified precisely.

Backfill report email

UI/UX improvements

Of course the improvements are not limited to those, we’ve made various backend and front-end cleanups and optimizations not really visible to the UI:

• The graph view tooltip was adjusted to avoid it obscuring the target text
• The empty alert summaries were removed from the database
• The Perfherder UI was improved to better indicate mouse-overs and actionable elements