Since last “What’s new in Perfherder” article a lot has changed. Our development team is making progresses towards automating the regression detection process. This post will cover the various improvements that have been made to Perfherder since July 2020.
We added tags for tests. They are meant to describe what type of test it is. For example, the alert below is the PerceptualSpeedIndex visual metric for the cold variant of reddit.
We improved the checkbox of the alert summaries so can all alert items be selected by a specific status.
Talos tests now have links to documentation for every alert item, so if one isn’t very familiar to the regressed/improved test, this documentation can help for a better understanding of it. The alert items can be sorted by the various columns present in the alerts view. We split the Test and platform column into Test and Platform and we are now able to sort by platform also.
The Previous Value, New Value, Absolute Difference, and Magnitude of Change were joined together into a single Magnitude of Change column as they were showing basically the same information. Last but not least, the graph link at the end of each test name was moved under the star as a graph icon.
We’ve almost automated the filing of the regression bugs. We don’t have to copy-paste anymore the details from the regressor bug but just to input its number in the dialog below and the new bug screen the fields will auto-populate. The only thing that’s left to be automated is setting the Version of the bug, which should be the latest release of Firefox. It is currently set to unspecified.
Another cool thing that we improved is a link to the visual recordings of a browsertime pageload test. In the comment 0/description of the bug, the old and new (regressed) values are linked to a tgz archive that contains the video recording of the pageload test for each page cycle.
We added pagination for compare view when the number of results is higher than 10 and now we don’t have the problem of loading too much results in one page anymore.
The backfill tool is probably the biggest surprise. We call it Sherlock. It runs every hour and checks that there are data points for the revision the alert is created on and the previous one. It basically makes sure that there’s no data points gap so the culprit is identified precisely.
Of course the improvements are not limited to those, we’ve made various backend and front-end cleanups and optimizations not really visible to the UI:
- The graph view tooltip was adjusted to avoid it obscuring the target text
- The empty alert summaries were removed from the database
- The Perfherder UI was improved to better indicate mouse-overs and actionable elements