Early in the fall, I was talking about integrating the side-by-side tool in Continuous Integration (CI). This started as a script for generating a side-by-side comparison between two page load test runs, and emphasizes the visual differences. Using this tool, you can see more clearly which metrics have regressed, and how they might be experienced by a user. The first milestone was to have a Minimum Viable Product (MVP).
What is the side-by-side comparison?
The page load tests measure the performance of Firefox (and competitors) browsers and run in Taskcluster (which is our CI) which are then visualized in Treeherder. Perfherder is our tool for catching performance regressions. When a regression bug is filed, the author of the regressing commit is needinfo-ed and asked for a fix. Sometimes the issue is not obvious and additional debugging information is needed, in which case the side-by-side comparison is very useful to help preview the impact the regression would have on the end-user.
Side-by-side integration
Last summer, we refactored the script to use it as a pypi library in mozilla-central. Then we added it as a perftest-tools command so we can use it just like mach perftest-tools side-by-side. Then we integrated it into CI so we can generate comparisons on demand. In December last year, we launched it to have side-by-side video comparisons generated automatically on regressions.
Usage
We can use this tool in three ways:
- Locally via
./mach perftest-tools side-by-side. - On CI, as an action task
- either by specifying the repo and revision to compare against
- or leaving the fields as they are and letting the tool search for the nearest revision the job instance ran on
- Automatically on every browsertime page load regression.
How to generate a side-by-side video locally
The local command will download the video results of the specified test and generate the comparisons for the cold and warm modes, such as MP4 videos, GIF animations, and GIF slow motion animations.
The command for generating the side-by-side comparison locally
The FFMPEG codec working
The expected output of the side-by-side comparison
We currently recommend running the tool on Linux, as this is the platform we used in development, and in our CI environment.
How to generate a side-by-side video in CI
If you want to generate a comparison against two revisions for a particular test that ran on autoland, you don’t have to push to try. You need to go to the Treeherder job view, select a browsertime tp6 page load test, and either trigger the comparison from the action task or run it locally as explained in the previous section – whichever is more convenient for you. The results of the task triggered on CI will be available as artifacts in the side-by-side job.
Details Panel of Amazon page load job
Custom Action menu of the Amazon page load job
side-by-side Action Task
The comparison is triggered from the revision that caused the regression (called the “after” revision) because the regression bugs are filed for the commits that caused it, so the side-by-side comparison will show the “before” test on the left and the “after” test on the right.
Currently, there are project and revision parameters available for running the comparison. The first one is usually left as it is because most comparisons are made on revisions that triggered performance alerts on autoland and our recommendation is to not compare tests against two different projects/repositories unless you know what you’re doing. If you want to compare two tests from try, then you’ll have to copy the before revision, go to the after revision on try, select the page load test you want to compare, go to Custom Action… from the Details Panel menu, and paste the before revision to the revision field along with typing try instead of autoland.
side-by-side Action Task on try
Automatic side-by-side videos for every regression
Automatic Backfilling Report
Ebay page load test and associated side-by-side comparison job
This is one of the most impactful features because it improves the developer experience. Every time Perfherder detects a performance regression, an alert is created and the perf sheriff bot does the backfills automatically. A backfill triggers the missing jobs for the regressed test so the performance sheriff can identify the regressor. After this happens, the side-by-side comparison is triggered with the backfills and by the time the regression bug gets to the author of the regressing commit, the comparison is ready to be visualized. The thing with this side-by-side comparison is it shows the visual impact of the regression. Sometimes the numbers don’t give a feel about the impact of the performance changes, but visualizing the generated comparison will help determine how the end-user perceives it.
What’s next?
As I was saying, this is an MVP of the side-by-side tool that mostly does the backend job. The future plans are to improve the Treeherder Job View so that the comparison can be triggered more easily, to embed the videos in various places, to include more details about the comparison job in the Details panel, and the development will continue according to the needs of making the developer experience as smooth as possible.
For more details or questions you can find us on the #perftools element channel.
No comments yet
Comments are closed, but trackbacks are open.