{"id":373,"date":"2022-06-30T17:15:24","date_gmt":"2022-07-01T00:15:24","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=373"},"modified":"2022-07-04T08:03:58","modified_gmt":"2022-07-04T15:03:58","slug":"introducing-a-new-performance-comparison-tool-perfcompare","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2022\/06\/30\/introducing-a-new-performance-comparison-tool-perfcompare\/","title":{"rendered":"Introducing A New Performance Comparison Tool – PerfCompare"},"content":{"rendered":"

Introduction<\/span><\/h2>\n

I have been a member of the performance test team for the past two and a half years. During that time, the performance team has had many discussions about improving the developer experience when running performance tests.\u00a0<\/span><\/p>\n

The most significant pain points for developers are that: <\/span><\/p>\n

1) tests are difficult to run, or they don’t know which tests to run, and <\/span><\/p>\n

2) our tools for comparing these results are complex or confusing.<\/span><\/p>\n

One of the tools we currently use is called Perfherder<\/a>. Perfherder does many things, such as alerting on performance improvements and regressions, comparing results of performance tests, and graphing those results.<\/span><\/p>\n

\"perfherder

Perfherder Alerts View<\/p><\/div>\n

Our performance sheriffs use Perfherder Alerts View to investigate possible improvements or regressions. They first determine if the alert is valid and then identify the culprit revision that caused a regression or improvement.<\/span><\/p>\n

The sheriff then files a bug in Bugzilla. If it is a regression, a :needinfo<\/code> flag is added for the author of that commit.<\/span><\/p>\n

\"performance

Example of a performance regression bug<\/p><\/div>\n

The user workflows for the Compare View and Graphs View are usually one of two scenarios:<\/span><\/p>\n

Scenario A:<\/strong> An engineer is tagged in a regression (or improvement) by a performance sheriff, as described above. The engineer then needs to determine why the patch caused a regression, which can take many iterations of altering code, running tests, and comparing the results.<\/span><\/p>\n

or<\/em>\u00a0<\/span><\/p>\n

Scenario B:<\/strong> A firefox engineer working on a patch will run performance tests to ensure the latest changes have not caused performance to regress. Once tests are complete, the engineer compares the results before and after those changes were made.<\/span><\/p>\n

The proactive approach, Scenario B, is highly preferable to a reactive one, as in Scenario A.<\/span><\/p>\n

A primary goal of the performance test team is to simplify and improve the experience of running performance tests and comparing the results. <\/span><\/p>\n

If we succeed in this, I believe we can improve the performance of Firefox as a whole.<\/span><\/p>\n

By providing better tools and workflows, we will empower our engineers to include performance testing as a part of their standard development process.<\/span><\/p>\n

Perfherder User Experience<\/span><\/h2>\n
\"Perfherder

Perfherder Compare View – Search for Revision<\/p><\/div>\n

\"perfherder

Perfherder Compare View – Comparison Results<\/p><\/div>\n

We have conducted usability testing and user research to identify developer pain points for Perfherder and attempted to address some of these issues. <\/span><\/p>\n

However, sometimes these changes created an even more cluttered UI. Some of these changes included adding a User Guide or more documentation to explain how to use the tools, which is a problematic approach.<\/span><\/p>\n

If we need to write more documentation to explain how to use a tool that should be reasonably straightforward, it suggests that there are fundamental weaknesses in our UX design. <\/span><\/p>\n

Perfherder was initially developed for a particular use case and has been expanded to include more features and test harnesses over the years. <\/span><\/p>\n

While it has served its purpose, the time has come to develop a new comparison tool to support the wide range of features that we have added to Perfherder–and some new ones too!<\/span><\/p>\n

Introducing PerfCompare<\/span><\/h2>\n

\"PerfCompare<\/p>\n

PerfCompare is a performance comparison tool that aims to replace Perfherder Compare View. This is a task I have wanted to undertake for quite some time, but we had a smaller team and lacked the resources to justify a project of this size.<\/span><\/p>\n

Although I am an engineer, I am also an artist. I studied art for years, attending a residential arts school my junior year, and (very briefly) pursuing a double major in Computer Engineering and Art. <\/span><\/p>\n

As a career, however, I knew that software engineering was what I wanted to pursue, while art remained an enjoyable pastime.\u00a0<\/span><\/p>\n

As such, I am very opinionated on matters of aesthetics and design. My goal is to create a tool that is easy to use–but also enjoyable<\/em> to use! <\/span><\/p>\n

I want to create something to delight engineers and make them want<\/strong> to run performance tests.<\/span><\/p>\n

I approached this project with a blank page–literally! When creating initial designs, I sat in front of a white sheet of paper with a black pen. In my mind, I asked, “What does the user want to do?”<\/em> <\/span><\/p>\n

I broke this down into the most simple actions, step by step. I wanted to remove anything unnecessary or confusing, and create a clean and aesthetic UI.<\/span><\/p>\n

Since then, three other engineers have joined the project. Together we have been working to create an MVP to announce and demo at the All Hands in September.<\/span><\/p>\n

We aim to have a product ready for initial user testing and enroll it in Mozilla’s foxfooding program.<\/span><\/p>\n

While some of these are still under development, below are some highlights of improvements and features we have implemented so far.<\/span><\/p>\n

Highlights<\/h2>\n