The Road to Better Performance Profiles – Part 2

Following the symbolication work from Part 1, I undertook two more efforts to improve performance profiles for the Performance team.

Native Profiling (Bugs 2030161, 2030166, 2047451, and 2030423)

The first effort was to use platform-specific profilers to generate profiles in CI of browsers running the Speedometer 3 benchmark, the industry-leading benchmark for browser performance. These ‘native’ profiles provide rich, system-wide insight into how browsers handle Speedometer 3, helping developers analyze and diagnose behaviour. They also provide us with profiles we can use to generate comparison reports for patches. We needed Raptor to generate native profiles for all of our platforms, namely Windows, macOS, Linux, and Android.

At a high level, the pipeline to generate native profiles is as follows:

1. Start the system profiler.
2. Run Speedometer 3 on Firefox or Chromium as Release using Raptor and Browsertime.
3. Stop the system profiler.
4. Symbolicate the profiles using samply and the corresponding build symbols.
5. Post-process the symbolicated profiles using profiler-edit (formerly symbolicator-cli) and create compact and labelled variants of the profiles.
6. Upload the processed profiles as artifacts.

To run tests with native profiling, you can use ./mach try fuzzy --full or ./mach try perf --full and run any test that has the -native-profiling suffix.

On Windows, we produced profiles using xperf, a tracer that uses the Event Tracing for Windows framework (ETW) to capture system-level data on applications running on Windows. To use xperf with our CI machines, scheduled tasks to run the tracer were configured in Puppet (see our ronin_puppet repository).

After adding support to Raptor to trigger these xperf tasks, we can start a trace and run the Speedometer 3 benchmark on Firefox or Chromium as Release (our custom Chromium build configured with release flags) using Browsertime. The Speedometer 3 benchmark runs 20 times, with each run in a separate browser cycle, to ensure we collect enough samples before stopping the trace.

Once completed, xperf provides a user trace and a kernel trace, which are combined into a full trace. We can then use samply to convert and symbolicate these traces into Firefox Profiler profiles, complete with markers and JIT information (Figure 1 and Figure 2).

Figure 1: Native profile of Speedometer 3 running on Firefox on Windows

Figure 2: Native profile of Speedometer 3 running on Chromium as Release on Windows

On macOS, we took a similar approach. We used samply to collect and symbolicate profiles. After implementing a workaround to allow samply to profile across multiple browser cycles and sorting out permission issues in CI, samply could profile Speedometer 3 on Firefox with minimal CI configuration and symbolicate the profiles afterwards.

On Linux, we ran perf with elevated privileges to collect profiling data system-wide before using samply to symbolicate and convert the data into a Firefox Profiler profile. Other than enabling our CI machines to run sudo perf, no additional machine configuration or workarounds were needed.

For Android, we took a slightly different approach. We used simpleperf to collect profiles on our mobile devices in CI (Samsung A55, Google Pixel 6, and Samsung S24) and samply to symbolicate them.

Initially, we wanted to profile 20 Speedometer 3 runs in a single simpleperf session, matching our desktop profiling workflow, but this proved unreliable during testing, as the on-device profiling appeared to be resource-intensive and would intermittently fail or crash.

We decided to use our simpleperf support introduced in Browsertime in Part 1 to profile the 20 individual test suites that make up a single Speedometer 3 run separately, which proved to be more stable (Figure 3). In the long term, we plan to support profiling complete Speedometer 3 runs with simpleperf on Android, bringing the workflow closer to what we currently use for desktop profiling (Bug 2032007). We will introduce native profiles for Chromium as Release (mobile) on Android in Bug 2067157.

Figure 3: Native profile of the TodoMVC-Vue Speedometer 3 test running on Fenix (Firefox for Android)

To make our Speedometer 3 profiles clearer and more helpful, we post-process them with profiler-edit, which adds labels (groupings of JS frames) to the profile (Figure 5) and also provides a more compact version (Figure 4) where all runs are placed on the same process track.

Figure 4: Compact Speedometer 3 native profile

Figure 5: Labelled Speedometer 3 native profile

In CI, we routinely generate Firefox profiles on autoland (Figure 6) and Chromium as Release profiles on mozilla-central. These native profiles have already been used to produce Speedometer 3 comparison reports (Figure 7) and investigate incidents in CI. Currently, profiles for Chromium as Release on macOS and Linux are works in progress (see Bug 2050869 and Bug 2065970, respectively).

Figure 6: Native profiles running routinely on autoland

Figure 7:  Comparison reports using native profiles (preview courtesy of Markus Stange)

Modernizing mozgeckoprofiler (Bug 1992000)

The second effort was to fully modernize symbolication in mozgeckoprofiler, the module responsible for symbolication across four performance testing frameworks: Raptor, Talos, XPCShell, and Mochitest. This mainly involved phasing out Eliot, an older symbolication API service scheduled to sunset in the near future, and fully transitioning the module’s symbolication workflow to use samply and profiler-edit.

In Part 1, we added support that allowed Raptor and Talos to symbolicate their profiles using our new approach. This left two main cases that were still handled by Eliot: XPCShell and Mochitest symbolication, and local profile symbolication.

Implementing XPCShell and Mochitest profile symbolication (Bug 1998767) was relatively straightforward. We followed the same approach used for Talos and Raptor by adding our new symbolication dependencies to XPCShell and Mochitest taskgraph configurations. Since these tests and their profiles (Figure 8) can be generated in parallel, we made sure to perform symbolication only after all profiles had been generated. You can try these jobs by running ./mach try fuzzy --profiler and selecting any XPCShell and/or Mochitest job (Figure 9).

Figure 8:  Symbolicated Mochitest profile

Figure 9:  Mochitest tests running in CI produce symbolicated profiles

Originally, when generating profiles locally from one of our performance frameworks, Eliot was used to symbolicate those profiles. To replace it with our new approach, we needed to bootstrap samply and profiler-edit onto local developer machines. Now, when you run ./mach bootstrap, the latest builds of samply and profiler-edit available in CI for your platform are automatically installed under the local .mozbuild/ directory. With this change, the use of Eliot in PerfTest has been completely replaced by our new symbolication approach, and Eliot can safely be retired from mozgeckoprofiler!

What’s Next

These patches are part of an ongoing effort to make performance profiling easier, more standardized, and more useful for Firefox developers. Here are a few related bugs to keep an eye on:

• In Bug 2050869, Bug 2065970, and Bug 2067157, we’ll add Chromium-as-Release profiling support for macOS, Linux, and Android, respectively.
• In Bug 2066906, we’ll officially remove Eliot as mozgeckoprofiler’s fallback symbolication service.
• In Bug 2032007, we’ll add support for profiling full Speedometer 3 runs with simpleperf, rather than producing a profile for each test suite.
• In Bug 2036104, we’ll continue developing reports that compare native profiles across patches, making it easier for developers to investigate performance differences and regressions.