{"id":561,"date":"2026-08-26T12:28:35","date_gmt":"2026-08-26T19:28:35","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=561"},"modified":"2026-08-31T12:14:29","modified_gmt":"2026-08-31T19:14:29","slug":"the-road-to-better-performance-profiles-part-2","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2026\/08\/26\/the-road-to-better-performance-profiles-part-2\/","title":{"rendered":"The Road to Better Performance Profiles – Part 2"},"content":{"rendered":"

Following the symbolication work from Part 1<\/a>, I undertook two more efforts to improve performance profiles for the Performance team.<\/p>\n

Native Profiling (Bugs 2030161<\/a>, 2030166<\/a>, 2047451<\/a>, and 2030423<\/a>)<\/b><\/h3>\n

The first effort was to use platform-specific profilers to generate profiles in CI of browsers running the Speedometer 3 benchmark<\/a>, the industry-leading benchmark for browser performance. These \u2018native\u2019 profiles provide rich, system-wide insight into how browsers handle Speedometer 3, helping developers analyze and diagnose behaviour. They also provide us with profiles we can use to generate comparison reports for patches. We needed Raptor<\/a> to generate native profiles for all of our platforms, namely Windows, macOS, Linux, and Android.<\/p>\n

At a high level, the pipeline to generate native profiles is as follows:<\/p>\n

    \n
  1. Start the system profiler.<\/li>\n
  2. Run Speedometer 3 on Firefox or Chromium as Release using Raptor and Browsertime<\/a>.<\/li>\n
  3. Stop the system profiler.<\/li>\n
  4. Symbolicate the profiles using samply<\/a><\/code> and the corresponding build symbols.<\/li>\n
  5. Post-process the symbolicated profiles using profiler-edit<\/code><\/a> (formerly symbolicator-cli<\/code>) and create compact and labelled variants of the profiles.<\/li>\n
  6. Upload the processed profiles as artifacts.<\/li>\n<\/ol>\n

    To run tests with native profiling, you can use .\/mach try fuzzy --full<\/code> or .\/mach try perf --full<\/code> and run any test that has the -native-profiling<\/code> suffix.<\/p>\n

    On Windows, we produced profiles using xperf<\/a><\/code>, a tracer that uses the Event Tracing for Windows framework<\/a> (ETW) to capture system-level data on applications running on Windows. To use xperf<\/code> with our CI machines, scheduled tasks to run the tracer were configured in Puppet (see our ronin_puppet<\/a><\/code> repository).<\/p>\n

    After adding support to Raptor to trigger these xperf<\/code> tasks, we can start a trace and run the Speedometer 3 benchmark on Firefox or Chromium as Release (our custom Chromium build configured with release flags) using Browsertime. The Speedometer 3 benchmark runs 20 times, with each run in a separate browser cycle, to ensure we collect enough samples before stopping the trace.<\/p>\n

    Once completed, xperf<\/code> provides a user trace and a kernel trace, which are combined into a full trace. We can then use samply<\/code> to convert and symbolicate these traces into Firefox Profiler profiles, complete with markers and JIT information (Figure 1 and Figure 2).<\/p>\n

    \"Native

    Figure 1: Native profile of Speedometer 3 running on Firefox on Windows<\/p><\/div>\n

    \"Native

    Figure 2: Native profile of Speedometer 3 running on Chromium as Release on Windows<\/p><\/div>\n

    On macOS, we took a similar approach. We used samply<\/code> to collect and symbolicate profiles. After implementing a workaround<\/a> to allow samply<\/code> to profile across multiple browser cycles and sorting out permission issues in CI, samply<\/code> could profile Speedometer 3 on Firefox with minimal CI configuration and symbolicate the profiles afterwards.<\/p>\n

    On Linux, we ran perf<\/code> with elevated privileges to collect profiling data system-wide before using samply<\/code> to symbolicate and convert the data into a Firefox Profiler<\/a> profile. Other than enabling our CI machines to run sudo perf<\/code>, no additional machine configuration or workarounds were needed.<\/p>\n

    For Android, we took a slightly different approach. We used simpleperf<\/code> to collect profiles on our mobile devices in CI (Samsung A55, Google Pixel 6, and Samsung S24) and samply<\/code> to symbolicate them.<\/p>\n

    Initially, we wanted to profile 20 Speedometer 3 runs in a single simpleperf<\/code> session, matching our desktop profiling workflow, but this proved unreliable during testing, as the on-device profiling appeared to be resource-intensive and would intermittently fail or crash.<\/p>\n

    We decided to use our simpleperf<\/code> support introduced in Browsertime in Part 1 to profile the 20 individual test suites that make up a single Speedometer 3 run separately, which proved to be more stable (Figure 3). In the long term, we plan to support profiling complete Speedometer 3 runs with simpleperf<\/code> on Android, bringing the workflow closer to what we currently use for desktop profiling (Bug 2032007<\/a>). We will introduce native profiles for Chromium as Release (mobile) on Android in Bug 2067157<\/a>.<\/p>\n

    \"Native

    Figure 3: Native profile of the TodoMVC-Vue<\/code> Speedometer 3 test running on Fenix (Firefox for Android)<\/p><\/div>\n

    To make our Speedometer 3 profiles clearer and more helpful, we post-process them with profiler-edit<\/code>, which adds labels (groupings of JS frames) to the profile (Figure 5) and also provides a more compact version (Figure 4) where all runs are placed on the same process track.<\/p>\n

    \"Compact

    Figure 4: Compact Speedometer 3 native profile<\/p><\/div>\n

    \"Labelled

    Figure 5: Labelled Speedometer 3 native profile<\/p><\/div>\n

    In CI, we routinely generate Firefox profiles on autoland<\/a> (Figure 6) and Chromium as Release profiles on mozilla-central<\/a>. These native profiles have already been used to produce Speedometer 3 comparison reports (Figure 7) and investigate incidents in CI. Currently, profiles for Chromium as Release on macOS and Linux are works in progress (see Bug 2050869<\/a> and Bug 2065970<\/a>, respectively).<\/p>\n

    \"Native

    Figure 6: Native profiles running routinely on autoland<\/p><\/div>\n

    \"Comparison

    Figure 7:\u00a0 Comparison reports using native profiles (preview courtesy of Markus Stange)<\/a><\/p><\/div>\n

    Modernizing mozgeckoprofiler (Bug 1992000<\/a>)<\/b><\/h3>\n

    The second effort was to fully modernize symbolication in mozgeckoprofiler<\/a><\/code>, the module responsible for symbolication across four performance testing frameworks: Raptor,\u00a0Talos<\/a>, XPCShell<\/a>, and Mochitest<\/a>. This mainly involved phasing out Eliot<\/a>, an older symbolication API service scheduled to sunset in the near future, and fully transitioning the module\u2019s symbolication workflow to use samply<\/code> and profiler-edit<\/code>.<\/p>\n

    In Part 1, we added support that allowed Raptor and Talos to symbolicate their profiles using our new approach. This left two main cases that were still handled by Eliot: XPCShell and Mochitest symbolication, and local profile symbolication.<\/p>\n

    Implementing XPCShell and Mochitest profile symbolication (Bug 1998767<\/a>) was relatively straightforward. We followed the same approach used for Talos and Raptor by adding our new symbolication dependencies to XPCShell and Mochitest taskgraph configurations. Since these tests and their profiles (Figure 8) can be generated in parallel, we made sure to perform symbolication only after all profiles had been generated. You can try these jobs by running .\/mach try fuzzy --profiler<\/code> and selecting any XPCShell and\/or Mochitest job (Figure 9).<\/p>\n

    \"Symbolicated

    Figure 8:\u00a0 Symbolicated Mochitest profile<\/p><\/div>\n

    \"Mochitest

    Figure 9:\u00a0 Mochitest tests running in CI produce symbolicated profiles<\/p><\/div>\n

    Originally, when generating profiles locally from one of our performance frameworks, Eliot was used to symbolicate those profiles. To replace it with our new approach, we needed to bootstrap samply<\/code> and profiler-edit<\/code> onto local developer machines. Now, when you run .\/mach bootstrap<\/code>, the latest builds of samply<\/code> and profiler-edit<\/code> available in CI for your platform are automatically installed under the local .mozbuild\/<\/code> directory. With this change, the use of Eliot in PerfTest has been completely replaced by our new symbolication approach, and Eliot can safely be retired from mozgeckoprofiler<\/code>!<\/p>\n

    What’s Next<\/h3>\n

    These patches are part of an ongoing effort to make performance profiling easier, more standardized, and more useful for Firefox developers. Here are a few related bugs to keep an eye on:<\/p>\n