Background

We have recently begun the process of updating our warm page load tests to be more representative of real user behavior.

Cold page load is defined as the first page load, just after the initial startup of the browser. Warm page load is any load of a page after the first page load, and the cache has been populated with some data for the page, i.e. the cache is “warmed up”. The results of these tests can be found on the AreWeFastYet dashboard, also referred to as AWFY. Below is an example of one of our AWFY graphs:

Our page load tests currently load the test URL 25 times, and navigate to about:blank in between each iteration.

This is not realistic behavior for a user to perform, however. Most users will never navigate to about:blank before loading a new url, and rarely reload the same URL again and again (unless perhaps waiting for tickets to go on sale 🎟).

Why These Changes?

Project Fission is Mozilla’s implementation of site isolation in Firefox.

When Fission is enabled, each site is run in a different process. Loading a sub-page of the same domain will not destroy the process, but when navigating to about:blank, the process may be destroyed. With the bfcache (back-forward cache) in Fission, the old page may be kept in a frozen state for a while, but not every page can enter the bfcache, such as those using unload handlers or webRTC.

For these reasons, we are currently updating these tests to navigate to another page on the same site between each iteration. This is more representative of user behavior: to load a site, navigate to another page on that site, navigate back to the home page, and repeat. The process termination when navigating to about:blank has been temporarily disabled until all tests have been updated.

Issues Encountered

At the time of writing this post, the update process is around 60% complete–19 of 33 tests have been updated. Several of the tests have encountered unexpected errors when re-recording these tests.

As an example, amazon is the only test we run on mozilla-central with the profiler enabled. (While developers have the ability to run any test with the profiler enabled, we have one site running regularly to detect issues related to this functionality.)

When running the test with a secondary URL while the profiler is running, the browser will successfully load the secondary URL, but crashes when it navigates back to the test URL.

Mitmproxy is the third-party software we use to record page loads. It has recently been updated from version 6 to 7. Several of the tests that encountered issues with testing using a secondary url while running mitmproxy 6 seem to be resolved when running using mitmproxy 7.

Going Forward

In the future, we plan to continue making our tests more representative of how users behave. Greg Mierzwinski has begun working on this with his responsiveness tests.

Thanks for reading! For more information on the update to mitmproxy 7, read my earlier post Upgrading Page Load Tests to Use Mitmproxy 7.

/kimberlythegeek

Related Links

• Page Load Test Update Tracking Doc
• Notes on Page Load Recordings
• Tracking Bug for Secondary URL Page Load Tests