Running Nightly 23 from an SD card

I’ve noticed that most Mozilla developers are using recent laptops with fast SSDs for their work. This observation shouldn’t be surprising, as developers tend to be tech enthusiasts with higher requirements, but I wonder if these fast machines could also be masking some of the performance problems in the code we write?

We’ve known for a while that I/O operations on the main thread are a major source of Firefox jank, but I think we sometimes under-estimate the urgency of refactoring the remaining sources of main-thread I/O. While Firefox might feel fast on powerful hardware, I believe a significant share of our users are still using relatively old hardware. For example Firefox Health Report data shows that 73.5% of our more-technically-inclined Beta users (Release channel data not yet available) are on a computer with 1 or 2 cores, including hyper-threaded cores.

Even when users are on modern machines, their storage systems might be slow because of hardware problems, I/O contention, data fragmentation, Firefox profiles/binaries being accessed over a network share, power-saving settings, slow laptop hard-drives, and so on. I think it would be beneficial to test certain types of code changes against slow storage by running Firefox from a network share, an SD card, or some other consistently slow I/O device.

The Experiment

As an experiment to see what it’s like to run Firefox when I/O is consistently slow, I decided to create a Firefox profile on an SD card and surfed the web for a couple of hours, capturing profiles of jank along the way. I used an SD card which advertised maximum transfer rates of 20MB/s for  reads and 15MB/s for writes, although in practice, large transfers to and from the card peaked around 10MB/s. For reference, my mechanical hard drive performs the same operations an order of magnitude faster. I left the Firefox binaries on my hard drive — I was more interested in the impact of I/O operations that could be re-factored than the impact of Firefox code size.

As I visited pages to set up the new Firefox profile with my usual customizations (extensions, pinned tabs, etc), I observed regular severe jank at every step. After entering the URL of a new site and hitting Enter, Firefox would become unresponsive and Windows would mark it as “Not Responding”. After about 5 seconds, it would start handling events again. Profiles showed that the network cache (currently being re-designed) was the most common source of these hangs. I also hit noticeable jank from other I/O bottlenecks when I performed common actions such as entering data into forms, logging into sites, downloading files, bookmarking pages, etc. Most of these issues are being worked on already, and I’m hopeful that this experiment will produce very different results in 6 months.

This is a screen recording of a simple 5-minute browsing session. The janks shown in the video are usually absent when the profile is stored on my hard drive instead. You’ll need to listen to the audio to get all the details.

Observations

The following is a selection of some of the I/O bottlenecks I encountered during my brief & unrepresentative browsing sessions with this profile.

1) Initializing a new profile takes a very long time. After first launch, even with the application binaries in the disk cache, Firefox did not paint anything for 20 seconds. It took an additional 5 seconds to show the “Welcome to Firefox” page. The “Slow SQL” section of about:telemetry showed ~40 SQL statements creating tables, indexes and views, with many taking multiple seconds each. To the best of my knowledge, there hasn’t been much research into improving profile-creation time recently. We discussed packing pre-generated databases into omni.jar a year ago (bug 699615).

2) Installing extensions is janky. The new extension’s data is inserted into the addons.sqlite and extensions.sqlite DBs on the main thread. An equal amount of time is spent storing the downloaded XPI in the HTTP cache. See profile here. Surprisingly, the URL classifier also calls into the HTTP cache, triggering jank.

3) The AwesomeBar is pretty janky at first. It seems the Places DB gets cloned several times on the main thread.

4) Unsurprisingly, startup & shutdown are slower, especially if the Add-on Manager has to update extension DBs or if extensions need to do extra initialization or cleanup. For example, AdBlock triggers main-thread I/O shortly after startup by loading the AdBlock icon from its XPI into the browser toolbar.

5) New bugs/bugs needing attention:

  • The URL classifier opens/writes/closes “urlclassifierkey3.txt” on the main thread. Filed bug 867776
  • The cookie DB is closed on the main-thread after being read into memory. Filed bug 867798
  • The startupCache file might be another good candidate for read-ahead. Filed bug 867804
  • bug 818725: localStore.rdf is fsync’ed on the main thread during GC . Not completely new, but this bug could use some attention
  • bug 789945: Preferences are flushed & fsynced on the main thread several times during a session and they can cause noticeable jank. They also show up frequently in chrome hangs.
  • bug 833545: Telemetry eagerly loads saved pings on the main thread

6) Known issues observed during browsing:

  • Network cache creates a ton of jank when storage is slow/contended
  • Password Manager, Form History, Places and Download Manager do main-thread I/O
  • SQLite DBs cause jank when opened on the main-thread
  • Font loading causes jank

13 Responses to Running Nightly 23 from an SD card

  1. For no doubt similar reasons, Firefox is also very sensitive to having profiles stored on a network mount, e.g a Linux system with NFS-mounted home directories. For my machine at work, I’ve had to explicitly move the profiles out of the home directory (which in turn means arranging separate backups for them).

  2. Hey, I was just thinking about this a couple hours ago! I got a speedy new USB3 flash drive recently, because my old one was so abysmally slow. And it just occured to me that it might be really interesting to run Firefox off it. And now you’ve already done the work. :)

    I wonder if there’s a sizable number of users who might be doing this deliberately, thinking it’s faster (because clearly solid-state is always faster than spinning platters, mumble mumble flash is like a ram disk, Vista’s “ReadyBoost”, etc). Also, Firefox Portable.

  3. I just did a quick benchmark for fun — my crappy old USB drive clocks in at 5.6MB/s for writes, and 22MB/s for reads. Based on a 500MB file copy. And my 128MB (!) minisd (!!) card from Nokia that clocks in at an astounding 3.6MB/s for writes, 6.2MB/s for reads). Not even going to bother measuring the 16MB (!!!) SD card I got with my camera eons ago. :D

  4. Hmm, this would affect everybody running Portable Firefox, wouldn’t it? Since they actually launch the application off an external drive, using a profile on that external drive (and hopefully write nothing to the hard disk)…

    • Vladan Djeric

      It depends on the USB stick. If Portable Firefox is running from a USB3 drive, it shouldn’t be nearly as bad. It could get pretty bad for older USB sticks (see Justin’s numbers above).

  5. May be the some automated test should be done on super slow I/O, Super Slow Network I/O, 1 core CPU and low memory system.

    So to iron out all the bits of performance concern. Afterall majority of Firefox users aren’t using SSD or a super fast Dual Core System.

    • Vladan Djeric

      Aaron Klotz and Joel Maher are already working on detecting main thread I/O in our test suites using xperf, see bug 644744

  6. Emanuel Hoogeveen

    I think it’s great that you’re looking into this. I’ve run into this problem for years by using my Firefox profile off a USB drive at my university (I prefer to just use my laptop but I don’t always have that luxury), and it always seemed like the project simply wasn’t ready for looking into this, what with the amount of work that needed doing even for high performance systems. It would be wonderful if inroads could be made into making this a more pleasant experience.

    One observation: The amount of jank close to the startup of the browser in this usecase has always made me question the goal of showing a usable browser as soon as possible. IMO there’s nothing more frustrating than being able to see, the browser, only to find that in the first 30-60 seconds of using it, 90% of it is waiting for it to stop hanging!

    Maybe it’s just me, but if I’m going to be using Firefox off a flash device I don’t expect it to start up fast – and I’m probably going to be using it a while before restarting (with the exception of updating Nightly). That means I don’t really care how long it takes to show up, but I do want it to be usable after that. Shutdown is a different matter, since I often have a train to catch when I leave – however, this hasn’t been much of an issue in practice.

  7. Thank you so much for doing this! I notice this all the time when I’m compiling, or sometimes when I undock my laptop and it goes into a power-preservation mode.

    The browser janks really bad, and is sometimes unusable. It’s great to see more attention towards these issues.

  8. For me the most excruciating issue with Firefox is the time I have to wait until I can use it when I first start the computer. I have an old IDE HDD and it takes 2 minutes until the browser becomes responsive. I used Windows 7 and I noticed that when windows starts, the trustedinstaller.exe process runs (I think this checks to see if windows is up to date)…

    this totally kills the browser, until windows does not finishes with this thing, Firefox cannot be used. I think it has something to do with the fact the the HDD is busy scanning things and Firefox just waits and it’s locked.

    • Vladan Djeric

      Thank you for the comment. Do you find other programs take minutes to start after reboot as well?

      • There is a lot of activity on the HDD after Win7 x64 starts as that trustedinstaller process is running and it’s causing congestion. As usually the first program I start is firefox, it’s in Firefox where I notice all the jank. I have stumble upon addon installed, gmail watcher, adblock plus, do not track me and fasterfox lite set to turbo charged.

        So as soon as firefox starts up (trusted installer is still running) I click on the gmail icon to launch my gmail page and the browser freezes up 3-4 times and I cannot use it at all.

        I have 4GB om ram and a Quad Core 6600!

        If you tell me how I can send you a debug/trace to inspect it.

        Regards and keep up the good work. I still love firefox!

        • Vladan Djeric

          Can you install the profiler extension, reboot and then capture a profile?

          https://developer.mozilla.org/en-US/docs/Performance/Reporting_a_Performance_Problem

          This won’t capture the entire startup (the profiler itself doesn’t initialize immediately on start), but it will capture some of the main thread activity after start.
          You will also want to first increase the profiler’s sampling interval to 50ms — we don’t need to record details when we’re trying to profile noticeable freezing.
          After installing the extension, click on the profiler logo, then the “Settings” button, then “Advanced Configuration”, select “Custom” sampling frequency and enter 50 in the “Sampling Interval (ms)” box and finally click “Set Values”. You can then reboot, start Firefox, and click “Analyze” after the browser has finished janking. Send the profile link to vdjeric at mozilla.com