Add-on performance problems

NOTE TO READERS: I wrote this post to draw attention to a common pitfall when using SQLite DBs in extension and Mozilla code. I am not suggesting people stop using the addons mentioned below. The list of top sources of slow SQL is skewed by various factors: more popular add-ons are more likely to be in this list, add-ons that do slow operations against their own DBs will be in the list but add-ons that do slow operations against Firefox DBs will not, etc etc.

The perf team has been using Telemetry data to find and fix performance problems in Firefox code, but the same data can also be used to find performance problems in add-ons.

The Telemetry data on long-running SQL statements, in particular, contains information about slow SQL operations carried out by add-ons. The aggregated reports are publicly accessible from the Metrics dashboardΒ  (BrowserID account needed). If you navigate to the slow SQL section or click on this link directly, you will see stats on SQL statements that took longer than 100ms to execute on the main thread across at least 100 browsing sessions.Β In general, main thread I/O (including SQL) is a major performance problem in Firefox as the duration of I/O operations is essentially unbounded and any long running operation on the main thread causes browser unresponsiveness. As a privacy precaution, Telemetry never reports statement parameters and only reports the name of the DB if the SQL was executed against an extension DB. Note that the dashboard doesn’t specify whether an SQL string was executed by Firefox or add-on code, but SQL executed against extension DBs has the form “Untracked SQL for <extension DB name>.sqlite”. As can be seen from that dashboard, the majority of main-thread SQL work is still coming from Mozilla code, but addon SQL is well represented.

These are the top 20 addons doing main thread DB work*, found by sorting the slow SQL table on frequency (“Total Doc”):

  1. Various extensions which use the customizable toolbar code. These toolbars sometimes ship with malware, but they also seem to be used in legitimate AMO extensions
  2. The popular ForecastFox weather extension
  3. Fast Dial visual bookmarks
  4. Evernote Web Clipper
  5. Zotero research tool
  6. toolbar
  7. MailCheck + MailCheck
  8. Yoono sidebar
  9. Yandex toolbar
  10. Lazarus form data recovery
  11. DownThemAll! downloader
  12. Xmarks Sync bookmarking
  13. Various extensions using Brand Thunder themes code
  14. Ant Video Downloader
  15. TRUSTe Tracker Protection
  16. a DB which might be associated with PrivacySuite and Targeted Advertising Cookie Opt-Out
  17. GMail Checker
  18. Brief RSS reading
  19. Screenshot Pimp + MediaPimp + Notifications […] for Facebook
  20. Form History Control

* The top offender is actually SQL executed against “cache.sqlite”. This is the name of the DB used by the Lightning calendaring addon for Thunderbird. Its inclusion in the Firefox SQL list might be a bug in the dashboard or it might be an intentionally misleading name for a spyware DB. There were other highly ranked DBs I could not identify: addonstat.sqlite, store-pp.sqlite, livemargins_appcenter.sqlite, database.sqlite.

We will need a joint effort between Mozilla developers and add-on authors to make Firefox snappier for users. Our next step is to reach out to the authors of the above addons and ask them to change how they use SQLite DBs. We’ll also need to improve our documentation on best-practices for extension developers.

UPDATE: A few commenters pointed out that I never mentioned the async SQL APIs in my post. Indeed, developers should be using the asynchronous APIs to execute SQL statements off the main thread:

28 Responses to Add-on performance problems

  1. “We will need a joint effort between Mozilla developers and add-on authors to make Firefox snappier for users.” Be prepared for a long, slow slog…

  2. Any chance of resurrecting the “one process per jetpack” model for add-ons? Perhaps with a multithreaded/async approach?

  3. Wasn’t there a discussions some time ago about using LevelDB instead of SQLite? Or we wait for SQLite 4 to fix this?

  4. And would there be one day we have Off Main Thread DB, Off Main Thread Add on etc?

    • Vladan Djeric

      We already have APIs for doing DB work off the main thread, but moving addons to a different thread is going to be a lot harder and might break existing addons

  5. Why not make “no main thread file/db-access” (i.e. only async access) and requirement for addon approval? Could be enforced at runtime by letting sync access fail.

    • Vladan Djeric

      That’s probably feasible for JetPack, but it would break a lot of addons that use XPCOM interfaces. Sadly, it would break a lot of Mozilla code too πŸ™

      • But if it is the right thing to do, then we should get on that path at least. Making it a “requirement for addon approval” wouldn’t “break a lot of addons”. If an async-api that is easy to use is offered, it might just delay approval a little, while the author updates their addon. Getting an idea of how much and which Mozilla code is effected, would also be valuable.

        • Vladan Djeric

          Code has to be re-factored to use async APIs and there’s no way around it. As we’ve seen with Mozilla code, it’s not trivial to make this change even with async-APIs that are easy to use.
          However, we should look into making this a requirement for new add-ons

  6. What other thread would you do database work on?

    The only available option to add-ons is chrome workers, and access to XPCOM was removed in Firefox 8.

    There is no threading.

    • Vladan Djeric

      As you and Ben Bucksch pointed out further below, there are APIs for executing SQL off the main thread. Each DB connection has a helper thread just for this

  7. Is there a better view of that histogram?

    The name of the database is hidden after the ellipses in most cases and I’m not sure how you tied it to add-ons?

    • Vladan Djeric

      If you mouse-over the SQL, the tooltip will show more of the string. It’s not the most polished UI

    • Vladan Djeric

      .. and I tied DB names to specific addons by searching AMO extension code for the DB names. For the non-AMO addons, I had to use Google. I had to make do with the data & tools at my disposal, so it’s possible I might have gotten an association wrong.

  8. One more comment.

    So to be clear here, what you are saying is that:

    “Add-on developers should use the async APIs because the async APIs are multithreaded.”

    But there’s not a single use of the word “async” in your post.

    Please change your post to make it clear what add-on developers need to do.

  9. Just tell people to use the async API. Might be obvious to you, but not everybody.
    The API is quite arkward, I made myself a wrapper.

    Seems like the async API uses a new thread, as I would have expected:

  10. DownThemAll! is moving away from main thread DB operations, and main thread I/O in general, well, mostly at least.
    Rapid release and other “challenges” just keep delaying DownThemAll! 3.0, and I’m not too confident on porting back those changes to our 2.0.x branch.

    If you like, you can test our Nightlies and see how they do.

  11. livemargins_appcenter.sqlite is from one of the extensions distributed with Fx by Beijing office to users in China mainland. We also use a lot of json files in profile folder for customized data storage, which may also lead to performance issues, but not apparent from telemetry data.
    I’m not quite familiar with these problems, but we are definitely considering about improving our code quality in the coming year. It would be great if you can give us some pointers. Thanks!

  12. Seems like you guys need to deprecate any method of developing addons which will have a bad impact on performance in any way. Not necessarily make it harder to develop for firebox but make it impossible to have a bad performance impact.

    Can you please tell us about generational garbage collector bug and if there are any timeframe to land it.

  13. Is there any data about the performance of individual statements?

    There are some statements that most folks just do a executeSimpleSQL for, like maybe DROPing a table.

    Does everything need to be async? Or are some things fast enough that doesn’t matter?

  14. Michael Kaply, everything needs to be async. Even trivial amounts of IO can take a long time while disk is blocked doing other stuff in background.

    The bigger problem with occasional sync statements is that they will wait for other async statements to complete before executing. Blocking main thread by async operations followed by a sync one in this way is a very common bug πŸ™