PSA: DOM Local Storage considered harmful

Recently there have been a number of blog posts on optimizations possible via Local StorageAPI. When Microsoft, Google, Amazon and a number of others aggressively adopt a new feature, people notice.

The optimization is to use Local Storage to reduce network requests and/or payload size. This should result in a more responsive experience for the user… except when it doesn’t.

This strategy can backfire because:

  • Local Storage is a synchronous API
  • Local Storage does IO

Disk IO is particularly problematic because it’s non-deterministic for a multitude of reasons. A simple disk operation can take anywhere from zero milliseconds to a few seconds. Browsers cope with this by preloading the entire Local Storage key/value store into memory on first request. While testing a website the developer is likely to frequently reload the same page multiple times. This means that the relevant disk IO is very likely to be cached in the os file cache.

Now consider the case of a user turning on their computer, firing up the browser and going to an optimized site:

  1. The webpage starts to load
  2. Data is read in from Local Storage
  3. Webpage completely freezes for a few seconds while the browser populates Local Storage key/value store.

This freeze can last anywhere from a few milliseconds on an unburdened computer to dozens of seconds on a computer with stock Windows settings (AntiVirus, Windows Indexing Service, etc). Firefox, Chrome, Safari suffer this fate to various degrees.

Local Storage has the following costs:

  1. It can take a long unresponsive while to read in maximum allowable 5megs of data
  2. This data is then in memory for the lifetime of the webpage session wasting memory. Imagine if every one of your tabs decided to do this.
  3. LS is persistent. My profile has 0.5MB of meebo.com data that will haunt me forever even though meebo is long gone from my history.

What should webdevs do instead?

Cry, rely on browser cache. There are no viable alternative at the moment. IndexedDB is complex, requires user to approve it and isn’t widely implemented. WebSQL is all about bringing SQLite problems we’ve been studying and fixing within Firefox to every single webpage.

My best guess is that we’ll end up with webdevs using convenience libraries built on top of IndexedDB. We will likely add promptless operation to IndexedDB.

We have already made Local Storage hurt less, we have a plan to make it relatively painless, but it will always degrade user experience when compared to something like IndexedDB.

Are there any other convenient APIs with terrible complications?

Of course, see sync XHR.

 

25 comments

  1. Can’t we add an async localStorage API?

  2. We have. It’s called IndexedDB.

  3. No, I mean something exactly like localStorage except async. So site maintainers don’t have to change anything else at the same time.

  4. localStorage solves other problems though, like offline mode for webapps, and can’t Ctrl+Shift+Del to clear the localStorage, so items are more reliably available.

    As always with programming, you pick the tools that best fit your needs.

  5. Don’t forget about the offline appcache for people who are just stuffing assets into localStorage- sadly, these people exist, but their solution is already at hand!

  6. Can’t you partition it? E.g. split into separate files for each site/domain, or even each key/value pair?

    Always wondered why Firefox used DBMS (SQLite) to store data where a bunch of text files would be enough – that feels so overengineered… Why not use filesystem as database and leave caching to the OS?

  7. Nice post!
    Zack, asynchronous means maintainers have to make every logic based on localStorage asynchronous.
    WebSQL or IndexedDB are fine, the abstract interface to recreate a simple key/value asynchronous storage could be implemented already with many fallbacks but the IndexedDB prompt is too damn annoying.
    Imagine you have to confirm that for every bloody page you visit …

  8. IIRC, IE8 Beta had an asynchronous localStorage implementation.

  9. Well yes, either add asynchronous operation to existing localStorage or add promptless operation to IndexedDB. Having to choose between a sync component or a complex one is a bit silly when things could just be easier. And maybe getting async operation in localStorage would be quicker than expecting IndexedDB to be adopted more broadly?

    @Zack site maintainers would still have to change something, as async API’s are always different (as in, current methods would have to take callbacks). But the change would be minimal and much less painful than using something entirely different as IndexedDB.

  10. Isn’t it incredibly ironic that I’ve been criticised so many times for commenting on posts to hacks.mozilla covering ‘how to’ use features that are incomplete and not supported across the board. Then just this week another post. The topic? Surprise, surprise … Local Storage! Of course as is usual for hacks.mozila, the article describes how to use Local Storage as if it’s ready now. LOL. Then you come along Taras – Mr Snappy – and say Local Storage is not a good solution! LOL

    If I didn’t laugh I’d cry.

    Does anybody at Mozilla actually have any idea what current web development problems exist and are there any websites I can read that have Mozilla experts solving common developer problems that exist right now? Why should that sort of website be limited to A List Apart and the like?

    SQLite, Local Storage, IndexedDB, WebSQL … How many attempts does it take, and how many ruddy years, for the web to get what Flash has had for ages? There really is nobody at the steering wheel or in the navigator’s seat when it comes to ensuring that the direction of web features has anything to do with the present day reality of web development, is there? No wonder people are adopting native app frameworks on mobile.

  11. Anyone know if cookies are kept in memory vs. IO?

    That makes me wonder if the browser could use some idle IO time to lazy load localStorage from the disk to memory when visiting a website. Then when needed, at least in some cases it will already be in memory. GC removes when no longer needed.

  12. @RCL. I ask that question every day. SQLite provides a flexible way to robustly persist mixed data to disk. On one hand it is easy to cause it be slow, on the other hand handwritten data storage options are often quite good at loosing data.

  13. @Robert, cookies are complicated in that they are kept ondisk, but the file is basically in memory the whole time.

  14. Thanks for the pointer to app cache. Looks like it also requires prompting the user :(

  15. @Aaron, what does ctrl+shift+del do?

  16. @pd. I actually agree with you that it’s frustrating and that useful features are available elsewhere. Difference is that in a single-vendor native app framework you can iterate very quickly. And if your design happens to be crap, it can get scrapped/revised in the next version.

    Once you add a crap feature to the web… it sticks around. There is no good way to phase out bad ideas on the web. As a result it takes longer to agree/implement APIs.

    Regarding flash being awesome…I bet people who invested into flash were real happy to see adobe pull out of linux, mobile.

  17. Um, run a web worker whose sole job is to do local storage? Async behaviour, but of course you’d have to do everything behind postMessages.

  18. +1 with what Robert is saying. Why can’t webdev open a web worker (or iframe?) dedicated to local storage? The main thread could then be dedicated to the rest.

  19. Nicolas Chambrier

    Sounds like bullshit to me: are you really telling us we shouldn’t store a few KB of data in localStorage because it will take forever to load this poor data in memory from disk ?
    1. Please, benchmarks, cause it sounds very dumb to me. Loading cookie from disk will be as slow, retrieving page and js and any other blocking asset from cache will be slower, and retrieving it from network will be like years… This localStorage thing appears to be absolutely neglictible imho.
    2. If benchmarks prove you’ré right, let’s use workers. You’d better evangelize about this alternative instead of just whining about a massively loved technology.

  20. FIY. I’m not responding to any of the “use a worker” suggestions because they do not make sense. It’s called DOM Local Storage…no DOM in workers.

    The problem with LS isn’t that it’s slow, it’s the fact that it causes significant stalls in the browser(by poor design). I could live with slow, but slow + blocking due to io -> bad

  21. I disagree with the blocking io part, it’s not like i will put my whole application in the localStorage, but it also depends on the case, as for some devs it could be better to leave the user with some sync disc io than to abuse of the server/client network connection, as said, it depends on the case, not that it’s all bad.

  22. “FIY. I’m not responding to any of the “use a worker” suggestions because they do not make sense. It’s called DOM Local Storage…no DOM in workers.”
    => And I guess you always transfer XML when you use XMLHttpRequest?
    “DOM local storage” is a bad name for an API used to store persistant data. The spec http://www.w3.org/TR/webstorage/ has been renamed to “WebStorage” (we probably update the doc as well). Not sure it’s any better, but at least the “DOM” part has been removed.
    Anyway, nothing in the API of the local storage is related to the DOM. The only thing it shares with the DOM is the DOMString interface and the notion of origin.

    I haven’t tried to use the local storage API in a worker, but there is no reason why it couldn’t work. According to the spec, workers have an origin.

    I buy the argument that the API is designed to be blocking, but I don’t buy that you can’t use them in workers.

  23. http://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker

    We could expose this API to workers, but then we run into an ethical question of bringing crappy APIs to new environments(and it would still not be cross-browser).

  24. “We could expose this API to workers, but then we run into an ethical question of bringing crappy APIs to new environments”
    => Which, I admit, is a valid concern. I suggested this to work around main thread blocking. It was more an idea to do “better”, not really what I’d consider to be a “good” idea ;-)

    It would probably be an easy-to-implement & easy-to-ship solution if the “LS is blocking the main thread” is such a big problem.

    “(and it would still not be cross-browser).”
    => This is a non-problem. First, LS itself isn’t cross-browser. Moreover, support of LS in web worker can be feature-detected and main-thread-LS can still be used as fallback if available.