DataStore – an abstraction layer for storing data and processing requests

Note: this is a repost from my blog. You can find the original post here. Please head over there for comments.

Table of contents

  1. Idea
  2. From past to present, what now?
  3. What does better mean?
  4. Why should you care?
  5. My scenario
  6. What worked for me
  7. Proposed features
  8. Help!

Idea

A library to unify all the different data storage/retrieval/sending/receiving API’s such as XMLHttpRequest, WebSockets, localStorage, IndexedDB, and make it easier to use any number of them at once.

From past to present, what now?

Past: Before, all we had was AJAX requests. Really.

To present: With the new technologies coming up in the HTML5 era, we’ve got localStorage and IndexedDB, WebSockets, node.js, and more. Hectic.

What now? Don’t you wish there was a better way to send and receive data in the browser?

What does better mean?

My general goals for this are:

  1. Simple key/value store common abstraction.
  2. Pluggable handlers for each type of send/receive.
  3. Use other abstractions specified in each handler (library surfaces your API as well).
  4. Straightforward way to define flow of data. More on this later.

Anything else you wish it could do?

Why should you care?

Short answer: maintenance, scalability, flexibility.

As these technologies become widely supported, you will start seeing a common problem for websites heavily relying on AJAX (or any kind of data transfer without page reloads): how do you take advantage of them without rewriting your entire codebase every time there’s a new technology (API/storage engine/etc) coming out?

My scenario

The whole reason I got thinking about this was because it happened to me. And it was frustrating.

I had this client-side application using jQuery.ajax requests, and I wanted to take advantage of localStorage for some of them, for data that I didn’t need to get from the server on every page load.

I considered:

  • Quick’n’dirty: Rewrite these pieces of the application to do both localStorage and ajax requests as fallback.
  • Slightly better: A library that’s flexible enough for my purposes.
  • Ideal: A library that would allow me to enable/disable localStorage as an intermediary step on a per-request basis, make it easy to add IndexedDB support later, etc.

What worked for me

The simpler thing I went with was a Data object with a couple of functions.

Example usage:

 1 // main.js
 2 window.data = new DataStore({
 3     url: '/fetch_new_data',
 4     // show a spinny tangy
 5     sync_before: function showSyncInProgress() { ... },
 6     // hide the spinny thingy, maybe show a fading notification
 7     sync_success: function showSyncDone() { ... },
 8     // hide the spinny thingy, definitely show some message
 9     sync_error: function showSyncFailed() { ... }
10 }
11 
12 // example request
13 var i = 0;
14 window.data.process_request({
15     ajax: {url: '/new_comment', type: 'POST',
16            data: $('#comment-form').serialize()},
17     key: 'comment_' + (i++),
18     value: {'author': $('#comment-form .author').val(),
19             'text': $('#comment-form .text').val()}
20 });

ajax.data and value are actually very similar, with an important exception in most applications (e.g. Django): the csrftoken. We don’t need to store that in localStorage for every request. So I chose to keep the two completely separate. You could subclass DataStore and make it save you this extra work per request.

Below is an example implementation (raw file):

  1 /* This depends on Crockford's json2.js
  2  * from https://github.com/douglascrockford/JSON-js
  3  * Options:
  4  *     - url: function()
  5  *     - sync_before: function()
  6  *     - sync_success: function()
  7  *     - sync_error: function()
  8  */
  9 function DataStore(options) {
 10     window.data = this;
 11     this.storage = window.localStorage;
 12     // date of last time we synced
 13     this.last_sync = null;
 14     // queue of requests, populated if offline
 15     this.queue = [];
 16 
 17     /**
 18      * Gets data stored at `key`; `key` is a string
 19      */
 20     this.get_data = function (key) {
 21         var str_data = this.storage.getItem(key);
 22         return JSON.parse(str_data);
 23     }
 24 
 25     /**
 26      * Sets data at `key`; `key` is a string
 27      */
 28     this.set_data = function (key, data) {
 29         var str_data = JSON.stringify(data);
 30         this.storage.setItem(key, str_data);
 31     }
 32 
 33     /**
 34      * Syncs data between local storage and server, depending on
 35      * modifications and online status.
 36      */
 37     this.sync_data = function () {
 38         // must be online to sync
 39         if (!this.is_online()) {
 40             return false;
 41         }
 42 
 43         this.last_sync = this.get_data('last_sync');
 44 
 45         // have we never synced before in this browser?
 46         if (!this.last_sync) {
 47             // first-time setup
 48             // ...
 49             this.last_sync = {};
 50             this.last_sync.when = new Date().getTime();
 51             this.last_sync.is_modified = false;
 52         }
 53 
 54         if (this.last_sync.is_modified) {
 55             var request_options;
 56             // sync modified data
 57             // you can pass callbacks here too
 58             while (this.queue.length > 0) {
 59                 request_options = this.queue.pop();
 60                 $.ajax(request_options.ajax);
 61             }
 62             this.set_data('queue', []);
 63             this.last_sync.is_modified = false;
 64         }
 65         // data is synced, update sync time
 66         this.set_data('last_sync', this.last_sync);
 67 
 68         // get modified data from the server here
 69        $.ajax({
 70             type: 'POST',
 71             url: options.url,
 72             dataType: 'json',
 73             data: {'last_sync': this.last_sync.sync_date},
 74             beforeSend:
 75                 // here you can show some "sync in progress" icon
 76                 options.sync_before,
 77             error:
 78                 // an error callback should be passed in to this Data
 79                 // object and would be called here
 80                 options.sync_error,
 81             success: function (response, textStatus, request) {
 82                 // callback for success
 83                 options.sync_success(
 84                     response, textStatus, request);
 85             }
 86         });
 87 
 88 
 89     /**
 90      * Process a request. This is where all the magic happens.
 91      */
 92     this.process_request = function(request_options) {
 93         request_options.beforeSend();
 94         this.set_data(request_options.key, request_options.value);
 95 
 96         if (this.is_online()) {
 97             $.ajax(request_options.ajax);
 98         } else {
 99             this.queue.push(request_options);
100             this.last_sync.is_modified = true;
101             this.set_data('last_sync', this.last_sync);
102             // there are issues with this, storing functions as
103             // strings is not a good idea :)
104             this.set_data('queue', this.queue);
105         }
106 
107         request_options.processed();
108     }
109 
110     /**
111      * Return true if online, false otherwise.
112      */
113     this.is_online = function () {
114         if (navigator && navigator.onLine !== undefined) {
115             return navigator.onLine;
116         }
117         try {
118             var request = new XMLHttpRequest();
119             request.open('GET', '/', false);
120             request.send(null);
121             return (request.status === 200);
122         }
123         catch(e) {
124             return false;
125         }
126     }
127 }

Proposed Features

The example API isn’t bad, but I think it could be better. Perhaps something along the lines of Lawnchair. As I’m writing this, I realize that writing an API is going to take longer than I’d like – therefore, this will serve as a teaser and food for thought. Feedback is welcome.

  • Add an .each method for iterating over retrieved objects (inspired by Lawnchair)
  • Standard DataStore.save, .get, .remove, etc.
  • Support for these “storage engines”: localStorage, IndexedDB, send-to-server.
  • Support for these request types: XMLHttpRequest, WebSockets.
  • Store, at the very least, primitive values and JSON.
  • Include callbacks for various stages in the process of a request, similar to jQuery.ajax, e.g. beforeSend, complete, success, error. Figure out a good way to do this at each layer (minimize confusion).
  • For each request, specify which layers and in what order to go through. For example, if you want to store something in localStorage, IndexedDB, and send it to the server, you could do it in that order or the reverse.
  • Control whether to go to the next layer type depending on whether the previous succeeded or failed. Say, if you want to send the request to server but that fails, try localStorage as a fallback. Or the opposite.
  • Include a .get_then_store shortcut for getting the data from layer A and storing it in layer B?
  • Extensible: as easy as DataStore.addLayer(layerName, layerHandler), where layerHandler (obviously) implements some common API along with exposing some of its own, if necessary (e.g. ability to query or find, for IndexedDB).

Help!

Hopefully my rant has gotten you thinking about the right approach. What would you like to see? What would make this something you would use and be happy with?

If you are interested in getting involved with coding this, contact me at paulc at mozilla.com.