Note: this is a repost from my blog. You can find the original post here. Please head over there for comments.
Table of contents
- Idea
- From past to present, what now?
- What does better mean?
- Why should you care?
- My scenario
- What worked for me
- Proposed features
- Help!
Idea
A library to unify all the different data storage/retrieval/sending/receiving API’s such as XMLHttpRequest, WebSockets, localStorage, IndexedDB, and make it easier to use any number of them at once.
From past to present, what now?
Past: Before, all we had was AJAX requests. Really.
To present: With the new technologies coming up in the HTML5 era, we’ve got localStorage and IndexedDB, WebSockets, node.js, and more. Hectic.
What now? Don’t you wish there was a better way to send and receive data in the browser?
What does better mean?
My general goals for this are:
- Simple key/value store common abstraction.
- Pluggable handlers for each type of send/receive.
- Use other abstractions specified in each handler (library surfaces your API as well).
- Straightforward way to define flow of data. More on this later.
Anything else you wish it could do?
Why should you care?
Short answer: maintenance, scalability, flexibility.
As these technologies become widely supported, you will start seeing a common problem for websites heavily relying on AJAX (or any kind of data transfer without page reloads): how do you take advantage of them without rewriting your entire codebase every time there’s a new technology (API/storage engine/etc) coming out?
My scenario
The whole reason I got thinking about this was because it happened to me. And it was frustrating.
I had this client-side application using jQuery.ajax requests, and I wanted to take advantage of localStorage for some of them, for data that I didn’t need to get from the server on every page load.
I considered:
- Quick’n’dirty: Rewrite these pieces of the application to do both localStorage and ajax requests as fallback.
- Slightly better: A library that’s flexible enough for my purposes.
- Ideal: A library that would allow me to enable/disable localStorage as an intermediary step on a per-request basis, make it easy to add IndexedDB support later, etc.
What worked for me
The simpler thing I went with was a Data object with a couple of functions.
Example usage:
1 // main.js
2 window.data = new DataStore({
3 url: '/fetch_new_data',
4 // show a spinny tangy
5 sync_before: function showSyncInProgress() { ... },
6 // hide the spinny thingy, maybe show a fading notification
7 sync_success: function showSyncDone() { ... },
8 // hide the spinny thingy, definitely show some message
9 sync_error: function showSyncFailed() { ... }
10 }
11
12 // example request
13 var i = 0;
14 window.data.process_request({
15 ajax: {url: '/new_comment', type: 'POST',
16 data: $('#comment-form').serialize()},
17 key: 'comment_' + (i++),
18 value: {'author': $('#comment-form .author').val(),
19 'text': $('#comment-form .text').val()}
20 });
ajax.data
and value
are actually very similar, with an important exception in most applications (e.g. Django): the csrftoken. We don’t need to store that in localStorage for every request. So I chose to keep the two completely separate. You could subclass DataStore and make it save you this extra work per request.
Below is an example implementation (raw file):
1 /* This depends on Crockford's json2.js
2 * from https://github.com/douglascrockford/JSON-js
3 * Options:
4 * - url: function()
5 * - sync_before: function()
6 * - sync_success: function()
7 * - sync_error: function()
8 */
9 function DataStore(options) {
10 window.data = this;
11 this.storage = window.localStorage;
12 // date of last time we synced
13 this.last_sync = null;
14 // queue of requests, populated if offline
15 this.queue = [];
16
17 /**
18 * Gets data stored at `key`; `key` is a string
19 */
20 this.get_data = function (key) {
21 var str_data = this.storage.getItem(key);
22 return JSON.parse(str_data);
23 }
24
25 /**
26 * Sets data at `key`; `key` is a string
27 */
28 this.set_data = function (key, data) {
29 var str_data = JSON.stringify(data);
30 this.storage.setItem(key, str_data);
31 }
32
33 /**
34 * Syncs data between local storage and server, depending on
35 * modifications and online status.
36 */
37 this.sync_data = function () {
38 // must be online to sync
39 if (!this.is_online()) {
40 return false;
41 }
42
43 this.last_sync = this.get_data('last_sync');
44
45 // have we never synced before in this browser?
46 if (!this.last_sync) {
47 // first-time setup
48 // ...
49 this.last_sync = {};
50 this.last_sync.when = new Date().getTime();
51 this.last_sync.is_modified = false;
52 }
53
54 if (this.last_sync.is_modified) {
55 var request_options;
56 // sync modified data
57 // you can pass callbacks here too
58 while (this.queue.length > 0) {
59 request_options = this.queue.pop();
60 $.ajax(request_options.ajax);
61 }
62 this.set_data('queue', []);
63 this.last_sync.is_modified = false;
64 }
65 // data is synced, update sync time
66 this.set_data('last_sync', this.last_sync);
67
68 // get modified data from the server here
69 $.ajax({
70 type: 'POST',
71 url: options.url,
72 dataType: 'json',
73 data: {'last_sync': this.last_sync.sync_date},
74 beforeSend:
75 // here you can show some "sync in progress" icon
76 options.sync_before,
77 error:
78 // an error callback should be passed in to this Data
79 // object and would be called here
80 options.sync_error,
81 success: function (response, textStatus, request) {
82 // callback for success
83 options.sync_success(
84 response, textStatus, request);
85 }
86 });
87
88
89 /**
90 * Process a request. This is where all the magic happens.
91 */
92 this.process_request = function(request_options) {
93 request_options.beforeSend();
94 this.set_data(request_options.key, request_options.value);
95
96 if (this.is_online()) {
97 $.ajax(request_options.ajax);
98 } else {
99 this.queue.push(request_options);
100 this.last_sync.is_modified = true;
101 this.set_data('last_sync', this.last_sync);
102 // there are issues with this, storing functions as
103 // strings is not a good idea :)
104 this.set_data('queue', this.queue);
105 }
106
107 request_options.processed();
108 }
109
110 /**
111 * Return true if online, false otherwise.
112 */
113 this.is_online = function () {
114 if (navigator && navigator.onLine !== undefined) {
115 return navigator.onLine;
116 }
117 try {
118 var request = new XMLHttpRequest();
119 request.open('GET', '/', false);
120 request.send(null);
121 return (request.status === 200);
122 }
123 catch(e) {
124 return false;
125 }
126 }
127 }
Proposed Features
The example API isn’t bad, but I think it could be better. Perhaps something along the lines of Lawnchair. As I’m writing this, I realize that writing an API is going to take longer than I’d like – therefore, this will serve as a teaser and food for thought. Feedback is welcome.
- Add an .each method for iterating over retrieved objects (inspired by Lawnchair)
- Standard
DataStore.save, .get, .remove, etc.
- Support for these “storage engines”: localStorage, IndexedDB, send-to-server.
- Support for these request types: XMLHttpRequest, WebSockets.
- Store, at the very least, primitive values and JSON.
- Include callbacks for various stages in the process of a request, similar to jQuery.ajax, e.g.
beforeSend, complete, success, error
. Figure out a good way to do this at each layer (minimize confusion). - For each request, specify which layers and in what order to go through. For example, if you want to store something in localStorage, IndexedDB, and send it to the server, you could do it in that order or the reverse.
- Control whether to go to the next layer type depending on whether the previous succeeded or failed. Say, if you want to send the request to server but that fails, try localStorage as a fallback. Or the opposite.
- Include a
.get_then_store
shortcut for getting the data from layer A and storing it in layer B? - Extensible: as easy as
DataStore.addLayer(layerName, layerHandler)
, where layerHandler (obviously) implements some common API along with exposing some of its own, if necessary (e.g. ability to query or find, for IndexedDB).
Help!
Hopefully my rant has gotten you thinking about the right approach. What would you like to see? What would make this something you would use and be happy with?
If you are interested in getting involved with coding this, contact me at paulc at mozilla.com.