"Take It With You" Wiki

Although this blog has been silent for a while, I haven't been idle. I was working on AJAX-based web application with transparent support for disconnected operations.

TiwyWiki is a prototype wiki that runs both online and offline without any install (besides Flash Player).
Here's a demonstration scenario:

  1. load the demo (requires Flash 8) and browse a couple of pages,
  2. pull the network plug off your computer and put your browser in Offline mode,
  3. re-open the wiki using the same url,
  4. while offline, continue reading and editing the cached pages of the wiki, create new pages,
  5. go back online and sync your updates back to the server.

I've only tested TiwyWiki on IE and Firefox on Windows, and heard that it runs properly on Mac (Safari I think), let me know if it runs for you on any other platform.

This is just the skeleton of a wiki, but it gives a feeling of the possibilities of web applications that can deal gracefully with being intermittantly disconnected. I'm especially interested in hearing back about whether this approach is valuable to you, in comparison to the traditional web and rich client models.
What other applications you'd find most appealing and why?
Here are the ones I brainstormed so far: a personal wiki, various other personal or group GTD tools (such as todo list or calendar), a community wiki, an email reader and/or composer, a blog editor, an RSS reader, an app for driving directions.


Some background:

Two problems ran in circles in my head while I was on vacation a couple of weeks ago: how to make cross-domain XMLHttp requests before cross-domain is actually supported by browsers and how to allow web applications to run offline?
I started by focusing on the first one, probably because I've been toying recently with cross-domain XMLHttp and client-side storage through Greasemonkey. Also, I was thinking that it would help for running local/offline copies of web apps.

The problem with using Greasemonkey to extend the browser is that it's not widely available and it doesn't offer good control over cross-domain requests. A Flash and Javascript combination, such as the Flash-based Canvas or AMASS storage, seemed like a better solution.
As I learnt more about Flash 8 and its security model, my original plan of running a local copy of a web page for offline use didn't seem convenient enough: you would have to explicitly save the app locally and synch it before going offline.

When I found out that Flash Player did cache Flash apps properly, the idea of running both the online and offline scenarios took the lead. This avoided the new security restrictions for local apps in Flash 8, keeping two local caches of the data (one for the online domain and one for the local copy) and no installation problem.
Instead you would be able to use the app locally as soon as you used it online. First, whatever content you had already accessed would be cached and persisted locally (in the Flash app/storage). You could use pre-fetching to ensure your local cache would have the data that you want.
Second, the Flash app would act as a buffer for disconnected operations, such as local updates while running offline.


Design philosophy:

One interesting thing to realize is how and why the pieces fit together.
As a starting point, you should understand that the AJAX trend is not simply about rich UI and eye candy, but more generally about providing a more responsive experience by optimizing the bottleneck resource (the network): you cache the data that doesn't change (some HTML, Javascript or CSS), and transfer only the information that is dynamic.
Once you have a web application that is entirely cacheable, you can support offline operations. You just need to have all the dynamic data go through a smart proxy that can do disconnected reads and updates.
That's where Flash comes into play, as it offers large persistent local storage and easy interfacing with Javascript.

I don't see Flash as the long term solution, but rather a temporary workaround that allows for some early experimentation. Instead of waiting for new browser infrastructures, I wanted to demonstrate that web apps with offline support and no install were already feasible, relying only on a new combination of existing techniques.

That's why I tried to keep the extensions to the browser as clean and simple as possible, minimizing the amount of Flash and relying more on the common skillset (HTML+Javascript). I think this will motivate other developpers to try this approach.

In this case, Flash actually turned out to be rather un-obstrusive.
First, if you don't have it installed, the web app will still work fine, except with no offline support or persistent data caching.
Second, Flash offers some benefits that I hadn't anticipated. For example, the storage is shared between IE and Firefox. This makes for a nicer experience that I would expect from any native browser API, such as IE's client storage API or the drafted storage API from the WHAT working group.

For those who want to avoid Flash, other alternative storage techniques could possibly be used to achieve similar results, such as IE's storage API, a Java applet, an ActiveX object or some other kind of browser extension.

In the long run, I hope this proof of concept and the following uses of this technique will help identify the right set of APIs to implement natively in browsers.

Caching:

Caching is at the heart of this solution and needs to be configured properly. When the expiration header (Expires, using mod_expires in Apache or directly in IIS) is correctly set for all the static content, both Firefox and IE let you run the application offline without complaining.
Overall, IE appears to be more sensitive to mis-configured caching headers and in that case, it would often display some prompts to work offline or return online to continue the current operation.

Loading Flash when offline:
During a troubleshooting session, I noticed something unexpected. The common markup for including Flash objects in IE actually causes a request to Macromedia, which usually replies with a 302 (but no caching headers).
Besides my surprise of discovering that Macromedia's server is hit every time a Flash app is opened in IE, this meant that the Flash object wouldn't load offline. So TiwyWiki uses its own Flash loading technique (yet another) to support running offline.


Busting the cache:
One downside of forcing the application to be cached is that if a new version of the application becomes available, the browser won't notice it until the current version expires from the cache.

I'm still looking for some ideas on how to let the application deal with this update scenario, so that it could have some logic to check for updates and trick the browser into reloading its cache (force refresh). There may be some solutions by using the XMLHttp API with the right request headers, if the different browser could cache the responses properly.
As a last resort, one could imagine a new browser API that would allow invalidating the cache for a given domain and path.

Locking files in the cache:
The other problem with running the application out of the browser's cache is that the user could "uninstall" the application by accidentally clearing the cache or the application could erased from the cache to make room when the cache gets filled up.
I'm still looking for ideas on how to achieve proper locking of the files in the cache.

In IE, that should be possible using the "Offline Favorites" feature. Whenever you bookmark a page, IE gives you the option to "Make [the favorite] available offline". If you check that option, IE will use a crawler (MSIECrawler) to pre-fetch and cache the content for offline reading. You can hint the crawler using a CDF file, linked from a <link rel="offline" href="file.cdf"/> tag.
But I implemented and ran various experiments with "Offline Favorites", and couldn't get the files to be properly frozen in the cache (they would still get scavenged to make room).


Making a framework:

A wiki turned out to be a rather complex application in terms of synchronization and error handling. I originally wanted to write a generic framework for occasionally connected web applications, to deal with these problems.
But besides the reusable Flash component, most of the code so far is specific to the schema and synchronization model for the application. My work on a second application (an RSS reader) hasn't helped me bubble the right abstractions yet.

Do you know any generic synchronization framework which could be ported or mimicked in Javascript? Something like TrimQuery would be great if it supported INSERT and UPDATE.

Also, is there some existing libraries that would offer a rich logical view of a persistent storage that only supports sets of name-value pairs?

Developing with Flash:

This was my first time working with Flash and overall I found it easier than expected. ActionScript is a sibling of Javascript (both follow the ECMAScript specification), which made it easy to pick up. I was happy to interact with Flash authoring tools as little as possible and end up building the Flash component completely using the MTASC compiler.
I haven't met too many problems with the Flash APIs. ExternalInterface is quite convenient, although I've had to work around a performance issue when passing large data accross.
I wouldn't expect too much performance of the storage API, SharedObject, which serializes objects into files. But this hasn't been a problem so far.

Open problems:

Besides the problems already mentioned (building a richer storage abstraction, building a generic synchronization framework, getting more control on the caching), I've hit my head on trying to fix the back button behavior in IE.
The usual hacks rely on iframes pointing to a blank html page on the server, with some unique querystring parameters. Unfortunately, such queries don't work offline, because the unique querystring values essentially keep busting the cache.

I've also encountered some weird issues with Flash in Firefox 1.5, showing "Bad NPObject as private data" in the Javascript console and sometimes popping up warnings that an extension mis-behaved. My guess at this point is that it was some interaction between Flash and some other extension, possibly AdBlock.

And finally, I'm still battling some memory leaks issue. Although the code does use closures quite a bit, I can't see how it would create circular references chains between the DOM and the Javascript engine.


Related:


Update (2009/01/06):

Google Gears has been available for a year and a half now, and has been used in a few Google applications and a couple others. Yet there does not seem to be a good online/offline wiki app yet. Disappointing.

Posted by Julien on January 17, 2006. Permalink
Comments
comments powered by Disqus

Dude, this is the shit. You. Rock.

I've been trying to crack this nut _forever_; it's why I made AMASS. Very cool that you figured it out!

We're brother in arms; I have a a library named the Really Simple History library that solves the back button issues in IE and Firefox: http://www.onjava.com/pub/a/onjava/2005/10/26/ajax-handling-bookmarks-and-back-button.html

I've been bringing my AMASS and Flash work into the Dojo project into the dojo.flash and dojo.storage packages; if you are interested, we could coordinate to create a dojo.offline package that uses both of these to extend the capabilities you've discovered to all apps. Email me at bkn3 [at] columbia -dot- edu if you are interested.

Posted by: Brad Neuberg at January 17, 2006 10:53 PM

Oh, just read the issues about the back button stuff when offline and busting the cache. I'm sure we can find something in IE. Need to study the issue some more. One possibility is to use document.write into the hidden iframe, which might not bust the cache but which would go into history (and you write some state info into the iframe using the document.write), rather than loading a blank.html document.

Posted by: Brad Neuberg at January 17, 2006 10:55 PM

Hi Brad,
Thanks for your feedback, I'll definitely follow up with you.
TiwyWiki isn't actually using AMASS. Instead it uses yet another Flash object, that also support cross-domain XML requests for the Tiwy RSS reader I've been prototyping.
Some reasons for not using AMASS: I needed something really simple that I would completely understand, I wanted to learn some Flash and also wanted to write the X-domain request feature. Also, as with most Flash framewoks, the HTML inserted by AMASS probably wouldn't let the Flash run offline, as I discovered later.

Posted by: Julien Couvreur at January 18, 2006 09:05 AM

Congrats on the coverage ;)
Ajaxian is first on my opml, before your blog, and it felt nice to see something about you!

Posted by: Regis at January 20, 2006 06:20 PM

on Safari 1.2.4 is just refeshes repeatedly in a freaky sort of way - but in mac FF works fine.

Posted by: redjade at January 25, 2006 02:54 PM

Julien,

This is very cool. We've looking for something like this for long time. How far have you reached with the framework. I would happy to test for you.

Ram

Posted by: Ram at February 18, 2006 07:02 PM

Ram,
Unfortunately, I don't have a good solution for making a framework out of this, especially the synchronization part.

I do have some ideas on improving the storage with an abstraction layer, but I'm actually focusing on the server-side storage and versioning problem right now.

Posted by: Julien Couvreur at February 20, 2006 10:32 AM

Great stuff, Julien. Thanks for pinging me on this.

Posted by: Jean Sini at February 24, 2006 08:15 AM

Beware. By depending on Flash 8, you are locking out all Linux users. The newest version of Flash for Linux is still 7.0.63.0, released on Mar 14, 2006.

Posted by: Alan at April 30, 2006 01:04 AM

Alan,
I understand this limitation. This was more of a proof of concept than a full fledged application.
Also, I'm looking into converting the flash object to support Flash 6 or 7.

Posted by: Julien Couvreur at May 1, 2006 09:00 AM

Julien,

I'm playing with an javascript/ajax app myself and trying to get it to work offline. It really seems like it should work, however I'm having some of the problems in the IE caching area that you mentioned.

It seems that even though I'm not trying to post anything to the sever, it still displays the "connect/work offline" dialoge box.

Could you elaborate on the types of problems you were having?

- Regan

Posted by: Regan at June 6, 2006 01:03 PM
Trackbacks
Occasionally connected Wikis
Excerpt: Julien Couvreur has been working on something that sounds very, very good. "...I haven't been idle....
Weblog: Alex Barnett blog
Tracked: January 18, 2006 11:33 PM
Curiosity is bliss: "Take It With You" Wiki
Excerpt: This would be something that I will be checking out. The online prototype gives you the glimpse of what is the possibility of synching your between your desktop wiki and online wiki space. Having both an online version and being to work when off the gr...
Weblog: Educational Weblogs
Tracked: January 22, 2006 08:23 AM
Ajax离线存储技术–初探
Excerpt: 你可曾想过原来不仅可以离线阅读feeds,还可以根本不需要客户端软件也能离线写Blog,上线后自动同步?
Weblog: Devin
Tracked: January 29, 2006 11:36 AM
Solving the "Offline Problem"
Excerpt: Simon points to Berlind who examines Sun's JavaDB (AKA Apache Derby, AKA IBM Cloudscape) as a potential option for solving what is (un)affectionately known as the "offline problem" amongst thin client advocates such as yours truly. The very same proble...
Weblog: tecosystems
Tracked: May 1, 2006 07:44 PM
Solving the "Offline Problem"
Excerpt: Simon points to Berlind who examines Sun's JavaDB (AKA Apache Derby, AKA IBM Cloudscape) as a potential option for solving what is (un)affectionately known as the "offline problem" amongst thin client advocates such as yours truly. The very same proble...
Weblog: tecosystems
Tracked: May 2, 2006 08:30 AM