"Take It With You" Wiki
Although this blog has been silent for a while, I haven't been idle. I was working on AJAX-based web application with transparent support for disconnected operations.
TiwyWiki is a prototype wiki that runs both online and offline without any install (besides Flash Player).
Here's a demonstration scenario:
- load the demo (requires Flash 8) and browse a couple of pages,
- pull the network plug off your computer and put your browser in Offline mode,
- re-open the wiki using the same url,
- while offline, continue reading and editing the cached pages of the wiki, create new pages,
- go back online and sync your updates back to the server.
I've only tested TiwyWiki on IE and Firefox on Windows, and heard that it runs properly on Mac (Safari I think), let me know if it runs for you on any other platform.
This is just the skeleton of a wiki, but it gives a feeling of the possibilities of web applications that can deal gracefully with being intermittantly disconnected. I'm especially interested in hearing back about whether this approach is valuable to you, in comparison to the traditional web and rich client models.
What other applications you'd find most appealing and why?
Here are the ones I brainstormed so far: a personal wiki, various other personal or group GTD tools (such as todo list or calendar), a community wiki, an email reader and/or composer, a blog editor, an RSS reader, an app for driving directions.
Some background:Two problems ran in circles in my head while I was on vacation a couple of weeks ago: how to make cross-domain XMLHttp requests before cross-domain is actually supported by browsers and how to allow web applications to run offline?
I started by focusing on the first one, probably because I've been toying recently with cross-domain XMLHttp and client-side storage through Greasemonkey. Also, I was thinking that it would help for running local/offline copies of web apps.
As I learnt more about Flash 8 and its security model, my original plan of running a local copy of a web page for offline use didn't seem convenient enough: you would have to explicitly save the app locally and synch it before going offline.
When I found out that Flash Player did cache Flash apps properly, the idea of running both the online and offline scenarios took the lead. This avoided the new security restrictions for local apps in Flash 8, keeping two local caches of the data (one for the online domain and one for the local copy) and no installation problem.
Instead you would be able to use the app locally as soon as you used it online. First, whatever content you had already accessed would be cached and persisted locally (in the Flash app/storage). You could use pre-fetching to ensure your local cache would have the data that you want.
Second, the Flash app would act as a buffer for disconnected operations, such as local updates while running offline.
Design philosophy:One interesting thing to realize is how and why the pieces fit together.
Once you have a web application that is entirely cacheable, you can support offline operations. You just need to have all the dynamic data go through a smart proxy that can do disconnected reads and updates.
I don't see Flash as the long term solution, but rather a temporary workaround that allows for some early experimentation. Instead of waiting for new browser infrastructures, I wanted to demonstrate that web apps with offline support and no install were already feasible, relying only on a new combination of existing techniques.
In this case, Flash actually turned out to be rather un-obstrusive.
First, if you don't have it installed, the web app will still work fine, except with no offline support or persistent data caching.
Second, Flash offers some benefits that I hadn't anticipated. For example, the storage is shared between IE and Firefox. This makes for a nicer experience that I would expect from any native browser API, such as IE's client storage API or the drafted storage API from the WHAT working group.
For those who want to avoid Flash, other alternative storage techniques could possibly be used to achieve similar results, such as IE's storage API, a Java applet, an ActiveX object or some other kind of browser extension.
In the long run, I hope this proof of concept and the following uses of this technique will help identify the right set of APIs to implement natively in browsers.
Caching:Caching is at the heart of this solution and needs to be configured properly. When the expiration header (Expires, using mod_expires in Apache or directly in IIS) is correctly set for all the static content, both Firefox and IE let you run the application offline without complaining.
Overall, IE appears to be more sensitive to mis-configured caching headers and in that case, it would often display some prompts to work offline or return online to continue the current operation.
Loading Flash when offline:
During a troubleshooting session, I noticed something unexpected. The common markup for including Flash objects in IE actually causes a request to Macromedia, which usually replies with a 302 (but no caching headers).
Besides my surprise of discovering that Macromedia's server is hit every time a Flash app is opened in IE, this meant that the Flash object wouldn't load offline. So TiwyWiki uses its own Flash loading technique (yet another) to support running offline.
Busting the cache:
One downside of forcing the application to be cached is that if a new version of the application becomes available, the browser won't notice it until the current version expires from the cache.
I'm still looking for some ideas on how to let the application deal with this update scenario, so that it could have some logic to check for updates and trick the browser into reloading its cache (force refresh). There may be some solutions by using the XMLHttp API with the right request headers, if the different browser could cache the responses properly.
As a last resort, one could imagine a new browser API that would allow invalidating the cache for a given domain and path.
Locking files in the cache:
The other problem with running the application out of the browser's cache is that the user could "uninstall" the application by accidentally clearing the cache or the application could erased from the cache to make room when the cache gets filled up.
I'm still looking for ideas on how to achieve proper locking of the files in the cache.
In IE, that should be possible using the "Offline Favorites" feature. Whenever you bookmark a page, IE gives you the option to "Make [the favorite] available offline". If you check that option, IE will use a crawler (MSIECrawler) to pre-fetch and cache the content for offline reading. You can hint the crawler using a CDF file, linked from a <link rel="offline" href="file.cdf"/> tag.
But I implemented and ran various experiments with "Offline Favorites", and couldn't get the files to be properly frozen in the cache (they would still get scavenged to make room).
Making a framework:A wiki turned out to be a rather complex application in terms of synchronization and error handling. I originally wanted to write a generic framework for occasionally connected web applications, to deal with these problems.
But besides the reusable Flash component, most of the code so far is specific to the schema and synchronization model for the application. My work on a second application (an RSS reader) hasn't helped me bubble the right abstractions yet.
Also, is there some existing libraries that would offer a rich logical view of a persistent storage that only supports sets of name-value pairs?
I haven't met too many problems with the Flash APIs. ExternalInterface is quite convenient, although I've had to work around a performance issue when passing large data accross.
I wouldn't expect too much performance of the storage API, SharedObject, which serializes objects into files. But this hasn't been a problem so far.
Open problems:Besides the problems already mentioned (building a richer storage abstraction, building a generic synchronization framework, getting more control on the caching), I've hit my head on trying to fix the back button behavior in IE.
The usual hacks rely on iframes pointing to a blank html page on the server, with some unique querystring parameters. Unfortunately, such queries don't work offline, because the unique querystring values essentially keep busting the cache.
- Adam's Bosworth Alchemy project and famous post on modifying information offline,
- TiddlyWiki, a SPA (Single Page Application) wiki, that runs entirely in the browser (no need for a server) and uses a funky "File->Save Page" persistence model,
- The TODO list for TiwyWiki and the internal documentation (explains a bit about the storage structure and different tiers in the APIs).
Update (2009/01/06):Google Gears has been available for a year and a half now, and has been used in a few Google applications and a couple others. Yet there does not seem to be a good online/offline wiki app yet. Disappointing. ______________________________________
Dude, this is the shit. You. Rock.
I've been trying to crack this nut _forever_; it's why I made AMASS. Very cool that you figured it out!
We're brother in arms; I have a a library named the Really Simple History library that solves the back button issues in IE and Firefox: http://www.onjava.com/pub/a/onjava/2005/10/26/ajax-handling-bookmarks-and-back-button.html
I've been bringing my AMASS and Flash work into the Dojo project into the dojo.flash and dojo.storage packages; if you are interested, we could coordinate to create a dojo.offline package that uses both of these to extend the capabilities you've discovered to all apps. Email me at bkn3 [at] columbia -dot- edu if you are interested.Posted by: Brad Neuberg (January 17, 2006 10:53 PM) ______________________________________
Oh, just read the issues about the back button stuff when offline and busting the cache. I'm sure we can find something in IE. Need to study the issue some more. One possibility is to use document.write into the hidden iframe, which might not bust the cache but which would go into history (and you write some state info into the iframe using the document.write), rather than loading a blank.html document.Posted by: Brad Neuberg (January 17, 2006 10:55 PM) ______________________________________
Thanks for your feedback, I'll definitely follow up with you.
TiwyWiki isn't actually using AMASS. Instead it uses yet another Flash object, that also support cross-domain XML requests for the Tiwy RSS reader I've been prototyping.
Some reasons for not using AMASS: I needed something really simple that I would completely understand, I wanted to learn some Flash and also wanted to write the X-domain request feature. Also, as with most Flash framewoks, the HTML inserted by AMASS probably wouldn't let the Flash run offline, as I discovered later.
Congrats on the coverage ;)
Ajaxian is first on my opml, before your blog, and it felt nice to see something about you!
on Safari 1.2.4 is just refeshes repeatedly in a freaky sort of way - but in mac FF works fine.Posted by: redjade (January 25, 2006 02:54 PM) ______________________________________
This is very cool. We've looking for something like this for long time. How far have you reached with the framework. I would happy to test for you.
RamPosted by: Ram (February 18, 2006 07:02 PM) ______________________________________
Unfortunately, I don't have a good solution for making a framework out of this, especially the synchronization part.
I do have some ideas on improving the storage with an abstraction layer, but I'm actually focusing on the server-side storage and versioning problem right now.Posted by: Julien Couvreur (February 20, 2006 10:32 AM) ______________________________________
Great stuff, Julien. Thanks for pinging me on this.Posted by: Jean Sini (February 24, 2006 08:15 AM) ______________________________________
Beware. By depending on Flash 8, you are locking out all Linux users. The newest version of Flash for Linux is still 220.127.116.11, released on Mar 14, 2006.Posted by: Alan (April 30, 2006 01:04 AM) ______________________________________
I understand this limitation. This was more of a proof of concept than a full fledged application.
Also, I'm looking into converting the flash object to support Flash 6 or 7.
It seems that even though I'm not trying to post anything to the sever, it still displays the "connect/work offline" dialoge box.
Could you elaborate on the types of problems you were having?
- ReganPosted by: Regan (June 6, 2006 01:03 PM)