"Take It With You" Wiki

Although this blog has been silent for a while, I haven't been idle. I was working on AJAX-based web application with transparent support for disconnected operations.

TiwyWiki is a prototype wiki that runs both online and offline without any install (besides Flash Player).
Here's a demonstration scenario:

1. load the demo (requires Flash 8) and browse a couple of pages,
2. pull the network plug off your computer and put your browser in Offline mode,
3. re-open the wiki using the same url,
4. while offline, continue reading and editing the cached pages of the wiki, create new pages,
5. go back online and sync your updates back to the server.

I've only tested TiwyWiki on IE and Firefox on Windows, and heard that it runs properly on Mac (Safari I think), let me know if it runs for you on any other platform.

This is just the skeleton of a wiki, but it gives a feeling of the possibilities of web applications that can deal gracefully with being intermittantly disconnected. I'm especially interested in hearing back about whether this approach is valuable to you, in comparison to the traditional web and rich client models.
What other applications you'd find most appealing and why?
Here are the ones I brainstormed so far: a personal wiki, various other personal or group GTD tools (such as todo list or calendar), a community wiki, an email reader and/or composer, a blog editor, an RSS reader, an app for driving directions.

Some background:

The problem with using Greasemonkey to extend the browser is that it's not widely available and it doesn't offer good control over cross-domain requests. A Flash and Javascript combination, such as the Flash-based Canvas or AMASS storage, seemed like a better solution.
As I learnt more about Flash 8 and its security model, my original plan of running a local copy of a web page for offline use didn't seem convenient enough: you would have to explicitly save the app locally and synch it before going offline.

When I found out that Flash Player did cache Flash apps properly, the idea of running both the online and offline scenarios took the lead. This avoided the new security restrictions for local apps in Flash 8, keeping two local caches of the data (one for the online domain and one for the local copy) and no installation problem.
Instead you would be able to use the app locally as soon as you used it online. First, whatever content you had already accessed would be cached and persisted locally (in the Flash app/storage). You could use pre-fetching to ensure your local cache would have the data that you want.
Second, the Flash app would act as a buffer for disconnected operations, such as local updates while running offline.

Design philosophy:

I don't see Flash as the long term solution, but rather a temporary workaround that allows for some early experimentation. Instead of waiting for new browser infrastructures, I wanted to demonstrate that web apps with offline support and no install were already feasible, relying only on a new combination of existing techniques.

That's why I tried to keep the extensions to the browser as clean and simple as possible, minimizing the amount of Flash and relying more on the common skillset (HTML+Javascript). I think this will motivate other developpers to try this approach.

In this case, Flash actually turned out to be rather un-obstrusive.
First, if you don't have it installed, the web app will still work fine, except with no offline support or persistent data caching.
Second, Flash offers some benefits that I hadn't anticipated. For example, the storage is shared between IE and Firefox. This makes for a nicer experience that I would expect from any native browser API, such as IE's client storage API or the drafted storage API from the WHAT working group.

For those who want to avoid Flash, other alternative storage techniques could possibly be used to achieve similar results, such as IE's storage API, a Java applet, an ActiveX object or some other kind of browser extension.

In the long run, I hope this proof of concept and the following uses of this technique will help identify the right set of APIs to implement natively in browsers.

Caching:

Loading Flash when offline:
During a troubleshooting session, I noticed something unexpected. The common markup for including Flash objects in IE actually causes a request to Macromedia, which usually replies with a 302 (but no caching headers).
Besides my surprise of discovering that Macromedia's server is hit every time a Flash app is opened in IE, this meant that the Flash object wouldn't load offline. So TiwyWiki uses its own Flash loading technique (yet another) to support running offline.

Busting the cache:
One downside of forcing the application to be cached is that if a new version of the application becomes available, the browser won't notice it until the current version expires from the cache.

I'm still looking for some ideas on how to let the application deal with this update scenario, so that it could have some logic to check for updates and trick the browser into reloading its cache (force refresh). There may be some solutions by using the XMLHttp API with the right request headers, if the different browser could cache the responses properly.
As a last resort, one could imagine a new browser API that would allow invalidating the cache for a given domain and path.

Locking files in the cache:
The other problem with running the application out of the browser's cache is that the user could "uninstall" the application by accidentally clearing the cache or the application could erased from the cache to make room when the cache gets filled up.
I'm still looking for ideas on how to achieve proper locking of the files in the cache.

In IE, that should be possible using the "Offline Favorites" feature. Whenever you bookmark a page, IE gives you the option to "Make [the favorite] available offline". If you check that option, IE will use a crawler (MSIECrawler) to pre-fetch and cache the content for offline reading. You can hint the crawler using a CDF file, linked from a <link rel="offline" href="file.cdf"/> tag.
But I implemented and ran various experiments with "Offline Favorites", and couldn't get the files to be properly frozen in the cache (they would still get scavenged to make room).

Making a framework:

Do you know any generic synchronization framework which could be ported or mimicked in Javascript? Something like TrimQuery would be great if it supported INSERT and UPDATE.

Also, is there some existing libraries that would offer a rich logical view of a persistent storage that only supports sets of name-value pairs?

Developing with Flash:

Open problems:

I've also encountered some weird issues with Flash in Firefox 1.5, showing "Bad NPObject as private data" in the Javascript console and sometimes popping up warnings that an extension mis-behaved. My guess at this point is that it was some interaction between Flash and some other extension, possibly AdBlock.

And finally, I'm still battling some memory leaks issue. Although the code does use closures quite a bit, I can't see how it would create circular references chains between the DOM and the Javascript engine.

Related:

• Adam's Bosworth Alchemy project and famous post on modifying information offline,
• TiddlyWiki, a SPA (Single Page Application) wiki, that runs entirely in the browser (no need for a server) and uses a funky "File->Save Page" persistence model,
• The TODO list for TiwyWiki and the internal documentation (explains a bit about the storage structure and different tiers in the APIs).

Update (2009/01/06):

Dude, this is the shit. You. Rock.

I've been trying to crack this nut _forever_; it's why I made AMASS. Very cool that you figured it out!

We're brother in arms; I have a a library named the Really Simple History library that solves the back button issues in IE and Firefox: http://www.onjava.com/pub/a/onjava/2005/10/26/ajax-handling-bookmarks-and-back-button.html

I've been bringing my AMASS and Flash work into the Dojo project into the dojo.flash and dojo.storage packages; if you are interested, we could coordinate to create a dojo.offline package that uses both of these to extend the capabilities you've discovered to all apps. Email me at bkn3 [at] columbia -dot- edu if you are interested.

Oh, just read the issues about the back button stuff when offline and busting the cache. I'm sure we can find something in IE. Need to study the issue some more. One possibility is to use document.write into the hidden iframe, which might not bust the cache but which would go into history (and you write some state info into the iframe using the document.write), rather than loading a blank.html document.

Hi Brad,
Thanks for your feedback, I'll definitely follow up with you.
TiwyWiki isn't actually using AMASS. Instead it uses yet another Flash object, that also support cross-domain XML requests for the Tiwy RSS reader I've been prototyping.
Some reasons for not using AMASS: I needed something really simple that I would completely understand, I wanted to learn some Flash and also wanted to write the X-domain request feature. Also, as with most Flash framewoks, the HTML inserted by AMASS probably wouldn't let the Flash run offline, as I discovered later.

Congrats on the coverage ;)
Ajaxian is first on my opml, before your blog, and it felt nice to see something about you!

on Safari 1.2.4 is just refeshes repeatedly in a freaky sort of way - but in mac FF works fine.

Julien,

This is very cool. We've looking for something like this for long time. How far have you reached with the framework. I would happy to test for you.

Ram

Ram,
Unfortunately, I don't have a good solution for making a framework out of this, especially the synchronization part.

I do have some ideas on improving the storage with an abstraction layer, but I'm actually focusing on the server-side storage and versioning problem right now.

Great stuff, Julien. Thanks for pinging me on this.

Beware. By depending on Flash 8, you are locking out all Linux users. The newest version of Flash for Linux is still 7.0.63.0, released on Mar 14, 2006.

Alan,
I understand this limitation. This was more of a proof of concept than a full fledged application.
Also, I'm looking into converting the flash object to support Flash 6 or 7.

Julien,

I'm playing with an javascript/ajax app myself and trying to get it to work offline. It really seems like it should work, however I'm having some of the problems in the IE caching area that you mentioned.

It seems that even though I'm not trying to post anything to the sever, it still displays the "connect/work offline" dialoge box.

Could you elaborate on the types of problems you were having?

- Regan