Javascript pen recognition

Here is a demonstration of a Graffiti-style block recognizer in javascript. It's based on the same algorithm as my previous C# implementation, and uses the firefox for the new <canvas> element in Firefox 1.5 (codename DeerPark).


The canvas is one of the many new features in Firefox 1.5. Its a screen area that can be painted programmatically. Apple first introduced it in Safari and the recent Firefox 1.5 (currently in beta) implements it as specified (Web Applications 1.0 spec from the WHATWG group).


Limitations:

The graffiti recognition is not perfect, as I didn't implement some of the manual tuning/heuristics from my previous implementation. Moreover, the alphabet that I used isn't quite the official Graffiti one, but that could be fixed.
Also, I was suprised at how much CPU is consumed by just tracking the mouse and drawing on the canvas; my 1GHz machine gets slown down even with this lightweight canvas demo.


Some further ideas:

Firefox 1.5 has a subset of SVG built-in. It should be easy to write an interface based on SVG rather than the canvas. This would be nice for allowing to the post resulting "ink" back in an XML format.

The canvas element in general opens lots of possibilities for games and my graffiti script could be used for some too (sudoku, crosswords?). Besides that, I've been thinking about a canvas-based graphical interpreter such as Logo (great for teaching programming to kids and reminds me of some good old APPLE II days).

Posted by Julien on September 26, 2005. Permalink
Comments
comments powered by Disqus

Julien, great demo.

I've been thinking for a while about a mouse gesture lib. With all the drag-and-drop going on, i think it would be nice to be able to delete something by striking it out, for example. With canvas, you could easily show the strokes like the FF gestures extension. (I don't think there's a conflict with that extension as ajax mouse gestures would use left mouse button.)

BTW On the pen demo, any chance of swapping the alerts for a "pad" showing the history of letter guesses?

Posted by: Michael Mahemoff at September 29, 2005 11:37 AM

Hey Micheal,

What kind of gesture are you thinking for scratching? A single line (left to right or vice versa) or a more complex scratching gesture?
The latter wouldn't work with my current implementation, as the number of back and forth would vary.

I removed the alert box. I actually had de-activated the "pad" code before putting it online ;-)

Posted by: Julien at September 29, 2005 12:23 PM

In the case of deletion, probably a single diagonal stroke is good enough provided it did the usual gesture-like things - "You appear to be using a mouse gesture for the first time. ..." - and a visual indication of what will happen before the button is actually released.

More generally though, I had in mind an engine like the FF extension, which transforms gestures into a sequence of numbers, so can easily be used generically. (I don't have the extension right now, I think it uses a number 1-8 for each possible direction.) You could then say:

portlet.onGesture = function(gesture) {
if (gesture.strokesMatch(1,8)) {
// User went left-up then right-down
}
}

Posted by: Michael Mahemoff at September 29, 2005 03:11 PM

I have just converted this over to work with dojo (dojotoolkit.org) and this trick so that it works with IE too:
http://me.eae.net/archive/2005/12/29/canvas-in-ie/

Email me if you want any further details.

I haven't made it public though because you are using GPL.

Morris

Posted by: Morris Johns at February 14, 2006 05:44 AM

Morris,
Nice work. I'd be happy to see it.
GPL doesn't prevent you from making it public, on the contrary. What it means is that you have to provide the source of any modifications that you make. It basically "prevents" someone from taking from the community without giving back too.

Let me know when you post it.

Posted by: Julien Couvreur at February 20, 2006 11:11 AM
Trackbacks
Javascript pen recognition
Excerpt:

Julien Couvreur have written a very nice javascript that allow pen recognition based on his previous


Weblog: AJAX
Tracked: September 26, 2005 10:44 AM