Multi-touch for Adobe Flex: Introduction in Hardware and Software

I’m convinced that keyboards and mice are going to be complemented with multi-touch screens in a staggering short period. This believe drives me to check the current state of the technology. I split the analysis into three areas: the hardware, the “software engine room” and “user gestures and applications”. Because we need a presentation wall at Faindu, I confine all aspects to large screen applications.

Generally, NUI Group is the hot spot. For diving into the technology, I also found the master thesis of Laurence Muller published this September quite useful.

Multi-touch Screens

All commercial hardware I found starts with a price tag of several 1′000 €. And until the 3D-camera sits in every laptop, we will need to settle for 2D surface interaction. There is a thread on NUIgroup that summarizes nicely the following four technologies. All four options are based on the same physical principle: finger tips or any other objects emit infrared (IR) light and are tracked by an IR-filtered camera. Those artifacts detected by an image processing algorithms are called “blobs“. The tracked object could actively emit light with an IR-LED but this required the user to wear gloves or something like a mini-torch. Because this is not geeky enough, all principles rely on reflecting IR-light on the object. So the real challenge is to illuminate several objects on the surface with evenly distributed intensity and indirection.

• Diffused Illumination, DI: Several IR-lamps spot form the the back onto the surface.
• Diffused Surface Illumination, DSI: Tim Roth introduced the DSI technology. Contrary to what the name implies, the setup is very similar to FTIR
• Frustrated Total Internal Reflection, FTIR: While DSI relies on scattering the light in the material, FTIR uses optical reflections at the surface. Think of a thick telceom multimode fiber that is pressed into a flat form. A nice diagramm of Jeff Han’s former group, comprehensive instructions pdf by Harry van der Veen and other instructions by Christopher Jette.
• Laser curtain or Laser Light Plane (LLP): One or several lasers span a curtain of light beams one millimeter in front of the screen. It was introduced by AlexP (update, newest post) on the NUIgroup forum and very well explained by Georg Kaindl.

If security issues can be resolved, LLP is my clear favorite.

Software on the “operating system” level

With “operating system” (OS) I mean the software that reads the data from the camera/s, processes it and sends the appropriate events in Actionscript. For the mouse or keyboard, this is on the operating system level, so I name it that way. TUIO is the standard for the communication between the multi-touch screen and applications.

• Touchlib: The software library touchlib seems to be the standard. How to setup a touchscreen. There is also WiimoteTUIO that tracks up to all four points of Wii Remote.
• tbeta: tbeta is a recently released package by NUIgroup. It looks quite handy and there is a getting started guide.
• IDEO: The well-known design firm compiled on the basis of processing an OS stack for Flash called ideo-multitouch.
• simtouch: TUIO message simulator for AS3 development.
• Processing: not sure what it is or rather what it isn’t.
• [Added 12th Dec 2008] wiiflash puts all events of a Wii remote into Flash Player.

I will explore tbeta and IDEO in more depth.

User gestures and Flex multi-touch applications

Apart from Bill Buxton’s page that also features an excellent overview, I could not find yet a lot of research about gestures. I want to know what people intuitively understand. Wiping horizontally with two fingers means to most users…? Tipping with three finger should do…? And the combination with two hands? Anyway, there are may applications around:

• A blog of Harry van der Veen list three Actionscript applications developed by the NUIgroup.
• Multi Touch Flickr Viewer lets you play with images.
• mapstouch is an Air application and lets you navigate google maps. It is based on touchlib.
• sparsh-ui is developed at the Iowa State University and features a gestures server. However, I could not find an Actionscript API.
• There are no Flex components optimized for the multi-touch paradigm yet.
• Website that collects videos of multi-touch applications.

---

Possibly related posts: (automatically generated)

• Flex and Flash Player: How they live together
• BlackBerry Storm update landing tomorrow, bringing lots of good stuff (upda…