Multitouch Display - Part One
Multitouch video collection |
Table of Contents
(last update November 13th 2007)
Part One
Part Two
Part Three
Intro
Have you seen the Jeff Han's video showing his incredible multitouch table? Me too, that's why I could not resist and I started to try building one of them!Here on the left you can see a photo taken from the video of my first multitouch experience.
In this page you'll find some information about what is a multitouch display and how it can be built, how I built mine and, hopefully, the updates about the evolution of my new creature. This project is always work in progress.
What is a Multitouch Display?
Accordingly to Wikipedia: "Multi-touch is a human-computer interaction technique and the hardware devices that implement it. Multi-touch consists of a touch screen or touch tablet (touchpad) that recognizes multiple simultaneous touch points and software to interpret simultaneous touches". So a multitouch display is a display capable of multiple touch recogniction. Probably this definition is not giving you the right idea, so I suggest you to take a look at my multitouch video collection on YouTube, just clicking on the play button in the following video:
BlaXwan's multitouch videos |
Just to point out one of the main big differences with other fancy devices that came out recently from Apple, multitouch means more than one touch, dozens, hundreds of touches. The Apple devices are actually capable of sensing two touches maximum.
The other big difference is that the kind of multitouch displays that I'm talking about in this page are based on a so simple principle that let them be really cheap and easy to build up. No hard industrial processes or sci-fi microelectronics, just a piece of plexiglas, a webcam and some LEDs, really!
How is this achieved? Well, basically this is done "looking" at the display. This technology is actually possible thanks to the high computational power of the modern PC. This power is used for image recogniction. The kind of multitouch display I'm dealing with in this page are based on computer vision so that a webcam is pointed to the display and a software running in the PC is able to recognize the points where the user is touching it.
Now, there are two main approaches when trying to grab the images from the display: the FTIR method and the DI method. In both cases infrared light (IR) is used to illuminate a sheet of plexiglass and the webcam used is modified to let it see just the infrared light and not the visible light. This is made to avoid that the visible light of the environment, and also emitted by the video projector, impacts the quality of the touch recogniction.
| Back to top |  |
DI Multitiouch
DI stands for Diffused Illumination. In this kind of Multitouch setup approach, the panel of plexiglas is illuminated from the back, with an IR beamer.
The IR light from the beamer is diffused on the back of the plexiglas surface and this way the webcam is able to see what's on the plexiglas surface or very near to it. This method has some advantages and, unfortunately, some disadvantages. The good things are that it is relatively easy to setup and it does not require a compliant surface like the silicone rubber layer to work properly. It is also the method that may recognize the "fiducials". Fiducials are symbols printed on object so that the computer is able to recognize which symbol is placed where and how on the screen. This technique is used, for example, in the Reactable to use plastic cubes with fiducials painted on the back as a controls for the sound generation, the cubes may be used just like knobs. Microsoft Surface is actually based on this kind of approach.As far as I know, the main problem with DI is that this kind of system are quite sensible to the ambient light. This is caused by the fact that they "sees" what's over the plexiglas surface and not just what hit it. When building DI systems at home, another problem that I've read about (I repeat: I didn't tried this multitouch approach) is about the correct positioning of the IR beamer in the multitouch box and about the material used to diffuse the IR light on the back of the plexiglas panel and to rear project the image onto.
Actually I'm considering to use this multitouch technique for my next multitouch display. The main reason will be for playing with the fiducials.
| Back to top | |
FTIR Multitouch
FTIR (frustrated total internal reflection) is a very simple principle that basically says that if you light up a certain material with some light, the light will stay inside the material itself, boucing on it's external surfaces until the external surfaces is "frustrated", for example, with a finger touch. In this case the light is bounced back from the finger and exit the material. So, with FTIR multitouch display, this material is plexiglas and we use the usual IR-pass modified camera to track the finger's position. This principle works also with infrared light so that the tracking light will be not disturbed by visible light.
To resume: a plexiglas plate is acting as a multitouch surface. A bunch of infrared LEDs is placed on the plexiglas edge and light up it. When a finger touches the plexiglas the IR light is bounced back and get caught by a modified webcam: easy!!!
Some additional triks must be used: a standard webcam is not able to catch the IR because it comes with an IR-block filter, you should get rid of this filter then the webcam will be able to see IR. In addition, some layers of exposed photo films are used as IR-pass filter so the webcam will be no more sensible to the normal light.
The image is projected with a video projector on the plexiglass plate, this requires a projection surface. Someone is using tracing paper or mylar but the best effects in terms of image quality are given by plastic projection films.
The projection surface will give good quality image but the FTIR effect will be disturbed by this additional surface, requiring hard pressing on the display to activate the effect and let the webcam see something. To solve this issue a compliant surface is used as layer to be placed beetween the plexiglas and the projection screen. The compliant surface will help to better couple the two materials and ease the FTIR effect and the sensibility and responsiveness of the display. The silicone rubber seems to work really well as compliant surface.
| Next page |  |
| Back to top | |
27 Comments
Leave a comment
Navigate this site!
|
|
|
|
 |
Hi, BlaXwan,
Great Videos!
We also built a multitouch display using both FTIR and DI. But we are still looking for the right tabletop surface. A very thin silicon rubber sheet could be the answer. In your current setup (silicon rubber, sketch paper), can the camera see fiducials? How about an etched fidicual (etch the shape of a fiducial on a acrylic board)?
Andy
Hi Andy, thanks for your comment!
Actually I didn't tested printed fiducials because they can't work out with an FTIR setup, they should go just with DI. Etched fiducials may work out indeed, this is a good idea. You have to think about the fact that FTIR tends to round the contours of the object that is touching it. So I think that it is not good to found the fiducial recogniction on the shape detection but probably there are more chances to let it work if it is based on the recogniction of a certain blob pattern, something like a matrix of dots like some of the fiducials used in the ReacTable but without any solid contour. I would try heavy objects with some jutting tips on one faces, like big Braille alphabet letters.
If you make some tests in this direction, let me know!
Hello Blaxwan,
I like this technology ;)
Can you explain me the connection between PcPowerPlug - ID Multitouch - Webcam IR - PC ? more details please.
Best Regards
Hi Aerohache,
the webcam is FireWire, so it is connected to the FireWire port but you can use also an USB one.
I don't understand the other questions, I mean: the PC power plug is connected to the power mains as usual! The LEDs are powered through an external 12V power supply.
If you want more details you should ask detailed questions! :o)
if i wanted to put it on the wall, how would i go about doing so
is this a question?
I first want to say that this is an excellent site!
I have a few questions:
First, are you using keystone correction on your projector? When I reflect the image off a mirror I get a really bad keystone and my projector does not have the ability to fix it.
Second, I am getting a lot a of noise(light)(including from the projector) that gives false bulbs. Did you have an issue with this and if so how did you fix it?
I purchased the silicon but yet to get the rear projection screen.
Finally, I finally got the calibration to work but I ran the pong and the smoke demo in hte beta2 realease and I get no reaction, any ideas?
@ Chris
Thanks Chris, I really appreaciate that you like the site!
Yes I used some correction on the projector, this is because otherwise it would be hard to match the projected image on the scree.
I also had a single false blob given by the central projector lamp light. I solved putting an additional glass (a transparent one) just in front of the projector, before the mirror, this attenuated very slightly the image brightness but blocked the projector's lamp IR.
Some time has passed since my last multitouch software testing, I hope that something was improved because I had a lot of trouble with it, it was quite hard to get it working!
I think that we should start a good opensource project for multitouch OS or similar.
hi, i have a question.
There is someone that sell a ftir multitouch surface already assembled and operating, without constructing one by myself?
Hi Pedro,
surely there are someone interested in selling multitouch panels, I would suggest you to post a question like this in the NUIgroup.com forum. AFAIK there are some guys that started a professional activity based on this technology. The price, of course, may be at least one order of magnitude higher than what you may spend in building one by yourself.
Hey BlaXwan,
I have to say without a doubt this is the best site I have come across in a long time. Its rare to find someone who is interested in sharing this kind of technology with the general public and not charging them an arm and a leg for it. Can ask what Projector you are using and maybe how you have it set up in the design of the whole unit.
And in reference to Kevin's question "can you put it on the wall?" would it be possible to work in a flat panel monitor to the back of the unit instead of using rear projection. Sorry is this comes off as a lame question I am not so technical and would love to see a set up using a flat panel monitor. Or maybe illustrations on how something like that might be done.
Again you have a great thing going here. Thanks so much for sharing this with everyone.
Hi Ben,
thanks a lot for your words, I'm really happy when I can see that my work is appreciated!
Unfortunately I don't have a projector, that's why I didn't developed my multitouch to a good and complete state. So I was using from time to time projectors lended from friends.
On the flat panel question, I knew that some guys were successfull in using an LCD panel under the plexiglas itself. You can find some info on this searching in the NUI Group forum. The question is to find a suitable LCD panel (with good IR transparency) at the first attempt, without disassembling/breaking tens of TV or monitors.
Thanks, regards
ey man i want to make some similar
but i only have a question where i can find the software for mac or for less fortune where i can find a opensource code in a language easy to traslate
how actionscript or perl or java or python or c or IDontKnow
i hope u can helpme
thanks :)
ey man how can i do some similar
software for mac???
Hi Markintosh,
unfortunately there's nothing for mac AFAIK. Touchlib is opensource so yuo may start from there for a porting.
Hey BlaXwan
Awesome tutorial, really sweet. It looks like i can do this :D. But... is there a way to make a multitouch like this, on a LCD screen or some other screen type. Because i cant get a projector.
Anyway thanks mate.
hi BlaXwan
Very cool tutorial! will i get better results with a 60 fps webcam?
Thanks and keep up the good work!
@Terramoto
Thanks! Yes, the higher framerate the webcam gives the better interaction experience you will have. Actually the best results are with Firefly MV camera from PTGrey and an optimized version of touchlib that reaches 102 frames per second. This means high and fast responsiveness of the display to moving finger touches.
Ciao,
complimenti per l'ottima guida.
Nel mio studio abbiamo già realizzato negli anni scorsi dei tappeti interattivi (facendo tutto da 0).
Adesso stiamo realizzando il multitouch in particolare per integrarlo con il nostro motore 3D.
L'unico problema incontrato è sulla carta da mettere tra lo strato di gomma siliconica e la pellicola da retroproiezione.
Ne ho già provate una decina e nessuna funziona.
In particolare le pellicole semitrasparenti (dovrebbero essere tracepaper) si incollano. LA sketch paper non riesco a trovarla.
L'unica che funziona è una velina, ma la resa della retroproiezione è terribile in quanto ha una grana molto vistosa.
Riesci a dirmi che tipo di carta hai usato?
Grazie
Gigi
@Gigi
Ciao, in generale considera che si dovrebbero avere ottimi risultati anche con la sola gomma siliconica e la pellicola di retroproiezione. Nella pratica però questo avviene solo con pochi tipi di gomma siliconica come ad esempio la Sorta Clear 40: Il tipo che ho usato io, Crystal Rubber, putroppo non rientra tra le più adatte. Per questo ho dovuto utilizzare uno strato intermedio. Nel mio caso ho ottenuto i migliori risultati con la "carta da schizzo" (sketch paper) che è una specie di carta da lucido ma molto più grezza (leggi più porosa) e per questo "appiccica" molto meno della carta da lucido (tracing paper). Si vende quasi solo nei negozi di belle arti perchè viene utilizzata come carta economica per schizzi e disegni di prova.
Fammi sapere come va.
Ciao
complimenti per l'ottima guida (anche se una traduzione in italiano ci starebbe). ottimo sistema di multitouch ma che sofware si deve utilizzare? (preferibilmente freeware)
saluti matz
e ancora complimenti
Ciao e grazie per la risposta,
avevo immaginato si trattasse di quella carta. Adesso abbiamo un pelo accantonato il discorso carta in quanto funziona abbastanza bene. Stiamo lavorando su un multitouch di grandi dimensioni in verticale ma utilizzando i laser x l'infrarosso e le pellicole holoscreen opache.
Stiamo valutando anche delle camere con un framerate alto e una latenza minore...
Ti terrò aggiornato.
Ciao
Gigi
Hi there i have just been trying this out i used two ir led taped to the sides of a small but of perspex and tried useing my wii remote to detect for a ir scorce being reflected back from putting my finger on the perspex. Has any one tried makeing a touch screen this way (i must add i could not get it to work even thought i can track the ir leds without the perspex with the wii remote)
Hi BlaXwan :)
I'm interested in multi-touch screen with LCD. But I can't buy plexiglas. Where did you buy it? and what product number is that? I can't find it. If you tell me it, I thanks you!!
PS) I'm korean, so my english is poor. :) your understanding is kindly requested.
Ciao,
complimenti davvero per l'ottimo sito e per l'accuratezza della guida, non pensavo che esistesse un modo così semplice per creare un display multitouch! Complimenti anche per l'ottimo lavoro svolto!
Volevo chiederti se è possibile sapere come configurare il software "Touchlib" per funzionare con l'hardware costruito.
Grazie in anticipo! E ancora complimenti!
Ciao,
Thomas.
ciao! complimenti davvero per l'opera e per la guida, � davvero ben fatta, per� ho riscontrato un problema con il film rosco gray e con la peroni panorama, non ho idea di dove comprarle!
sono italiano e non s� se possono spedirmelo qui, tu hai qualche suggerimento?
Grazie infinite!
Ciao! e ancora complimenti!
P.S potresti rispondermi al mio indirizzo email?
grazie!!!!
Hi BlaXwan, thank you for sharing your experiments (Ctrl+D!)
A friend and I just begun experimenting in this technology, but without much success. This is not really a question but a comment on our (very short) experience.
We begun by trying to estimate how few LEDs/cm we could use along the border of the plastic while a webcam is still able to detect IR light reflected on fingertips at about 20cm from the LEDs. The LED should also not be so separated as to leave low illumination zones between them.
We checked that the old webcam has no IR block (it saw the LED from a remote), added the visible light block (it still saw the LED and everything else went very dark).
Then we connected in series a few 3mm LEDs and applied about 15mA through them. The idea was first to verify that the illumination reached at least 20cm along a piece of paper (no Plexiglas yet), and then start separating the LEDs until it started to show low IR illumination between them.
Apparently the cam was not sensitive enough to IR and/or the LEDs we used not powerful enough, because the cam could barely see the IR illuminated paper, even with the LED pointing directly at it millimeters away. We fumbled with the brightness and the auto-gain settings without getting any improvement. We also tried using an old Sony Handycam, and that also saw the IR LEDs but not the IR illuminated paper.
Maybe next weekend we will go and buy a better webcam, perhaps one of those with night vision already with IR LEDs built-in, and cannibalize it. I guess those MUST be sensitive enough to IR, and also we can use the same LEDs that come with it (also taking things apart is fun).
Eventually we plan to try to mount the Plexiglas over a LCD monitor and the webcam IN FRONT and above it, to see if it sees the IR blobs from the fingertips reflected on the back side of the plastic (the side facing the LCD). If this works, fine! If not, we can always built a multitouch coffee table.