Multi-touch Table Development
The multi-touch table in action
Role: Interactive designer, developer, and fabricator
Primary goal: Create an installation for use at a wedding reception showing photos from the wedding and the bride and groom as children
Tertiary goals: Learn how to create and build multi-touch table for future software development opportunities using a multi-touch table. Also polish up my Actionscript 3.0 skills
View a Flickr photoset here with bigger photos.
Step 1: Inspiration
I’ve been interested in how computers are finally starting to learn how we as humans communicate, with our hands and fingers doing gestures, especially pointing. We learn how to point before we learn how to speak. One simple gesture conveys so much to those around us. “Look at that.” “Isn’t that cool.” “I want that.” etc.
The mouse and keyboard have served us well, but when you take a step back and think about them, they are just weird. Why would I type on a keyboard if I can speak words out of my mouth (for those who can speak)? Why would I use a mouse when I can touch the screen? The problem is that computers haven’t been able to step up to our level of communication in the past.
I’ve played around a little bit with webcam tracking movements in Flash and I had heard for a while about the whole multi-touch movement, but I didn’t think that I could afford the hardware.
For those unfamiliar to what a multi-touch table is, envision a large iPhone the size of a coffee table. Multiple people can stand around the table at once. Everyone can touch the table and interact with the media at one time. “Multi” is the key word here. A normal computer, even a touch screen, has one point of interaction: the mouse pointer. In a multi-touch environment there are many points of interaction. It could be two fingers in your one hand. Four fingers divided between both of your hands. Or, the exciting part, say 16 fingers from maybe six different peoples hands. For more information, look here.
Step 2: The Idea
The final screen of the multi-touch table
So, now I knew that I wanted to make a multi-touch table. I wanted to be on the front-end of the wave of multi-touch development. I wanted to understand how the technology worked, it’s limitations, and it’s possibilities. But, I didn’t want to make a table without having a good purpose for the interface. If I didn’t have a reason for users to interact with the table, then it just becomes a crazy looking box sitting in the corner.
Enter my wedding reception, stage right. I believe the conversation with my bride-to-be went something like this. “Hey sweetie. You know those wedding PowerPoint presentations you see all the time where the father-of-the-bride figures out he can time slides of his baby girl or baby boy to music and make the entire wedding reception watch for 30 minutes. Yeah. I don’t want one of those at our reception.” And my now wife was wonderful and gave me the green light to develop a prototype.
We were married in Ireland a month prior to the Minnesota reception with only 11 in attendance. A small wedding ceremony was important to us. It also meant that most of the reception attendees had not seen the photos of the wedding. This gave me the perfect opportunity for a focused scope on the multi-touch table: showcase photos of the wedding and childhood photos allowing visitors to browse at their own pace during the reception.
Step 3: Learning
Besides the MW2009 workshop, I learned of a many places on the web to explore how to build a multi-touch table. Here are my favorites:
- nuigroup.com – A community of individuals exploring the world of Multi-touch. Much of my research began in this forum of ideas. I posted many questions and received prompt feedback.
- peauproductions.com – Nolan, the site’s creator, distilled information from nuigroup.com into a perfect starting place for newbies. Then, he sends visitors into the wealth of information in nuigroup.com to for deeper learning.
Step 4: Planning
Although step 3 continues well beyond the life of the project, I needed to start hitting the pavement. First I chose to make a Rear DI setup multi-touch table for three reasons:
- I saw the Ideum multi-touch table, which is rear DI, and thus had the most experience in this setup.
- I wanted to plan long term to use fiducials, object placed on top of the table that can interact, and they only work with Rear DI setup.
- I did not want to mess with a compliant layer in the FTIR setup.
I added one other goal to the scope of the project in the planning stage: the table would have to be able to be disassembled and fit into my Toyota Prius.
Basically, the system works with a projector hooked up to a computer shines an image up at a diffused screen under a piece of acrylic. The user sees a familiar but large computer screen. The user then touches the tabletop. Under the surface, infrared (IR) light is pushed up through the surface and bounces back down in the locations where the fingers touch the surface (Imagine touching a piece of window glass. The areas where your figures mash flat on the glass provide a perfect area to bounce light). A webcam modified to see only IR light watches for these touches,called “blobs.” The computer then calculates where the blobs are, how they move, called “gestures,” and then modifies what is shown in the projection accordingly.
Step 5: Hardware
My parts list included:
- Projector: An old InFocus LP 350 projector that I borrowed from my employer. It was an available projector and I had a shoestring budget. Although the projector only used a 4:3 aspect ratio, I decided to build the table with a 16:9 aspect ratio with plans to eventually replace the projector. Also, I used the focal length calculator to estimate how much space I would need to allow for the projector. It turned out that I needed quite a bit, and had to build the table higher than ADA height.
- First Surface Mirror: In order to use space as efficiently as possible inside the box, the projector shines down and bounces of the mirror towards the flat surface at the top of the box. A regular mirror has the reflective material under the glass. Using a regular mirror to bounce a projected image would result in slight ghosting. A first surface (sometimes called a front surface) mirror eliminates the ghosting, but is easily scratched. I purchased one on eBay.
- Mirror Mounts: I wanted a way to be able to adjust the mirror angle as needed after setting up and tearing down the table. My years of drumming instilled in me a need for strong easily adjustable and tightened hardware. I designed the hardware from various existing parts:
- To protect the mirror (and not use glue) I used build-your-own frames (availible by the inch) from Michael’s craft store to surround the mirror.
- After drilling a few holes in the sides of the frame, I put a few long bolts extending out from the frame.
- Then I used turnbuckles as vertical struts allowing enough play to tighten and move as needed.
- Computer: Originally, I used my MacBook Pro running Tiger as the brains of the operation. However, when I started to get into the software development, I did not own a computer with Flash 9 or higher to access Actionscript 3.0. I then used my work laptop, a Dell XPS M1330 running Vista (although later I upgraded to Windows 7), which had Flash CS4, installed. After software development, both computers will run the software just fine, as long as I use Flash CS4 to export the .swf file.
- Acrylic: I used a piece of 0.25-inch thick acrylic purchased from a local large format print company. In the future, I may upgrade to a 0.375-inch thick piece of polycarbonate. The 0.25-inch thick piece has a slight give to it when suspended the distance of the table. Also, I have since worked with polycarbonate in graphic design and it seems to scratch less easily.
- Diffuser: First I tried a left over piece of Canon durable backlit film. Although the projected image held amazing color, the film did not allow IR light to travel through the surface, defeating the purpose of the diffuser. Then, I went to my local art supply store Wet Paint and tried a piece of Duralar matte drafting film which worked perfectly.
- Camera: Most web cams could work for this process as long as the IR blocker is removed, which is supposed to stop IR light in normal use but would defeat the purpose in this situation. I first tried using a Logitech Webcam Pro 9000 that I had sitting around, but removing the IR blocker seemed impossible without ruining the optics. I then ordered a Sony Playstation Eye 3 camera on eBay since it is such a favorite among the multi-touch community. This camera was much easier to modify removing the IR blocker.
- Lens: I tried both a 4.3MM CCTV Camera board Lens and a 2.1mm 160-Degree Wide Angle Lens. The wide-angle lens allowed me to keep the camera closer to the surface, which excited me if I could replace the projector with a shorter throw. However, when the camera was closer to the surface, I had difficulty getting blobs recognized at the far edges of the screen due to the tight angle required. I would have to use two cameras in sync if I were to go this route in the future. For this table, I used the standard 4.3mm lens. I could not use the lens that came with the camera due to focal length issues.
- Lens Mount: In order to use the new lens, I needed an M12 camera mount. The original mount in the Eye camera will not house the new lens. More recently, Peau Productions began selling the mounts. However, I was able to take one out of a cheap webcam on eBay.
- IR Filter: In order to make the camera only see IR light, I needed to block the rest of the light spectrum with an IR bandpass filter. I first tried to use an 850nm filter outside of the lens that I purchased on eBay store OmegaFiltersEbuyer2. However, this limited the viewable area of the camera. So, I then ordered a new filter specifically designed to fit between the lens and the webcam chip from Peau Productions. If you are reading this with intentions to build your own table, just order the whole camera, lens, mount, and IR filter combo from Peau Productions, which became available after my table was complete.
- IR LED Lights: I purchased enough IR LED lights by the foot to circle the outside of the box and extra around the first surface mirror all pointing up towards the bottom of the surface. This produced an even level of IR light throughout the surface. I purchased the lights, connector cables, and 12V power adapters from environmentallights.com who also have a nice multi-touch buying guide.
- Computer Fans: In order to help keep the inside of the box cool, I installed two computer fans on the bottom of the box, one pointed up and one pointed down, creating a nice circular airflow. I also added one smaller fan above the projector (with a modified air duct as a heat shield) to help the projector pull heat out. Note, it is against manufacturers recommendations to use this projector pointed vertically, due to heat issues.
- Power Supply: To feed power to the computer fans, I used an old power supply from a Dell computer. Although the power supply was much bigger than needed, it had one good thing going for it: it was free. I followed these instructions on how to use an ATX power supply without a motherboard. I used a heavy-duty toggle switch to turn the fans on and off, although a power strip could certainly do the same job. But how often do you get to make something with a switch like this? Also my gut felt better about electrical fires by using the switch to be sure things are turned off when I want them off.
- Misc. Wood and other hardware: Most other parts were found at my local hardware store including 0.25-inch MDF boards, 2×2 and 1×2 pieces of pine, 0.75-inch plywood, locking wheels, and nicer pine that I stained for the top.
Step 6: Software
I used various software applications to get the multi-touch table running, including:
- CCV: Community Core Vision – Formally named tbeta, this software tracks the blobs when fingers touch the surface of the table. It is easy to use and calibrate. Go here for a good starting place.
- FLOSC – This application comes with the CCV download. Basically it allows CCV to communicate with Flash. For more information, this post is a great starting place.
- Adobe Flash (with Actionscript 3.0) – Much of the software development takes place within Flash using Actionscript. You can download starter files here and be sure to set the security settings on your local machine.
- The above applications were all I needed to get going, but I did play around with some other packages after developing the table that I hope to use more in the future:
- Snowflake Suite – a nice package of multi-touch applications available for trial download here. If you want to impress some stakeholders who need convincing, let them play with these on the table.
- GestureWorks – a Flash multi-touch SDK to help develop applications
Step 7: The Results
The wedding reception event went quite smoothly. Below are the good and the bad from the evening:
- Nearly everyone at the reception used the table at least once. It had a mesmerizing affect on children, but even my nearly 90-year-old grandmother gave it a go.
- The small scope helped to focus the design nearly perfectly. Most of the time a group of 5-6 people would circle around the table, look at various photos, load more photos, and start conversations. The technology seemed to slip into the background as guests discussed childhood photos often starting with, “I remember when…”
- Setup and teardown did not take too long. And the entire system could fit into my car. However, it was easier to dismantle the night before and reassemble in the garage to then use a pickup truck as transportation the day of the event. Hey, we had a few other priorities for time at the wedding reception.
- I should have devoted more time to prototyping and beta testing. As a side project in my spare time, there was not much time to work with, and I did run a few beta tests with some friends. However, if I had the time, I should have beta tested in many more different demographic groups. In the future I will make sure to include my cousin Alex in the beta testing group. He is 8 years old, very smart, and loves to break things. Within 30 seconds at the reception, he figured out that you could zoom into a photo infinitely, rendering 6 pixels across the screen, and taking the computer to a screaming halt. About every 20 minutes that evening I had to close out and restart Flash to free up the memory. Before the event I had started implementing a maxZoomSize variable, but ran out of time. That will be the first thing I fix when back into development.
- The hardware held up very well. No one kicked the box hard enough to crash anything. The fans kept everything cool enough, including the projector, which I was very worried about.
- There were many complements my wife and I received that night, but the multi-touch table was regularly on the list.
Step 8: What to do Next?
Besides exploring more with GestureWorks and the Snowflake Suite, I plan to install Windows 7 drivers for Multi-touch. I have a vision of using Photoshop on the table for easy masking with my fingers. Oh what a dream. We’ll see how long it takes for that to be possible.
I had also wanted to add a feature to the photo application where users could add their own photos, taken with an external webcam, and leave a message. It would be a great visual guestbook for an event.
But of course the sky is the limit on directions the table can go. Hopefully I will have opportunities to develop many software applications to digitally tell stories. I’m crossing my fingers.