Of Multitouch and HCI

A buzz word running around at the moment is multitouch. The two big giants are throwing it around like anything. Apples iPhone and iPod Touch uses a multitouch display, Microsoft Surface uses a HUGE multitouch display. But what does it all mean?

The iPhones multitouch display goes as far as you can have two points of contact on it at any one time, this is how they achieve the 'pinch' zooming function. However, Surface can take in as many points of contact as you want, true multitouch.

I'll go into how it works another day, I want to talk about possible applications today. Focusing on two major areas of possible use. Business meetings and lectures. Lets take a typical meeting, six people sat round a table all from different departments. One more person taking the minutes of the meeting. Each person has an ID card tagged with RFID which the drop onto the table infront of them. The table glows into life. Where once it looked like a black glass surface now it turns into several displays, one for each person.

On their display shows a Home area that contains all their documents, a presentation window that allows information to be displayed to everyone in the meeting in real time. A notes page that can be hand written to with automatic conversion to text. Now imagine that all this can be interacted with with multiple touches of your fingers. Everything works with everything else. So you can drag the current showing slide from the presentation window straight to your notes. Or drag a keyword to a search box that automatically Google's it for you.

To head towards a perspective display, under all these main windows is a virtual, top down view of the table. Around its edge it shows who is at the meeting, exactly where they are sat in real life. Want to share a file with the entire table? Pick up the file from your area, tap the table to bring it to the top and drag the file to the middle of the table, let go and it automatically appears on everyones screen. Only want it to one person? Just simple drag it to the person you want to see it and it will appear on their screen.

Looking at another use, lectures. You don't have to worry about that the lecture is in America and you're in England. The lectures being streamed across the internet. Video of the speaker appears on your display, along with their presentation. Again you have access to files, etc. Even relevant E-books that can be searched through. Find something interesting on a web page thats relevant? Select it, and drag it on to the virtual lecture theater. Or type out a question and send it straight to the speaker. It doesn't matter where anyone is, they all appear in the virtual theater as if you are all in the same room.

Touch displays could be taken in any direction. Alot of the technology talked about here isn't just multitouch, but also networking. But this networking is something I think we should look at doing, why does the speaker need to be in the same room at a lecture? Bournemouth University are going to be doing their first live streaming of a lecture from one campus to another soon. If every person in the two theaters had their own display you could just have the video feed of the speaker upfront with their presentation or notes on the screen in front of you. But its the interaction with this system where multitouch comes into play. For example, placing your finger on a file to select and hold it then using your other hand to tap on the table of lecture theater to bring it to the surface so you can drag the file to who you want to show it to.

However HCI comes into it, what sort of hand gestures should we use? Already we've seen the pinching motion on the iPhone to zoom in and out of pictures. Of flicking your finger up the display to scroll down the contact list. Jeff Han shows a couple of other gestures that his team have come up with when using World Wind. For example placing two fingers from the same hand horizontal on the screen and then using the other hand to rotate the view around that axis created by the your first hand. These are important things to consider if one is looking to develop a multitouch system. What gestures are most natural? What gestures are harder to do than others? For example to delete or close a window place all five digits of a hand on the screen then bring them all together. It's not a regularly used gesture so you wouldn't accidentally close or delete something yet, it symbolizes scrunching something up in your hand and throwing it away. Symbolism, I believe, is something else to consider when looking at hand gestures. What motions do we do in real life and how can we relate them to this interaction with a display?