Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edu, jjl@eecs.ucf.edu, dnichols@ist.ucf.edu

chemistoddAI and Robotics

Nov 6, 2013 (3 years and 9 months ago)

78 views


Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson

Institute for Simulation and Training

University of Central Florida


pvarchol@ist.ucf.edu
,
jjl@eecs.ucf.edu
,
dnichols@ist.ucf.edu


Multi
-
Touch: Introduction

Framework Motivation


Primary multi
-
touch components

1.
Physical interaction surface

2.
Software system for collecting and
interpreting points of contact



Present a barrier to entry for
researchers focused on higher
-
level
interface issues or application
development

Multi
-
Touch Origins (pre
-
2000)


Keyboards


Touchscreens


Pen
-
based computing


1984


Bob Boie (Bell Labs) perhaps the first
multi
-
touch screen


1985


Bill Buxton,
Multi
-
Touch Tablet


1990


Sensor Frame, Paul McAvinney (CMU)


Optical Sensor


3 fingers (some trouble with ambiguous finger positions)


1998


Fingerworks

Multi
-
Touch Origins (2000 to present)


2001


Paul Deitz


Mitsubishi DiamondTouch


2003


Jazz Mutant


2004


Andy Wilson


TouchLight


Oct. 2005


Jeff Han


FTIR, UIST Paper


Feb. 2006


Jeff Han


TED Talk


2006


Jeff Han


Perceptive Pixel (CNN Magic Wall)


Spring 2007


Microsoft Surface


Summer 2007


Apple iPhone


2008


Open
-
Source Community


2009


Windows 7

Framework Requirements

Essential Component

Description

Multi
-
Touch Surface

Commercial
-
off
-
the
-
shelf hardware,

requiring
near zero pressure to detect an interaction
point

Software

Hit
-
Testing

Ability to determine the presence and
location of each point of surface contact;
supporting at least four concurrent

users

Software Point Tracking

Identifying a continuous point of contact and
reporting its velocity and duration

Framework Requirements

Secondary

Component

Description

Application Programming
Interface (API)

A software system upon which multiple multi
-
touch
applications can be developed

Multi
-
Platform Support

The ability to access multi
-
touch interaction data from
different computing environments (e.g. languages, OS’s,
整c.)

o散潮o楧畲慴楯i

M潤楦y楮朠t桥hs潦tw慲攠syst敭 w楴桯畴h牥r潭灩污p楯i

Software Service

Allowing multiple applications to access multi
-
touch
interaction data simultaneously, including over a computer
network

Presentation
-
Layer
Independence

Isolating the multi
-
touch interaction data from the system
to graphically present such data, allowing any GUI to be
employed when developing multi
-
touch applications

Mouse Emulation

Support for controlling traditional ‘Window, Icon, Menu,
Pointing Device’ interaction through a multi
-
touch surface

Tangible Interfaces

The ability to detect and interact with physical devices
placed on or near the multi
-
touch surface

Customizable Gesture System

Support for training arbitrary multi
-
touch gestures and
mapping them to software events

Shape Detection

e.g. Finger, blob,

hand

Multi
-
Touch Hardware


Frustrated Total Internal Reflection (FTIR)

1.
Optical waveguide

2.
Supporting structure (e.g. cabinet)

3.
IR sensing camera

4.
Projector and diffuser

5.
IR emission source

6.
Computer

Multi
-
Touch Efforts: Hardware

Software Framework


Image processing


Presentation layer


Multi
-
platform communication


Pen/writing style interaction (Ink)


Gesture recognition


Image Processing Pipeline

Webcam

RotateFlip

Background
Filter

Threshold

Scaling

Blob
Detection

GrayScale




Grayscale
Image 8bpp




Binary Image
1bpp

Calibration

Point
Tracking

Processed

Frame

Camera Frame

24bpp

Image Processing

Raw Camera
Frame

Background
Filtered

Threshold

Processed
Points

Multi
-
Touch Efforts: Software

Mouse Emulator

Function

Gesture Description

Left Click

Quick tap on the surface with one finger.

Left Click (alternate)

While holding down a finger, tap another finger to the left
side of the first.

Drag

Perform a Left Click (alternate) but do not release the left
side press. Drag both fingers to the destination and
release.

Right Click

While holding down a finger, tap another finger to the right
side of the first.

Double Click

Tap two fingers at the same time.

Mouse Wheel Scroll

While holding down a finger, drag another finger vertically
and to the right side of the first. Dragging up scrolls the
mouse wheel up and vice versa.

Alt
-
Tab

While not a mouse command, the Alt
-
Tab command is a
useful Windows feature that switches between
applications. To perform an Alt
-
Tab, hold down a finger and
drag another finger horizontally above the first. Dragging to
the left moves backward through the list of active
applications and dragging to the right moves forward.

Multi
-
Touch Efforts: Video Montage

Multi
-
Touch Efforts: Additional Videos


Demo Explorer


Improved Calibration


Multi
-
Touch Starcraft


Mouse Emulator


Ink Demonstration


Particle System Demonstration


SurfaceSimon


SurfaceCommand


Conclusions


The Framework:


In development since Fall 2007


Open
-
source availability since Spring 2008


Now in its fourth major release


Answers most of the specified requirements


Successfully employed by UCF and colleagues
from Brown University


Downloaded extensively by the open
-
source
community


Provides a robust, extensible platform for
research in multi
-
touch interaction


Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson

Institute for Simulation and Training

University of Central Florida


pvarchol@ist.ucf.edu
,
jjl@eecs.ucf.edu
,
dnichols@ist.ucf.edu


Resources: Commercial Platforms


Microsoft Surface


Mitsubishi DiamondTouch


Apple iPhone & iPod Touch


Perceptive Pixel


N
-
trig


MultiTouch OY

(Finland)


Jazz Mutant


HP TouchSmart


Dell Latitude XT2


TacTable

Resources: Open
-
Source Platforms


TouchLib

(NUI Group)


tBeta

(NUI Group)


reacTIVision


Bespoke Multi
-
Touch Framework


Sparsh UI

(Iowa State)


BBTouch


Touche`

Resources: Online


NUI

Group


TouchKit by NOR_/D


Multi
-
Touch Systems that I Have Known
and Loved



Bill Buxton, Microsoft
Research


Jeff Han’s TED Talk


Bespoke Software