HCI-2 Multi-Modal Interaction

mashpeemoveΚινητά – Ασύρματες Τεχνολογίες

24 Νοε 2013 (πριν από 3 χρόνια και 8 μήνες)

83 εμφανίσεις

HCI
-
2 Multi
-
Modal Interaction

Michael Nielsen

Lars Bo Larsen

Course contents

M.M.1: Introduction to Multi Modal Interaction



Vision and Interaction: The GUI looks at the User

M.M.2: Vision and Interaction: The User looks at the GUI

M.M.3: Vision and Interaction: The User is In the GUI

M.M.4: Speech and interaction I

M.M.5: Speech and interaction II


Course contents


The course will cover


conceptual and theoretical elements of various modalities of human
-
computer interaction,


when we
extend

interfacing explicitly to the 3
-
D spaces

in which, they
actually take place.



Technology enable the computer and the user to


perceive one another and


express themselves audio visually, and hence


Allowing for a rich communicative space


for natural and intuitive interaction.



The course will introduce, describe, and exemplify


fundamental concepts and technology as well as


the various means of communication that become available


Mini Module 1


The world is more than a personal computer
and mobile computing


Integration of modalities


Vision I: The GUI Looks at the User


Face, body, and gesture


Interaction Spaces


Case: Arthur


Exercise in Class


Introduction


Multi
-
Modal Interfaces


Ubiquitous Computing


Enabling technology for disabled people


Wearable computing


Intuitive and ergonomic interfaces


Human motion research


Communication research


Music and Body Motion research

Ubiquitous Computing


Ubiquitous Computing is “invisible,
everywhere computing that does not live on
a personal device of any sort, but is in the
woodwork everywhere.” [Mark Weiser]


1990: Tabs, Pads and Boards


Vision: Smart Spaces/Intelligent Rooms


Opposite virtual reality and seeing the
computer as an assistant


Ubiquitous Computing


HCI Myths:


“People know what they want and a smart
computer assistant can help them get it”


“People interface with the world and the user
interface should act as the boundary
between the user and the computer”

Ubiquitous Computing


Interdisciplinary Work merging human factors,
computer science, engineering, and social sciences


Integrating Senses and feedback (synthesis)


Tactile


Visual


Audio


(Scent and Taste?)


Context


User history


World Geometry


Immersive


Like steering a car


Ubiquitous Computing

-

Smart Spaces

Natural Integration of Modalities

Scent

Sound

Vision

Touch

Senses

Cognition

Taste

Expectation

Context


Memory


Personal
Attribution

Senses

Expectations

Interpretation

Response

Integration of Senses

Primary sense?


Vision


Spatial perceptions


Audio


Temporal perceptions


Synchronization


Accurate model vs Response time


Can cause or solve ambiguity


Redundancy


More often complementary

Multi
-
Modal Systems

Auditory:


Speech


Intonation

Visual:


Facial
expression

Body language

Gesture

Touch:

Tabs, pads,
devices

Senses

Cognition

Expectations

Context:

World
geometry

Application

Activity

Memory:

Grammar

Semantics

History

Personal
Attribution:

User
Configuration

Interpretation

Sensory
Feedback

(Scent)

(Taste)

Senses

Expectations

Auditory:


Speech


Sounds

Visual:


Agents/Avatars

Environment

Virt. HCI entities

Touch:

Force feedback

Low freq. Bass

Electrodes

Physical
augmentations

Feedback

(Scent)

(Taste)

Integration of Modalities


Types of cooperation (TYCOON)


Transfer



Serial chronology



Equivalence



Either or”


Complementary


(in parallel) “
Put that there”


Redundancy



Simultaneous”


Specialization



Monopoly”

Integration of Modalities


Dangers


Ambiguity


Expectation


Conflicting information


Expectation


Modality priority


Action priority


Confusion


Minimize memory load


Maximize Intuition


Integration of Modalities System
Design


Classical view

Gesture
Recognition

Tracking

Speech
Recognition


Application


Mouse
Keyboard

Joystick

Visual
Synthesis

Speech
Synthesis

Haptic
Synthesis

Integration of Modalities System
Design


Think larger!

Visual
Awareness

Aural
Awareness

Computer

System

Physical
Devices

Visual
Synthesis

Sound
Synthesis

Haptic
Synthesis

Integration of Modalities System
Design


Think larger!

Visual
Awareness

Sound
Awareness

Computer

Awareness

Physical
Devices

Visual
Synthesis

Sound
Synthesis

Haptic
Synthesis

MM 4
-
5

MM 4
-
5

MM1,3

MM 2,3