Recognizing and Reproducing Gestures - Moodle Archive - EPFL

builderanthologyΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 3 χρόνια και 11 μήνες)

106 εμφανίσεις

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC


Doctoral School


Robotics Program

Autonomous Robots Class

Spring 2008


Human
-
Robot Interaction



Aude G Billard


Learning Algorithms and Systems Laboratory
-

LASA

EPFL, Swiss Federal Institute of Technology

Lausanne, Switzerland


aude.billard@epfl.ch


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Overview of the Class

9h15
-
12h00

1.

Interfaces and interaction modalities



non
-
verbal cues and expressiveness in interactions: gesture,



posture, social spaces and facial expressions

2.
User
-
centred design of social robots: humanoids, androids, etc.




motivations and emotions in robots



social intelligence for robots

3.
Social learning and skill acquisition via teaching and imitation


14h15
-
17h00:

4.

Robots in education, therapy and rehabilitation


5.

Evaluation methods and methodologies for HRI research


6.

Ethical issues in human
-
robot interaction research


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Overview of the Class

9h15
-
12h00

1.

Interfaces and interaction modalities



non
-
verbal cues and expressiveness in interactions: gesture,



posture, social spaces and facial expressions

2.
User
-
centred design of social robots: humanoids, androids, etc.




motivations and emotions in robots



social intelligence for robots

3.
Social learning and skill acquisition via teaching and imitation


14h15
-
17h00:

4.

Robots in education, therapy and rehabilitation


5.

Evaluation methods and methodologies for HRI research


6.

Ethical issues in human
-
robot interaction research


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

WHY HRI ?

. Why is HRI beneficial?

. Assistance for the elderly and disabled

. Assistance with routine tasks which cannot be fully automated

. Service Robots

. Entertainment Robots

Ri
-
Man robot from Riken

Stair
-
Climbing Wheelchair "HELIOS
-
III

Hirose’s lab

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

ISSUES in HRI


Safety


Semi
-
structured/unstructured environment

.

Untrained users



Perception


Perception of the environment


Perception of the user


Perception of the user’s .intent.

. User Friendly Interaction

Interaction Lab, USC

Adaptive Systems Group, UH

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

To
communicate effectively

with humans, robots
should be able
to perceive and interpret

a wide
range of communicative modalities and cues.




Types of interactive human
-
robot interfaces that
are most
meaningful

and
expressive

for
collaborative scenarios


• gestures based interfaces;

• non
-
verbal emotive interfaces;



emotion
-
based;

• sound based interfaces;

• computer based interfaces.

Interfaces and interaction modalities

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Multi
-
Modal Means of Interaction

Steil et al, Robotics & Autonomous Systems 47:2
-
3, 129
-
141, 2004



Human
-
robot interaction requires the use of more than one modality at a time.



Multiple sources of information provide redundant information, which helps recover
from noise in each source.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

R. Dillmann, Robotics & Autonomous Systems 47:2
-
3, 109
-
116, 2004

Multi
-
Modal Means of Interaction

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Tracking Human Motions




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Data acquisition

3 Xsens motion captors

provide 3D absolute orientation

at 100 Hz

4 joint angle values are

computed

Gestures based interfaces

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

Interfaces


Motion Tracking

5 X
-
Sens Captors

Absolute orientation of 11 Degrees of freedom at 100Hz

3 DOF Torso, 3 DOF Shoulder, 1 DOF Elbow, 1 DOF Wrist


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

5 X
-
Sens Captors

Absolute orientation of 11 Degrees of freedom at 100Hz

3 DOF Torso, 3 DOF Shoulder, 1 DOF Elbow, 1 DOF Wrist


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Tracking Head Motions and
Pointing Motions




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

Interfaces


Head Motion Tracking

TOP: The user gazes at the robot to attract its attention (turn
-
taking cue).
Then, he looks and points at an object in the environment. The robot
follows his/her gaze, and observes the pointed object. The user gazes at
the robot again (turn
-
yielding cue).

BOTTOM: In turn, the robot gazes and points at an object, while the user
looks at the selected object. The robot gazes at the user again to request
an evaluation of its selection (turn
-
requesting cue). Finally the user signals
to the robot whether the correct object has been selected by
nodding/shaking his/her head.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

Interfaces


Learning Communicative Gestures

4 X
-
Sens Captors to track the motion of the head motion and the right Arm

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

The table and the 3 objects are represented with dotted lines
and circles. The ellipse represents the intersection of the
vision cone and the plane defined by the table.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

The gazing (or pointing) direction is estimated by computing

the intersection of a cone of vision with a plane (the surface of a table).


The intersection is described by a 2D covariance matrix which gives a
probabilistic estimate of where the user is looking at (or pointing at).

Calinon, S.

and
Billard, A.

(2006)
Teaching a Humanoid Robot to Recognize and Reproduce Social Cues
. In
Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO
-
MAN).

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gestures based interfaces

Interfaces


Head Motion Tracking

1 X
-
Sens sensor on the user’s head, 1 motion sensor on the user’s chest

Absolute orientation of 3 Degrees of freedom at 100Hz

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Recognizing and Reproducing
Gestures




Recognizing Motions: Which Representation?

The joint angle trajectories convey redundant information

PCA or ICA techniques are used to project data onto an uncorrelated

or independent representation, to reduce the dimensionality of the dataset

and to proceed to a classification across the principal components

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Dimensionality Reduction: 4


1

1
st

PC component

In hand path

Encoding and Recognizing Gestures

1
st

PC component

In joint trajectories

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Movement Decomposition through ICA

Dimensionality Reduction: 4


2

Encoding and Recognizing Gestures

Trajectories are then segmented and encoded as time series
in either Recurrent time
-
delay neural networks or into
Hidden Markov Models

1
st

PC component

In hand path

1
st

PC component

In joint trajectories

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

From Recognizing to Reproducing Gestures



Left
-
Right Model



Nm states optimized with BIC



Continuous density prob. for
encoding time and joint trajectories

Calinon & Billard, IEEE International Conference on Intelligent Robots and Systems,

2004, 2003

Calinon & Billard, International Conference on Machine Learning,

2005 (submitted)

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Recognizing and Reproducing Drawings of Letters of the Alphabet

From Recognizing to Reproducing Gestures

Calinon & Billard, IEEE International Conference on Intelligent Robots and Systems,

2004, 2003

Calinon & Billardl, International Conference on Machine Learning,

2005 (submitted)

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Encoding and Recognizing Gestures

The model performs better with PCA preprocessing than with ICA

PCA
-
HMM

Recognition

Threshold

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Guenter & Billard,

IEEE
IROS 04
; Calinon & Billard, ICML05 (Sumbitted), Calinon & Billard, 2005, MIT Press (To Appear)

Encoding and Recognizing Gestures

The method allows to disambiguate very robustly

across various gestures

Calinon & Billard, IEEE International Conference on Intelligent Robots and Systems,

2004, 2003

Calinon & Billardl, International Conference on Machine Learning,

2005

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gesture Recognition

Gestures are matched to gesture models backwards in time and at
several time scales. A Gaussian
-
like matching function gives an
indication of the likelihood of the match.





















Jamie Sherrah and Shaogang Gong, "VIGOUR: A System for Tracking and Recognition of Multiple People and
their Activities", to appear in Proceedings of the International Conference on Pattern Recognition, September
2000, Barcelona Spain.


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Schematic setup of E
cho State Network

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Schematic setup of ESN (II)

Output weights :

trained

Inputs (time series)

Internal state

Output (time series)

Input weights :

Random

values

Internal weights :

Random

values

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

How do we train
W
out

?


It is a
supervised

learning algorithm. The
training dataset is


the input time series

the desired output

time series

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Network inputs and outputs

Blue line : desired output

Red line : network output

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Gesture Recognition

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Tracking Gaze Direction

Tracking Eye Motion and

Inferring Direction of Gaze




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Eye
-
Tracking System

Applied Science Laboratory, http://www.a
-
s
-
l.com/

Mobile Eye from ASL
:

The eye image and scene image are
interleaved and saved on a DVCR
tape.


This method insures no loss of
resolution, Tape duration is 75 minutes,
battery operation, at full charge, is 130
minutes.



Visual range:


50
degrees horizontal, 40 degrees vertical

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Eye
-
Tracking System

Applied Science Laboratory, http://www.a
-
s
-
l.com/

Driving

Mobile Eye from ASL

Magazine Shopping

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Eye
-
Tracking System

Lorenzo Piccardi, Basilio Noris, Olivier Barbey, Aude Billard, Giuseppina Schiavone, Flavio Keller, Claes von Hofsten.
WearCam: A
head mounted wireless camera for monitoring gaze attention and for the diagnosis of developmental disorders in young
children
, IEEE RO
-
MAN 2007, Special Session on Assistive Technology.

The WEARCAM

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Face Detection and Recognition

W ZHAO, R CHELLAPPA, PJ PHILLIPS, A ROSENFELD

,
Face Recognition
: A Literature Survey , ACM
Computing Surveys, 2003



Huge literature on face detection and face recognition



Belongs more to research in Vision and Machine Vision

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Face Detection and Recognition

W ZHAO, R CHELLAPPA, PJ PHILLIPS, A ROSENFELD

,
Face Recognition
: A Literature Survey , ACM
Computing Surveys, 2003



Current algorithms for face detection may not be optimal for HRI, as they
assume that the person faces the camera.



But, in reality, face are often tilted importantly with respect to the camera;



Besides, detecting the person from behind may also be important.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC










Realistic set
-
ups, Severe lighting changes,
Extreme in
-
plane rotations, Cluttered background



Eye
-
Tracking System

The WEARCAM

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC



Enhanced algorithms for video
-
based detection of
faces that are
robust against important degradation of the
image’s quality
, and that are
invariant to in
-
plane rotation
.

Analysis of Head Mounted Wireless Camera Videos for Early Diagnosis of Autism

Noris, B.,
Benmachiche, K., Meynet, J., Thiran, J
-
P. and Billard, A.
(2007). International Conference on Recognition
Systems
-

CORES’07

Eye
-
Tracking System

The WEARCAM

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Audio
-
Visual Analysis of Scene Analysis

Eye Gaze Direction Detection + Object Detection

Object Detection

Object + Gaze Detection

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Eye
-
Tracking System

The WEARCAM

Tested acceptability of the WearCam with 18 children, 2
-
4 years old.

Lorenzo Piccardi, Basilio Noris, Olivier Barbey, Aude Billard, Giuseppina Schiavone, Flavio Keller, Claes von Hofsten.
WearCam: A
head mounted wireless camera for monitoring gaze attention and for the diagnosis of developmental disorders in young
children
, IEEE RO
-
MAN 2007, Special Session on Assistive Technology.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Handheld Interfaces

Speech / Vision




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

PDA
-
Based Interfaces

T. Fong, C. Thorpe, and C. Baur,
Multi
-
Robot Remote Driving with Collaborative Control
, IEEE
Transactions on Industrial Electronics 50(4), August 2003.

PdaDriver
,

a Personal Digital
Assistant (PDA) interface for
remote driving.


PdaDriver is designed to let any
user (novice or expert alike) to
remotely drive a mobile robot from
anywhere and at anytime.


Compaq iPAQ (PocketPC
2000/2002,
WindowsCE 3.0
)

802.11b (WiFi) wireless data link.


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

PDA
-
Based Interfaces

T. Fong, C. Thorpe, and C. Baur,
Multi
-
Robot Remote Driving with Collaborative Control
, IEEE
Transactions on Industrial Electronics 50(4), August 2003.

Displays
live images

from a camera
located on the robot.


The user can
pan and tilt the camera

by
clicking in the grey camera control box.


Yellow lines shown on the image indicate
the projected horizon line and robot width.


The user drives the robot by clicking a
series of waypoints on the image and then
pressing the
go

button. As the robot
moves, the
progress

bar displays the
robot's progress.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

PDA
-
Based Interfaces

T. Fong, C. Thorpe, and C. Baur,
Multi
-
Robot Remote Driving with Collaborative Control
, IEEE
Transactions on Industrial Electronics 50(4), August 2003.

Terrain visualization system
: transforms data collected from robot sensors into a
3D
graphical map
, which is continuously updated and displayed on the PDA.

Converts range data from a
Sick LMS
-
220 radar

into VRML using a line stripe classifier
and a regular quadrilateral mesh generator. Obstacles and the robot's path (traversed
and planned) are color coded.

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

PDA
-
Based Interfaces

Calinon, S.

and
Billard, A.

(2003)
PDA Interface for Humanoid Robots
. In Proceedings of the IEEE International
Conference on Humanoid Robots (Humanoids).

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

PDA
-
Based Interfaces

Calinon, S.

and
Billard, A.

(2003)
PDA Interface for Humanoid Robots
. In Proceedings of the IEEE International
Conference on Humanoid Robots (Humanoids).

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Other Promising Interfaces




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Tracking of Hand Motion and Facial Expression

Other modes of interactions currently in development but which may
become common in the future are:



Tracking of hand/finger motion either through vision or through
data gloves



Tracking and recognition of facial expressions

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Haptics

Haptics provides information in terms of
force

applied by the device
handled by the human demonstrator.


The forces are directly transmitted by the robotic device and can thus
be reapplied in an autonomous manner after the demonstration.

The Pantoscope for Laparoscopy Surgery Simulation (LSRO
-

EPFL), the Microsurgery Trainer
developed at EPFL and National UNiv. of Singapore (NUS).

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Brain
-
machine interfaces



Brain
-
computer interfaces recognize the subject’s voluntary
intent through EEG waves.





These can be interpret as states and be used to convey the
user’s intention to a robot on the order of milliseconds.

José del R. Millán, Frédéric Renkens, Josep Mouriño and Wulfram Gerstner, Noninvasive Brain
-
Actuated Control
of a Mobile Robot by Human EEG, IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 51, NO. 6,
JUNE 2004

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC




Major Issue in HRI




http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Safety in HRI

Existing Approaches in Industry

. Safeguarding [ANSI . RIA R15.06 . 1999, EN 775]

. Physical barriers

. Sensor Curtains

. Emergency Stop [ANSI . RIA R15.06 . 1999, EN 775]

. All power removed from manipulator

. Risk Assessment and Reduction [IEC 61508]

. Standard for electronic/electrical safety components

. Recognize that it may not be possible to design a failsafe system


try
to minimize risk

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Safety in HRI

Existing Approaches in Industry

Crash Tests (DLR + KUKA + ADAC)

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Existing Approaches in Research

. Safety through Mechanical Design

. Viscoelastic covering [Yamada et al. 1997]

. Spherical joints [Yamada et al. 1999]

. Compliant joints [Bicchi et al. 2001]

. Safety in the Control Loop

. Safeguarded zones [Bearveldt 1993]

. Robot moves at normal speed

. Robot moves at reduced speed

. Robot stops

. Evasive Robot [Traver et al. 2000]

. Danger Index [Ikuta et al. 1997]

http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Summary

DESIDERATA FOR THE INTERFACES:




The interfaces must be
user
-
friendly

and require as little prior
knowledge as possible.




They are meant for lay people, children or disabled people.




They must provide
“natural” interactions
, i.e. types of interactions
similar to those humans have with one another




Tradeoff between high accuracy (wearable instruments) and non
-
intrusive system




The interfaces and interactions must be
multimodal




The interfaces must be

safe


http://lasa.epfl.ch

A.G. Billard, Autonomous Robots Class


EDPR/EDIC

Summary




Modes of interactions

considered sofar are:




vision



proprioception



speech.




Types of interactions

considered sofar are:




recognition and detection of faces



recognition and detection of facial expressions,



tracking of body and hand motion,



measurement of forces



measurement of brain signals