On-line Human-Robot Interaction by utilizing Dynamic Neural Network

bouncerarcheryAI and Robotics

Nov 14, 2013 (3 years and 9 months ago)

70 views

On
-
line Human
-
Robot Interaction

by utilizing Dynamic Neural Network

Contents

1.
Goal

2.
Method and materials


-

Data set


-

Network structure


-

Training

3. Experiment and analysis


-

Simulation


-

Real
-
time test with robot

4. Discussion

5. Research plan.

Goal

1.
On
-
line Recognition of Human movement gestures.


2.
Generation of contextual Robot re
-
Actions.


3.
Understanding the network.


4.
Development of the system.

Materials and Method

Data set



Sensor : Human gesture, command :

2
-
D human Kinect skeleton tracked hand position.



Motor : Robot action :


4
-
D robot motor (Right, Left shoulder pitch, roll)



angle.




3 human gesture
-
robot action pairs,


3 human commands (stop action, start action, repeat action)




Data set



Materials

and Method

Human gesture

Robot action

Human command

2
-
D

x,y

human hand movement.

4
-
D

Right, Left shoulder pitch, roll
motor angle

2
-
D

x,y

human hand movement?

or human hand position?

Data set




Materials

and Method

Repeat command

Data set




Materials

and Method

Repeat command

System structure for Human
-
Robot interaction




RNN

# of Sensor Dimension : 2 (x, y human hand position)

# of Motor Dimension : 4 (Right, Left shoulder pitch, roll)

# of
Softmax

vector size for each sensor, motor dimension : 11




# of :

# of :

# of
:

# of
:


Materials

and Method

Varies depending on

Experiments


Learning method :



Supervised learning.

Learning rule :


Update network parameter with gradient descent to achieve



one of below goals.



1. Maximize likelihood :




2. Error minimization :







3. Minimize mutual information :


(KL
-
divergence)




Materials

and Method

t
anh

Activation function

Softmax

activation function

Materials

and Method

Learning rule

Why learning with
softmax

transformed data and minimize mutual information

gives best result?

1.Limitation of Maximize likelihood case’s assumption?

Human movement’s noise is not Gaussian noise.

Mixture of Gaussian, 1/fb noise and Brownian motion. (Walk)


However, even function generated without noise data case also better

when we use
softmax

transform.

2. Each pattern
b
ecome more easy to



be separately recognized by expanding dimension?


1 input, output dimension
-
> 11
softmax

input, output dimension



then, why we can’t successfully train using error minimization rule.

Materials and Method

Result example

Sensor

Motor

2nd

Layer

3rd

Layer

Human
gesture recognition
phase

Robot action generation phase

2nd layer and 3rd layer contexts
performs as a switch
.

Different action makes
different
switching
signal

and

generates different reaction.

Motor contexts are
steady while human acts
, and
start to oscillate after human
stopped to act
.

In sensor cases, vice versa

Experiment

Simulation


Training with Segmented data



-
Emotional command



-
Repeat command



Real
-
robot experiment


Training with Segmented data

and repeat command.

Experiment

1
-
1 Training with segmented data : Emotional command

Stop Command speed

fast

Slow

Respond quickly

Respond slowly

Experiment

1
-
1 Training with segmented data : Emotional command

Experiment

1
-
2 Training with segmented data : Repeat command

Training data

Experiment

1
-
2 Training with segmented data : Repeat command

Training data

Experiment

1
-
2 Training with segmented data : Repeat command

Result :

Action
-
reaction pair 1 with repeat (Training data)

Experiment

1
-
2 Training with segmented data : Repeat command

Result :

Action
-
reaction pair 2 with repeat (Training data)

Experiment

1
-
2 Training with segmented data : Repeat command

Result :

Action
-
reaction pair 2 with repeat (Training data)

Experiment

1
-
2 Training with segmented data : Repeat command

Result :

All Action
-
reaction pair with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command (Test data)

L

A

Y

E

R

PATTERN

1
st

layer Context PC 1:2 phase plot action 1 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

2
nd

layer context PC 1:2 phase plot action 1 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

3
rd

layer context PC 1:2 phase plot action 1 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

1
st

layer Context PC 1:2 phase plot action 2 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

2
nd

layer context PC 1:2 phase plot action 2 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

3
rd

layer context PC 1:2 phase plot action 2 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

1
st

layer Context PC 1:2 phase plot action 3 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

2
nd

layer context PC 1:2 phase plot action 3 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

3
rd

layer context PC 1:2 phase plot action 1 with repeat (Test data)

Experiment

1
-
2 Training with segmented data : Repeat command

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 1 without repeat (training and test data)


















1
st

Layer


















2
nd

Layer


















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 1 with repeat (training and test data)


















1
st

Layer


















2
nd

Layer


















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 2 without repeat (training and test data)


















1
st

Layer


















2
nd

Layer


















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 2 with repeat (training and test data)


















1
st

Layer


















2
nd

Layer


















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 3 without repeat (training and test data)

















1
st

Layer

















2
nd

Layer

















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

PCA context Analysis :

All trials for Action
-
reaction pair 3 with repeat (training and test data)


















1
st

Layer


















2
nd

Layer


















3
rd

Layer

Experiment

1
-
2 Training with segmented data : Repeat command

Experiment

Real robot test video: Repeat command


Basics


Recognizing user’s command and respond appropriately.
(Table 1.)


Generating different actions according to the user’s different gesture
(e.g. waving)
.

User

Robot

t

Up

Gesture

Down

Middle

User’s

Hand
Position

Context

Desired

Robot’s Action

Up

Always

Go to the initial position and get ready to

recognize

user’s
gesture.

Middle

The robot is moving.

Pose

the initial posture, observe

user’s command.

Down

The user

previously rendered a
gesture.

Generate

the corresponding action. (Purple boxes at the
bottom)

User’s previous hand position
was middle.

Repeat

the previous action. (Orange boxed at the bottom)

Up

Recognizing

user’s
gesture

Recognition

Generating

the
corresponding actions

Action Generation

Recognizing

the command

Recognition

Repeating

the
corresponding actions

Action Generation

Waiting
a

user’s
gesture

Recognition

Down

Table 1. Types of command

Experiment

Real robot test video: Repeat command

Discussion

Role of higher level contexts

Higher level contexts order the primitives?



works as switch? commander?

Or just symbol represents contextual states?



Every decision done by the complex interaction?



Not just by someone’s command?

Discussion

Weight analysis

Thank you