Fingerspelling Recognition Using sub-gesture Hidden Markov Model

loutclankedΤεχνίτη Νοημοσύνη και Ρομποτική

13 Νοε 2013 (πριν από 3 χρόνια και 4 μήνες)

60 εμφανίσεις

Experimental Results

Abstract

As

one

of

the

most

important

components

of

American

Sign

Language

(ASL),

fingerspelling

is

widely

used

for

education

and

communication

among

signers
.

In

this

poster,

we

propose

a

new

static

fingerspelling

gesture

recognition

framework

based

on

a

left

to

right

Hidden

Markov

model

(HMM
)
.

Overlapping

frames

are

segmented

from

each

image,

and

then

Histogram

of

Oriented

Gradient

(
HOG)

features

are

extracted
.

Each

image

is

modeled

with

sub
-
gestures

which

also

allow

for

precise

modeling

of

background

noise
.

The

focus

of

this

work

is

to

investigate

two

main

components

for

the

configuration

of

this

system
:

the

number

of

sub
-
gestures

and

the

number

of

states

in

the

HMMs
.

Moreover,

signer
-
dependent

and

signer
-
independent

performances

of

the

system

are

compared
.

Our

singer
-
independent

system

outperforms

our

baseline

system

by

12
.
5
%
.

Shuang Lu and Dr. Joseph Picone

Department of Electrical and Computer Engineering, Temple
University

Fingerspelling Recognition Using sub
-
gesture

Hidden Markov Model



College
of
Engineering

Temple
University

Fingerspelling
Recognition
Systems






Sub
-
gesture Hidden Markov Model



Difficulties and Future
W
ork






Conclusions

As

can

be

seen

from

the

above

results
,

the

proposed

sub
-
gesture

HMM

approach

obtained

good

performance
.

Different

n
umber

of

states

for

sub
-
gestures

and

background

noise

models

were

tested,

which

decided

the

best

configurations

of

the

left
-
right

HMM
.

By

using

overlapping

frames

and

HOG

feature,

our

signer
-
independent

test

results

is

12
.
5
%

better

than

the

existing

baseline

system
.

In

order

to

achieve

better

Future

work

will

focus

on

improving

HMM

topologies

and

finding

a

better

way

classify

similar

gestures
.



References


[
1
]

Pugeault
,

N
.
,

&

Bowden,

R
.

(
2011
)
.

Spelling

It

Out
:

Real
-
time

ASL

Fingerspelling

Recognition
.

Proceedings

of

the

IEEE

International

Conference

on

Computer

Vision

Workshops

(pp
.

1114
-
1119
)
.


[
3
]

Thangali
,

A
.
,

Nash,

J
.
,

Sclaroff
,

S
.
,

&

Neidle
,

C
.

(
2011
)
.

Exploiting

Phonological

Constraints

for

Handshape

Inference

in

ASL

Video
.

Proceedings

of

the

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

(pp
.

521
-
528
)
.





Applications







www.isip.piconepress.com


Education


Entertainment


Communication


Robotic control


Human computer
interaction

0.0
3.0
6.0
9.0
12.0
15.0
18.0
1
2
3
4
5
6
7
8
9
10
Avg
Subject1
Subject2
Subject3
Subject4
Subject5
Signer dependent test

Different signer variation

Big angle rotations

Best signer dependent average
error rate 10.9
%

Best signer independent average error rate 52.4%

Data: 24 fingerspelling gestures
;


approximate
48000
samples
;



different
background.

0.0
20.0
40.0
60.0
80.0
7/11/11/1
7/13/9/1
7/13/11/1
7/13/11/3
7/15/11/1
9/13/11/1
11/13/11/1
13/13/11/1
13/15/11/1
Subject 1
Subject 2
Subject 3
Subject 4
Subject 5
Signer
independent test


Sub
-
gesture HMM Topology

Model parameters

Left
-
Right
models

Grammar : LB
-
> gesture
-
> LB



(LB: long background )

Histogram of oriented gradient feature

Best system configuration:

Numbers of sub
-
gesture : 11

HMM order for sub
-
gesture : 13

Long background order : 11

Small background order : 1

Frame and window size: 5/ 30

HOG feature bin number: 9

Gaussian Mixture: 16

Robust to
variations

With background modeling

Modify HMM Topology

Continuous fingerspelling
recognition

Better classify similar gestures

Similar gestures
according to confusion matrix

Half training/ half testing open
-
loop
signer independent test:

Best error rate 22%