Fingerspelling Recognition Using sub-gesture Hidden Markov Model

loutclankedAI and Robotics

Nov 13, 2013 (3 years and 10 months ago)

89 views

Experimental Results

Abstract

As

one

of

the

most

important

components

of

American

Sign

Language

(ASL),

fingerspelling

is

widely

used

for

education

and

communication

among

signers
.

In

this

poster,

we

propose

a

new

static

fingerspelling

gesture

recognition

framework

based

on

a

left

to

right

Hidden

Markov

model

(HMM
)
.

Overlapping

frames

are

segmented

from

each

image,

and

then

Histogram

of

Oriented

Gradient

(
HOG)

features

are

extracted
.

Each

image

is

modeled

with

sub
-
gestures

which

also

allow

for

precise

modeling

of

background

noise
.

The

focus

of

this

work

is

to

investigate

two

main

components

for

the

configuration

of

this

system
:

the

number

of

sub
-
gestures

and

the

number

of

states

in

the

HMMs
.

Moreover,

signer
-
dependent

and

signer
-
independent

performances

of

the

system

are

compared
.

Our

singer
-
independent

system

outperforms

our

baseline

system

by

12
.
5
%
.

Shuang Lu and Dr. Joseph Picone

Department of Electrical and Computer Engineering, Temple
University

Fingerspelling Recognition Using sub
-
gesture

Hidden Markov Model



College
of
Engineering

Temple
University

Fingerspelling
Recognition
Systems






Sub
-
gesture Hidden Markov Model



Difficulties and Future
W
ork






Conclusions

As

can

be

seen

from

the

above

results
,

the

proposed

sub
-
gesture

HMM

approach

obtained

good

performance
.

Different

n
umber

of

states

for

sub
-
gestures

and

background

noise

models

were

tested,

which

decided

the

best

configurations

of

the

left
-
right

HMM
.

By

using

overlapping

frames

and

HOG

feature,

our

signer
-
independent

test

results

is

12
.
5
%

better

than

the

existing

baseline

system
.

In

order

to

achieve

better

Future

work

will

focus

on

improving

HMM

topologies

and

finding

a

better

way

classify

similar

gestures
.



References


[
1
]

Pugeault
,

N
.
,

&

Bowden,

R
.

(
2011
)
.

Spelling

It

Out
:

Real
-
time

ASL

Fingerspelling

Recognition
.

Proceedings

of

the

IEEE

International

Conference

on

Computer

Vision

Workshops

(pp
.

1114
-
1119
)
.


[
3
]

Thangali
,

A
.
,

Nash,

J
.
,

Sclaroff
,

S
.
,

&

Neidle
,

C
.

(
2011
)
.

Exploiting

Phonological

Constraints

for

Handshape

Inference

in

ASL

Video
.

Proceedings

of

the

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

(pp
.

521
-
528
)
.





Applications







www.isip.piconepress.com


Education


Entertainment


Communication


Robotic control


Human computer
interaction

0.0
3.0
6.0
9.0
12.0
15.0
18.0
1
2
3
4
5
6
7
8
9
10
Avg
Subject1
Subject2
Subject3
Subject4
Subject5
Signer dependent test

Different signer variation

Big angle rotations

Best signer dependent average
error rate 10.9
%

Best signer independent average error rate 52.4%

Data: 24 fingerspelling gestures
;


approximate
48000
samples
;



different
background.

0.0
20.0
40.0
60.0
80.0
7/11/11/1
7/13/9/1
7/13/11/1
7/13/11/3
7/15/11/1
9/13/11/1
11/13/11/1
13/13/11/1
13/15/11/1
Subject 1
Subject 2
Subject 3
Subject 4
Subject 5
Signer
independent test


Sub
-
gesture HMM Topology

Model parameters

Left
-
Right
models

Grammar : LB
-
> gesture
-
> LB



(LB: long background )

Histogram of oriented gradient feature

Best system configuration:

Numbers of sub
-
gesture : 11

HMM order for sub
-
gesture : 13

Long background order : 11

Small background order : 1

Frame and window size: 5/ 30

HOG feature bin number: 9

Gaussian Mixture: 16

Robust to
variations

With background modeling

Modify HMM Topology

Continuous fingerspelling
recognition

Better classify similar gestures

Similar gestures
according to confusion matrix

Half training/ half testing open
-
loop
signer independent test:

Best error rate 22%