An Infant Facial Expression Recognition SystemBased on Moment Feature Extraction

paraderollAI and Robotics

Nov 17, 2013 (3 years and 4 months ago)

54 views

An Infant Facial Expression Recognition System

Based on Moment Feature Extraction

C. Y. Fang, H. W. Lin,
S. W. Chen



Department of Computer Science and Information Engineering

National Taiwan Normal University

Taipei, Taiwan

1

Outline


Introduction


System Flowchart


Infant Face Detection


Feature Extraction


Correlation Coefficient Calculation


Infant Facial Expression Classification


Experimental Results


Conclusions and Future Work

2

Introduction



Infants can not protect themselves generally.


Vision
-
based surveillance systems can be used for infant care.


Warn the baby sitter


Avoid dangerous situations


This paper presents a vision
-
based infant facial expression recognition
system for infant safety surveillance.


camera

3

The classes of infant expressions


Five

infant

facial

expressions
:


crying,

gazing,

laughing,

yawning

and

vomiting



Three

poses

of

the

infant

head
:


front,

turn

left

and

turn

right


Total

classes
:

15

classes

crying

gazing

laughing

yawning

vomiting

front

turn right

turn left

4

System Flowchart


Infant

face

detection
:


to

remove

the

noises

and

to

reduce

the

effects

of

lights

and

shadows


to

segment

the

image

based

on

the

skin

color

information


Feature

extraction
:



to

extract

three

types

of

moments

as

features,

including

Hu

moments,

R

moments,

and

Zernike

moments



Feature

correlation

calculation
:



to

calculate

the

correlation

coefficients

between

two

moments

of

the

same

type

for

each

15
-
frame

sequence



Classification
:



to

construct

the

decision

trees

to

classify

the

infant

facial

expressions

I
n
f
a
n
t

F
a
c
e

D
e
t
e
c
t
i
o
n
F
e
a
t
u
r
e

E
x
t
r
a
c
t
i
o
n
C
l
a
s
s
i
f
i
c
a
t
i
o
n
F
e
a
t
u
r
e

C
o
r
r
e
l
a
t
i
o
n
C
a
l
c
u
l
a
t
i
o
n
I
m
a
g
e

S
e
q
u
e
n
c
e
s
5

Infant Face Detection


Lighting compensation


To make the skin color detection correctly


Infant face extraction


Step1: Skin color detection


Using three
bands



S

of
HSI



Cb

of
YCrCb


U

of
LUX


Step2: Noise reduction


Using 10x10 median filter


Step3: Infant face identification


Using temporal information


Lighting
compensation

Skin color
detection

Noise
reduction

6

Infant Face Detection


Step 3: Infant face identification


1

t
I
t
I
1

Case
2

Case
1

t
I
t
I
t
I
t
I
7

Moments


To calculate three types of moments


Hu moment [Hu1962]


R moment [Liu2008]


Zernike moment [Zhi2008]


Given an image
I

and let
f

be an image function. The digital (
p
,
q
)th moment
of
I

is given by




The central (
p
,
q
)th moments of
I
can be defined as


where and


The normalized central moments of
I



where

.
)
,
(

)
(
)
,
(



I
y
x
q
p
pq
y
x
f
y
x
I
m
).
,
(
)
(
)
(

)
(
)
,
(
0
0
y
x
f
y
y
x
x
I
I
y
x
q
p
pq










00
pq
pq

00
10
0
m
m
x

00
01
0
m
m
y

1
2



q
p

8

Hu

Moment


Hu moments are translation, scale, and rotation invariant.

]
)
(
)
(
3
)[
)(
3
(

]
)
(
3
)
)[(
)(
3
(
)
)(
(
4

]
)
(
)
)[(
(
]
)
(
)
(
3
)[
)(
3
(

]
)
(
3
)
)[(
)(
3
(
)
(
)
(
)
3
(
)
3
(
4
)
(
2
03
21
2
12
30
03
21
30
12
2
03
21
2
12
30
12
30
03
21
7
03
21
12
30
11
2
03
21
2
12
30
02
20
6
2
03
21
2
12
30
03
21
03
21
2
03
21
2
12
30
12
30
12
30
5
2
03
21
2
12
30
4
2
03
21
2
12
30
3
2
11
2
02
20
2
02
20
1





































































































H
H
H
H
H
H
H
normalized central moments

9

Example: Hu Moments

crying

10

Example: Hu Moments

yawning

11

Example: Hu Moments

yawning

crying


If the infant facial expressions are different then the values of
Hu

moments
are also different.

12

R Moment


Liu (2008) proposed ten R moments which can improve the scale
invariability of Hu moments.

4
3
5
10
5
2
6
9
2
3
6
8
5
1
6
7
3
1
6
6
5
4
5
5
3
4
4
3
3
2
1
2
1
2
1
2
1
|
|

|
|
|
|

|
|
|
|
|
|


|
|

|
|
|
|




H
H
H
R
H
H
H
R
H
H
H
R
H
H
H
R
H
H
H
R
H
H
R
H
H
R
H
H
R
H
H
H
H
R
H
H
R

















Hu moments

13

Example: R Moments

crying

Hu moments


R moments and
Hu

moments may have different properties.

14

Zernike Moment


Zernike moments of order
p

with repetition
q

for an image
function
f

is




where






To simplify the index, we use
Z
1
,
Z
2
,…,
Z
10

to represent
Z
80
,
Z
82
,…,
Z
99
, respectively.

)
,
(
4
cos
)
/
2
(
2
2

2
/
1
8
1
2
v
s
f
s
qv
N
s
R
N
p
C
N
s
s
v
pq
pq







)
,
(
4
sin
)
/
2
(
2
2

2
/
1
8
1
2
v
s
f
s
qv
N
s
R
N
p
S
N
s
s
v
pq
pq







2
/
1
2
2
)
(
pq
pq
pq
S
C
Z






s
p
q
p
s
pq
r
q
s
p
q
s
p
s
s
p
r
R
2
2
/
|)
|
(
0

!

2
/
|
|
2

!

2
/
|
|
2

!
)!
(
)
1
(
)
(











real part

imaginary part

15

Example:
Zernike

Moments

crying

16

Correlation Coefficients


A facial expression is a sequential change of the values of the moments.


The correlation coefficients of two moments may be used to represent the
facial expressions.


Let

A
i

= ,

i

= 1, 2,…,
m
,


indicates the
i
th

moment
A
i

of the frame
I
k
, k =
1, 2,…,
n
.


The correlation coefficients between
A
i

and
A
j

can be defined as






where and

j
i
j
i
j
i
S
S
S
r
A
A
A
A
A
A






n
k
i
iI
A
A
n
S
k
i
1
2
2
)
(
1
1
A






n
k
j
jI
i
iI
A
A
A
A
n
S
k
k
j
i
1
)]
)(
[(
1
1
A
A
i
A
: the mean of the elements in
A
i


}
,...,
,
{
2
1
n
iI
iI
iI
A
A
A
k
iI
A
17

Correlation Coefficients


The correlation coefficients between seven Hu moment sequences.

H
1

H
2

H
3

H
4

H
5

H
6

H
7

H
1

1

0.8778

0.9481

-
0.033

-
0.571

-
0.8052

0.8907

H
2

1

0.9474

0.1887

-
0.4389

-
0.8749

0.9241

H
3

1

0.1410

-
0.6336

-
0.9044

0.9719

H
4

1

0.0568

-
0.3431

0.2995

H
5

1

0.7138

-
0.6869

H
6

1

-
0.9727

H
7

1

yawning

Decision Tree


Decision trees are used to classify the infant facial expressions.

18



H
1
H
2

H
1
H
3

H
2
H
3

-

+

+

+

+

-

+

+

+

+

+

-

-

+

+

-

-

+

-

-

-

+

-

-

-

-

-

+

-

+

H
1
H
3
>0

Yes
No
correlation coefficients

19

Decision Tree


The correlation coefficients between two attributes

A
i

and
A
j

are used to
split the training instances.


Let the training instances in

S

be split into two subsets
S
1

and
S
2

by the
correlation coefficient, then
the measure function
is





The best correlation coefficient selected by the system is









K
h
K
h
S
h
S
S
h
S
S
h
S
S
h
S
r
N
N
N
N
N
N
N
N
S
E
j
i
1
1
2
2
,
log
log
)
(
2
2
2
2
1
1
1
1
A
A
)
(
min
arg
)
(

,
*
*
S
E
S
r
j
i
j
i
r
j
i
A
A
A
A

20

Decision tree construction

Step 1:
Initially, put all the training instances into the root
S
R
, regard
S
R

as an internal
decision node and input
S
R

into a decision node queue.


Step 2:
Select an internal decision node
S

from the decision node queue calculate the
entropy of node
S
.


If the entropy of node
S

larger then a threshold
T
s
, then
goto

Step 3, else label
node
S

as a leaf node,
goto

Step 4.


Step 3:
Find the best correlation coefficient to split the training instances in node
S
.


Split the training instances in

S

into two nodes
S
1

and
S
2

by correlation
coefficients and add
S
1
,
S
2

into the decision node queue.
Goto

Step 2.


Step 4:

If the queue is not empty, then
goto

Step 2, else stop the algorithm.

21

Experimental Results


Training: 59 sequences


Testing: 30 sequences


Five infant facial expressions: crying, laughing, dazing, yawning, vomiting


Three different poses of infant head: front, turn left, and turn right


Fifteen classes are classified.

crying

laughing

dazing

yawning

vomiting

Turn

left

Front

Turn
right

Feature type:
Hu

moments

Internal nodes: 16

Leaf nodes: 17

Height: 8


0
7
6

H
H
r
H
S
9
yes

no

0
4
1

H
H
r
H
S
10
crying

yes

no

0
6
2

H
H
r
H
S
11
yawning

yes

no

0
6
1

H
H
r
H
S
12
crying

yes

no

yawning

H
S
13
0
7
5

H
H
r
0
2
1

H
H
r
H
S
16
yes

no

H
S
14
0
4
3

H
H
r
no

laughing

no

dazing

dazing

yes

yes

H
S
15
0
3
1

H
H
r
no

crying

dazing

yes

H
S
2
yes

no

0
7
6

H
H
r
H
S
3
0
6
1

H
H
r
H
S
6
yes

no

0
6
3

H
H
r
H
S
5
vomiting

yes

no

0
6
3

H
H
r
H
S
5
vomiting

yes

no

laughing

vomiting

yes

no

0
5
1

H
H
r
H
S
7
yawning

yes

no

0
3
1

H
H
r
H
S
8
crying

yes

no

laughing

crying

yes

no

H
S
1
0
5
4

H
H
r
0
5
3

H
H
r
Experimental Results

Testing sequences

Classification results

23

laughing

laughing

dazing

vomiting


The classification results of the
Hu
-
moment decision tree

24

Feature type: R
moments

Internal nodes: 15

Leaf nodes: 17

Height: 10

no

yes

vomiting

yes

no

0
7
5

R
R
r
yawning

no

0
2
1

R
R
r
yes

0
7
4

R
R
r
yes

0
10
5

R
R
r
dazing

no

crying

vomiting

yes

no

0
6
4

R
R
r
laughing

no

0
1
4

R
R
r
yes

no

0
10
1

R
R
r
yes

no

0
10
5

R
R
r
yes

0
2
1

R
R
r
yes

no

crying

dazing

laughing

no

0
5
1

R
R
r
yes

no

0
4
1

R
R
r
yes

0
8
1

R
R
r
yes

no

dazing

crying

vomiting

no

0
10
9

R
R
r
yes

0
7
1

R
R
r
0
6
1

R
R
r
0
3
1

R
R
r
yes

yes

no

laughing

dazing

no

vomiting

yes

no

laughing

crying

25

Experimental Results


Testing sequences

Classification results

vomiting

dazing

yawn
ing

dazing


The classification results of the R
-
moment decision tree

Feature type: Zernike
moments

Internal nodes: 19

Leaf nodes: 20

Height: 7

no

yes

no

0
6
2

Z
Z
r
no

0
8
4

Z
Z
r
yes

0
6
3

Z
Z
r
yes

crying

vomiting

laughing

dazing

yawning

yes

no

0
10
9

Z
Z
r
yes

no

0
7
5

Z
Z
r
no

0
7
6

Z
Z
r
yes

yes

no

0
10
7

Z
Z
r
crying

yes

no

0
7
6

Z
Z
r
no

0
7
1

Z
Z
r
yes

crying

no

0
4
1

Z
Z
r
yes

no

0
5
1

Z
Z
r
yes

no

0
7
1

Z
Z
r
yes

crying

laughing

0
2
1

Z
Z
r
yes

no

0
8
1

Z
Z
r
vomiting

yes

no

0
3
1

Z
Z
r
crying

yes

no

0
4
1

Z
Z
r
dazing

yawning

yes

no

laughing

dazing

dazing

0
3
2

Z
Z
r
yes

0
2
1

Z
Z
r
no

vomiting

yes

no

laughing

dazing

no

0
9
1

Z
Z
r
yawning

yes

27

Experimental Results


Testing sequences

Classification results

crying

crying

vomiting

crying


The classification results of the Zernike
-
moment decision tree

28

Conclusions


The comparison of the results








The correlation coefficients of the moments are useful attributes to classify the
infant facial expressions.


The classification tree created by the
Hu

moments has less height and number
of node, but higher classification rate.







Height of
the decision
tree

Number
of nodes

Number of
training
sequences

Number of
testing
sequences

Classification
Rate

Hu

moments

8

16+17

59

30

90%

R
moments

10

15+17

59

30

80%

Zernike
moments

7

19+20

59

30

87%

29

Conclusions and Future Work


Conclusion


A vision
-
based infant facial expression recognition system


Infant face detection


Moment features extraction


Correlation coefficient calculation


Decision tree classification


Future work


To collect more experimental data


To
fuzzify

the decision tree


Binary decision trees may have less noise tolerant ability.


If the correlation coefficients are close to zero, the noises will greatly affect the
classification results.

30