Principal Component Analysis

Τεχνίτη Νοημοσύνη και Ρομποτική

17 Νοε 2013 (πριν από 4 χρόνια και 7 μήνες)

94 εμφανίσεις

Principal Component Analysis

and Applications in Image Analysis

Presented by Judy Jun Feng

Image Computing Group, Department of Computer Science

City University of Hong Kong

Outline

Introduction and History

Bivariate or two variables case

Mathematical derivation in multivariate case

Graphical representation of PCA

Application in Face Recognition (FR)

Application in Active Shape Models (ASM)

A useful deriving for computation in practice

Conclusion

Basic idea

One

of

the

multivariate

techniques

for

data

analysis

Main

purpose

is

to

reduce

the

dimension

of

the

data

set

and

retain

information

as

much

as

possible,

especially

retain

information

which

characterizes

the

most

important

variation

of

the

data
.

Transforming

the

original

data

to

a

new

set

of

uncorrelated

variables

(PCs)

History

1901: Karl Pearson
-----
First proposed

1933: Harold Hotelling
----
General procedures

1963: Anderson
----
Theoretical developments of variance of sample PCs

1963: Rao
----
New ideas about interpretation and extension

1966: Gower
----
The links between PCA and other statistic techniques and

geometric insights

1970’s
------
Popularly used in the variety of areas

1989: Esbensen
----
Multivariate Image Analysis

Introduction and History

Data Set:

Length and width of the similar shape objects in images: (
X
iL

X
iW

) (
i=1..n
)

Objective:

Find 2 uncorrelated components which are the normalized linear combinations of
X
L,
X
W

with maximum variance .

Procedures:

(1)
Transform the data set to the center:

(2)
Find two components:

with the constraint:

That is

PCA in Bivariate Case

LW
W
L
W
L
W
W
L
L
R
S
S
b
b
S
b
S
b
U
S
1
1
2
2
1
2
2
1
1
2
2
)
(

1
2
1
2
1

W
L
b
b
90
1
2

)
,
(
W
L
X
X
)
cos(
1
1

L
b
)
sin(
1
1

W
b
W
W
L
L
X
b
X
b
U
1
1
1

W
W
L
L
X
b
X
b
U
2
2
2

)
cos(
2
2

L
b
)
sin(
2
2

W
b

Statistic and Geometric meaning

Original Axes

Principal Components

Components

X
L

X
W

U
1

=
-
0.59
X
W

+0.81
X
L

U
2

=0.81
X
W

+0.59
X
L

Standard Deviation

0.81

0.63

0.95

0.39

Correlation Coefficient

-
0.69

0

Variance Proportion

62%

38%

86%

14%

)
/(
2
2
2
W
L
x
x
S
S
S
p

L
W
X
X
U
81
.
0
59
.
0
1

L
W
X
X
U
59
.
0
81
.
0
2

Transform each pair of (
X
L,

X
W
) to a pair (
U
1
,
U
2
), by:

Transform each pair of (U1, U2) to a pair (XL, XW), by:

2
1
59
.
0
81
.
0
U
U
X
L

2
1
81
.
0
59
.
0
U
U
X
W

Principal Components in the
multivariate case (1)

Data Set:

**
X=(X
1
, X
2
, X
3
…X
p
)

T

are vectors of p
-
dimensional random variables with
covariance matrix
C.

**
X

have been standardized to the same scales

Statistic background:

The symmetric matrix
C

is positive definite with eigenvalues .

Matrix formed by the eigenvectors is an orthogonal matrix

From the relations the covariance matrix
C

can be diagonalized to

And

0
...
2
1

p

]
[
i
e
T

I
T
T
T

p
T
CT
T

0
0
0
0
...
0
0
0
0
0
0
0
0
2
1
i
i
e
Ce

T
T
T
C

Principal Components in the
multivariate case (2)

If principal components are chosen to be :

Then

That means the covariance matrix of
U

has been diagonalized to

with the largest variance

So
U
i

is known as the
i
-
th principal component of the original variables and
U
i
can be

calculated from

And X can be reconstructed by the principle components:

X
T
U
or
X
e
U
T
T
i
i

j
i
for
j
i
for
Ce
e
x
e
x
e
Cov
U
U
Cov
i
j
T
i
T
j
T
i
j
i
0
)
,
(
)
,
(

CT
T
U
U
Cov
U
Cov
T
j
i
)
,
(
)
(
1
1
1
1
)
,
(
)
(

U
U
Cov
U
Var
X
e
U
T
i
i

U
e
X
i

Principal Components in the
multivariate case (3)

Total variance

Proportion variance

The cumulative proportion

is often calculated to find if
k

components have reproduced the data sufficiently well

Reduce the number of variables or dimension from p to k ( k<<p ).

p
i
p
i
i
i
total
S
S
1
1
2
2

2
/
total
i
S

2
1
/
)
(
total
k
i
i
S

Graphical representation of PCA

2
2
)
(
)
(
w
w
L
L
Euclid
x
x
x
x
D

2
2
2
2
2
2
2
2
)
(
)
)(
(
2
)
(
xy
y
x
x
xy
y
M
R
S
S
S
y
y
R
y
y
x
x
S
x
x
D

2
2
2
2
2
1
1
1
2

u
u
M
S
u
u
S
u
u
D
2
2
2
2
2
2
1
1
1
2
...

p
p
p
M
u
u
u
u
u
u
D

E=2.56

M=2.8

M=6.4

Mahalanobis Distance

Euclidean Distance

**The

ith

principal

component

lies

on

the

ith

longest

axes

of

ellipsoids
.

**All

of

the

points

on

the

contour

of

one

ellipse

or

on

the

surface

of

one

ellipsoid

with

the

orthogonal

axes

of

the

principal

components,

have

the

same

Mahalanobis

distance
.

Using PCA in the Face Recognition
(FR) (1)

Raw Faces

Eigen Faces

Faces: complex natural objects

with high dimension
.

Very difficult

to
develop a

computational model

Feature images decomposed

from face images

Speed, simplicity,

insensitivity to small or

Using PCA in the Face Recognition
(FR) (2)

Collect

a

number

of

face

images

as

the

training

set,

a

few

for

each

person
.

X

---

N

ⴭⴭ
a

ow

v散瑯e

N

2

1
.

The

eigenvectors

and

eigenvalues

can

be

found

from

the

covariance

matrix

C

from

the

face

vectors

of

those

images

.

Calculate

the

principal

components

from

linear

combinations
:

Use

principal

components

to

reconstruct

the

raw

image

We

call

these

eigenvectors

as

"eigenfaces"
.

The

reconstruction

can

also

be

viewed

as

the

combinations

of

eigenfaces

weighted

by

corresponding

values

of

PCs
.

In

practice,

each

individual

face

is

actually

approximated

by

only

a

few

k

eigenfaces

and

PCs

with

the

largest

eigenvalues
.

256

㈵2

8

†
㈰2

†

†††

2
2
N
N

X
e
U
T
i
i

U
e
X
i

Using PCA in the Face Recognition
(FR) (3)

Define a face class using a group of images of one person

Get a group of

k
-
d PC vectors

Defined the face class centered at the mean of these PC vectors.

Choose a distance threshold to define the maximum allowable distance .

Define a face space using all of the images in the training set

Get the PC vectors of all the face images in the training set

Find the mean and allowable distance to determine if an input image is a face
images

Face Recognition

A

new

face

is
:

(
1
)

A

known

people

:

if

it

belongs

to

a

face

class
.

(
2
)

An

unknown

people
:

to

training

set
.

(
3
)

Not

a

face

at

all
:

far

away

from

the

face

space
.

]
,...,
,
[
2
1
k
T
u
u
u
U

X
e
U
T
i
i

i
i
u
e
X

Applying PCA in the Active Shape
Models (1)

Flexible models, or deformable templates, can be used to allow for some
degree of variability in the shape of the image objects

PCA is often used to find some most important "mode of variation", and
helps to obtain a more compressed statistical description of the shapes and
their variation

Metacarpal shapes in X
-
ray images

Applying PCA in the Active Shape
Models (2)

Labeling and Aligning the shapes training set

Each shape has been described by a position vector( A points in 2p
-
d space )

Mean shape:

Get the most important modes of variation

Find the eigenvectors from the 2p

2瀠捯va物湣攠ma瑲楸

The position vectors can be reproduced by the eigenvectors weighted by the values of
PCs, and every eigenvectors accounts for a kind of mode of variation .

K largest eigenvalues and their eigenvectors, the most important modes of variations.

All of the PC vectors in the training set can consist of the "Allowable Shape Domain" .

A possible shape similar to the class of shapes can be expressed by those modes with
different weights in the “Allowable Shape Domain” .

)
,
,...
,
,
(
2
2
,
1
1
ip
ip
i
i
i
i
i
y
x
y
x
y
x
x

N
i
i
x
N
x
1
1

A useful deriving for computation in
practice

High computer load to find the eigenvectors in image analysis

Face Recognition,
256

256⁎
2

=65,536 And C = 65,536

65ⰵ,6

There are only M meaningful eigenvectors

( M is the number of the images in the training data)

Suppose
V

is the eigen vectors of

is the eigen vectors of

i
i
i
T
V
AV
A

i
i
i
T
AV
AV
AA

i
AV
T
M
i
T
i
i
AA
M
C

1
1
)
(
2
2
N
N

A
A
T
)
(
M
M

T
AA
C

I
]
,...,
,
[
2
1
M
A

)
1
(
2

N
)
(
2
M
N

M
k
k
ik
i
i
V
AV
U
1

Conclusion

PCA

has

been

proved

an

effective

approach

for

image

recognition,

segmentation

and

other

kind

of

analysis
.

(1)Extract uncorrelated basic feature sets to describe the properties of a data set.

(2)Reduce the dimensionality of the original image space.

(1) Provide little quantitative information and visualization implications

(2)
No way to discriminate between variance due to object of interest and variance
due to noise or background.

Researchers proposed different schemes and make many improvements for
better applying PCA in image analysis.