Principal Component Analysis

parathyroidsanchovyAI and Robotics

Nov 17, 2013 (3 years and 8 months ago)

70 views

Principal Component Analysis

and Applications in Image Analysis


Presented by Judy Jun Feng

Image Computing Group, Department of Computer Science

City University of Hong Kong

Image Computing Group. All rights reserved 2002.

Outline


Introduction and History


Bivariate or two variables case


Mathematical derivation in multivariate case


Graphical representation of PCA


Application in Face Recognition (FR)


Application in Active Shape Models (ASM)


A useful deriving for computation in practice


Conclusion

Image Computing Group. All rights reserved 2002.


Basic idea


One

of

the

multivariate

techniques

for

data

analysis


Main

purpose

is

to

reduce

the

dimension

of

the

data

set

and

retain

information

as

much

as

possible,

especially

retain

information

which

characterizes

the

most

important

variation

of

the

data
.


Transforming

the

original

data

to

a

new

set

of

uncorrelated

variables

(PCs)


History


1901: Karl Pearson
-----
First proposed


1933: Harold Hotelling
----
General procedures


1963: Anderson
----
Theoretical developments of variance of sample PCs


1963: Rao
----
New ideas about interpretation and extension


1966: Gower
----
The links between PCA and other statistic techniques and


geometric insights


1970’s
------
Popularly used in the variety of areas


1989: Esbensen
----
Multivariate Image Analysis

Introduction and History

Image Computing Group. All rights reserved 2002.



Data Set:

Length and width of the similar shape objects in images: (
X
iL

X
iW

) (
i=1..n
)



Objective:

Find 2 uncorrelated components which are the normalized linear combinations of
X
L,
X
W

with maximum variance .



Procedures:

(1)
Transform the data set to the center:

(2)
Find two components:


with the constraint:


That is

PCA in Bivariate Case

LW
W
L
W
L
W
W
L
L
R
S
S
b
b
S
b
S
b
U
S
1
1
2
2
1
2
2
1
1
2
2
)
(



1
2
1
2
1


W
L
b
b
90
1
2






)
,
(
W
L
X
X
)
cos(
1
1


L
b
)
sin(
1
1


W
b
W
W
L
L
X
b
X
b
U
1
1
1


W
W
L
L
X
b
X
b
U
2
2
2


)
cos(
2
2


L
b
)
sin(
2
2


W
b
Image Computing Group. All rights reserved 2002.

Statistic and Geometric meaning

Original Axes

Principal Components

Components

X
L

X
W

U
1

=
-
0.59
X
W

+0.81
X
L

U
2

=0.81
X
W

+0.59
X
L

Standard Deviation

0.81

0.63

0.95

0.39

Correlation Coefficient

-
0.69

0

Variance Proportion

62%

38%

86%

14%

)
/(
2
2
2
W
L
x
x
S
S
S
p


L
W
X
X
U
81
.
0
59
.
0
1



L
W
X
X
U
59
.
0
81
.
0
2


Transform each pair of (
X
L,

X
W
) to a pair (
U
1
,
U
2
), by:

Transform each pair of (U1, U2) to a pair (XL, XW), by:

2
1
59
.
0
81
.
0
U
U
X
L


2
1
81
.
0
59
.
0
U
U
X
W


Image Computing Group. All rights reserved 2002.

Principal Components in the
multivariate case (1)




Data Set:


**
X=(X
1
, X
2
, X
3
…X
p
)

T

are vectors of p
-
dimensional random variables with
covariance matrix
C.


**
X

have been standardized to the same scales



Statistic background:

The symmetric matrix
C

is positive definite with eigenvalues .


Matrix formed by the eigenvectors is an orthogonal matrix


From the relations the covariance matrix
C

can be diagonalized to





And

0
...
2
1




p



]
[
i
e
T

I
T
T
T


















p
T
CT
T



0
0
0
0
...
0
0
0
0
0
0
0
0
2
1
i
i
e
Ce


T
T
T
C


Image Computing Group. All rights reserved 2002.

Principal Components in the
multivariate case (2)



If principal components are chosen to be :


Then



That means the covariance matrix of
U

has been diagonalized to




with the largest variance

So
U
i

is known as the
i
-
th principal component of the original variables and
U
i
can be

calculated from

And X can be reconstructed by the principle components:

X
T
U
or
X
e
U
T
T
i
i










j
i
for
j
i
for
Ce
e
x
e
x
e
Cov
U
U
Cov
i
j
T
i
T
j
T
i
j
i
0
)
,
(
)
,
(





CT
T
U
U
Cov
U
Cov
T
j
i
)
,
(
)
(
1
1
1
1
)
,
(
)
(



U
U
Cov
U
Var
X
e
U
T
i
i

U
e
X
i

Image Computing Group. All rights reserved 2002.

Principal Components in the
multivariate case (3)




Total variance




Proportion variance





The cumulative proportion


is often calculated to find if
k

components have reproduced the data sufficiently well

Reduce the number of variables or dimension from p to k ( k<<p ).








p
i
p
i
i
i
total
S
S
1
1
2
2

2
/
total
i
S

2
1
/
)
(
total
k
i
i
S



Image Computing Group. All rights reserved 2002.

Graphical representation of PCA


2
2
)
(
)
(
w
w
L
L
Euclid
x
x
x
x
D






2
2
2
2
2
2
2
2
)
(
)
)(
(
2
)
(
xy
y
x
x
xy
y
M
R
S
S
S
y
y
R
y
y
x
x
S
x
x
D












2
2
2
2
2
1
1
1
2


























u
u
M
S
u
u
S
u
u
D
2
2
2
2
2
2
1
1
1
2
...







































p
p
p
M
u
u
u
u
u
u
D



E=2.56

M=2.8

M=6.4

Mahalanobis Distance

Euclidean Distance


**The

ith

principal

component

lies

on

the

ith

longest

axes

of

ellipsoids
.



**All

of

the

points

on

the

contour

of

one

ellipse

or

on

the

surface

of

one

ellipsoid

with

the

orthogonal

axes

of

the

principal

components,

have

the

same

Mahalanobis

distance
.

Image Computing Group. All rights reserved 2002.

Using PCA in the Face Recognition
(FR) (1)

Raw Faces

Eigen Faces

Faces: complex natural objects


with high dimension
.

Very difficult

to
develop a

computational model

Feature images decomposed

from face images

Speed, simplicity,

insensitivity to small or

gradual changes of the faces.


Image Computing Group. All rights reserved 2002.

Using PCA in the Face Recognition
(FR) (2)


Collect

a

number

of

face

images

as

the

training

set,

a

few

for

each

person
.

X

---

N



ⴭⴭ
a

ow

晡捥

v散瑯e



摩d敮獩en

N

2


1
.



The

eigenvectors

and

eigenvalues

can

be

found

from

the

covariance

matrix

C


from

the

face

vectors

of

those

images

.


Calculate

the

principal

components

from

linear

combinations
:


Use

principal

components

to

reconstruct

the

raw

image


We

call

these

eigenvectors

as

"eigenfaces"
.



The

reconstruction

can

also

be

viewed

as

the

combinations

of

eigenfaces

weighted

by

corresponding

values

of

PCs
.


In

practice,

each

individual

face

is

actually

approximated

by

only

a

few

k

eigenfaces

and

PCs

with

the

largest

eigenvalues
.



256

㈵2

8

扩b
†
㈰2

業ag敳
†


敩e敮晡捥e

†††


2
2
N
N

X
e
U
T
i
i

U
e
X
i

Image Computing Group. All rights reserved 2002.

Using PCA in the Face Recognition
(FR) (3)


Define a face class using a group of images of one person


Get a group of

k
-
d PC vectors


Defined the face class centered at the mean of these PC vectors.


Choose a distance threshold to define the maximum allowable distance .


Define a face space using all of the images in the training set


Get the PC vectors of all the face images in the training set


Find the mean and allowable distance to determine if an input image is a face
images


Face Recognition


A

new

face

is
:


(
1
)

A

known

people

:

if

it

belongs

to

a

face

class
.


(
2
)

An

unknown

people
:

add

to

training

set
.


(
3
)

Not

a

face

at

all
:

far

away

from

the

face

space
.


]
,...,
,
[
2
1
k
T
u
u
u
U

X
e
U
T
i
i

i
i
u
e
X

Image Computing Group. All rights reserved 2002.


Applying PCA in the Active Shape
Models (1)


Flexible models, or deformable templates, can be used to allow for some
degree of variability in the shape of the image objects


PCA is often used to find some most important "mode of variation", and
helps to obtain a more compressed statistical description of the shapes and
their variation

Metacarpal shapes in X
-
ray images

Image Computing Group. All rights reserved 2002.


Applying PCA in the Active Shape
Models (2)


Labeling and Aligning the shapes training set


Each shape has been described by a position vector( A points in 2p
-
d space )


Mean shape:


Get the most important modes of variation


Find the eigenvectors from the 2p

2瀠捯va物湣攠ma瑲楸


The position vectors can be reproduced by the eigenvectors weighted by the values of
PCs, and every eigenvectors accounts for a kind of mode of variation .


K largest eigenvalues and their eigenvectors, the most important modes of variations.


All of the PC vectors in the training set can consist of the "Allowable Shape Domain" .


A possible shape similar to the class of shapes can be expressed by those modes with
different weights in the “Allowable Shape Domain” .

)
,
,...
,
,
(
2
2
,
1
1
ip
ip
i
i
i
i
i
y
x
y
x
y
x
x




N
i
i
x
N
x
1
1
Image Computing Group. All rights reserved 2002.

A useful deriving for computation in
practice


High computer load to find the eigenvectors in image analysis


Face Recognition,
256

256⁎
2

=65,536 And C = 65,536


65ⰵ,6


There are only M meaningful eigenvectors


( M is the number of the images in the training data)





Suppose
V

is the eigen vectors of



is the eigen vectors of







i
i
i
T
V
AV
A


i
i
i
T
AV
AV
AA


i
AV
T
M
i
T
i
i
AA
M
C






1
1
)
(
2
2
N
N

A
A
T
)
(
M
M

T
AA
C





I
]
,...,
,
[
2
1
M
A




)
1
(
2

N
)
(
2
M
N






M
k
k
ik
i
i
V
AV
U
1
Image Computing Group. All rights reserved 2002.

Conclusion

PCA

has

been

proved

an

effective

approach

for

image

recognition,

segmentation

and

other

kind

of

analysis
.



Advantages:

(1)Extract uncorrelated basic feature sets to describe the properties of a data set.

(2)Reduce the dimensionality of the original image space.


Disadvantages:


(1) Provide little quantitative information and visualization implications


(2)
No way to discriminate between variance due to object of interest and variance
due to noise or background.


Researchers proposed different schemes and make many improvements for
better applying PCA in image analysis.

Image Computing Group. All rights reserved 2002.

The End

Thank you very much