Prof. Ioannis Pavlidis

U of H

COSC 6397
–

Lecture 10

#
1

U of H

COSC 6397

Face Recognition in the Infrared Spectrum

Prof. Ioannis Pavlidis

U of H

COSC 6397
–

Lecture 10

#
2

Primary Applications


•
Biometric Identification

–
Passwords/PINs.

–
Tokens (like ID cards).

–
You can be your own password.


•
Surveillance

–
Off
-
the
-
shelf facial recognition
system that identifies humans as they
pass through a camera’s field of
view.




U of H

COSC 6397
–

Lecture 10

#
3

Novel Applications

•
Wearable Recognition Systems


–
Adapt to a specific user and be more
intimately and actively involved in the
user's activities.

–
Face recognition software can help
you remember the name of the
person you are looking at.

•
Useful for Alzheimer's patients.



•
Smart Systems


–
Key goal is to give machines perceptual abilities that allow them to
function naturally with people.

–
Critical for a variety of human
-
machine interfaces.


U of H

COSC 6397
–

Lecture 10

#
4

Why Infrared?

•
Visible light has no effect on images
taken in the thermal infrared
spectrum.

•
Even images taken in total darkness
are clear in the thermal infrared.



U of H

COSC 6397
–

Lecture 10

#
5

Why Infrared? (Contd..)


•
Illumination Invariance

–
Major problem in visible domain.


•
Uniqueness and Repeatability

–
Sense thermal patterns of blood vessels under the skin,
which transport warm blood throughout the body.

–
Remain relatively unaffected by aging.

–
Even identical twins have different thermograms.


•
Immune from Forgery


–
Disguises can be easily detected.




U of H

COSC 6397
–

Lecture 10

#
6

Previous Work

•
Lot of research was done in the visible band but little attention was
given in the infrared spectrum.

•
Recent reduction in the cost of infrared cameras and availability of
large data sets encouraged active research in infrared face recognition.

•
Low
-
Level Models

–
Directly analyze the image pixels and impose probabilities on the features.

–
Examples are PCA, ICA, and FDA.

–
Not good in challenging conditions.

•
High
-
Level Models

–
Synthesize images from 3D templates of known objects and impose probabilities
on transformations.

–
Template matching approaches.

–
Computationally expensive.

•
Our Proposal

–
Intermediate model which takes advantage of both Low
-
Level and High
-
Level
models.



U of H

COSC 6397
–

Lecture 10

#
7

Principal Component Analysis

•
A
D = H x W

pixel image of a face,
represented as a vector occupies a
single point in
D
2
-
dimensional image
space.

•
Images of faces being similar in overall
configuration, will not be randomly
distributed in this huge image space.

•
Therefore, they can be described by a
low dimensional subspace.

•
Main idea of PCA (cutler96):

–
To find vectors that best account for
variation of face images in entire
image space.

–
These vectors are called eigen
vectors.

–
Construct a face space and project the
images into this face space
(eigenfaces).


U of H

COSC 6397
–

Lecture 10

#
8

Eigenfaces Approach
-

Training

•
Training set of images represented by





1
,

2
,

3
,…,

M



•
The average training set is defined by



Ψ = (1/M) ∑
M
i=1


i


•
Each face differs from the average by
vector Φ
i

= Γ
i

–

Ψ



•
A covariance matrix is constructed as:


C = AA
T
, where A=[Φ
1
,…,Φ
M
]


•
Finding eigenvectors of
N
2

x
N
2

matrix
is intractable. Hence, find only
M

meaningful eigenvectors.
M

is typically
the size of the database.






U of H

COSC 6397
–

Lecture 10

#
9

Eigenfaces Approach
-

Training

•
Consider eigenvectors
v
i

of
A
T
A

such that





A
T
A
v
i

=
μ
i
v
i



•
Pre
-
multiplying by
A
,
A
A
T
(
A
v
i
) =
μ
i
(
A
v
i
)


•
The eigenfaces are


u
i

=
A
v
i




•
A face image can be projected into this face space by




Ω
k

= U
T
(Γ
k

–

Ψ); k=1,…,M








U of H

COSC 6397
–

Lecture 10

#
10

Eigenfaces Approach
-

Testing

•
The test image, Γ, is projected into the face space to obtain a vector, Ω:


Ω = U
T
(Γ
–

Ψ)


•
The distance of Ω to each face class is defined by



Є
k
2

= ||Ω
-
Ω
k
||
2
; k = 1,…,M


•
A distance threshold,Ө
c
, is half the largest distance between any two
face classes:




Ө
c

= ½ max
j,k

{||Ω
j
-
Ω
k
||}; j,k = 1,…,M


U of H

COSC 6397
–

Lecture 10

#
11

Eigenfaces Approach
-

Testing


•
Find the distance, Є , between the original image, Γ, and its
reconstructed image from the eigenface space, Γ
f
,



Є
2

= || Γ
–

Γ
f

||
2
, where Γ
f
= U * Ω + Ψ


•
Recognition process:

–
IF Є≥Ө
c

then input image is not a face image;

–
IF Є<Ө
c

AND Є
k
≥Ө
c

for all k

then input image contains an unknown face;

–
IF Є<Ө
c

AND Є
k
*=min
k
{ Є
k
} < Ө
c


then input image contains the face of individual k*


U of H

COSC 6397
–

Lecture 10

#
12

Limitations of Eigenfaces Approach


•
Variations in lighting conditions

–
Different lighting conditions for enrolment and query.

–
Bright light causing image saturation.


•
Differences in pose
–

Head orientation

–
2D feature distances appear to distort.


•
Expression


–
Change in feature location and shape.


U of H

COSC 6397
–

Lecture 10

#
13

IR Face Recognition
–

Training Phase

U of H

COSC 6397
–

Lecture 10

#
14

IR Face Recognition
–

Test Phase

U of H

COSC 6397
–

Lecture 10

#
15

Segmentation

•
Noise in the background may effect
the performance of a face
recognition system.


•
Remove the background.


•
Use thermal information on face to
compute the features.


•
Adaptive Fuzzy Segmentation (kakadiaris02)

–
Fuzzy affinity is assigned to spels w.r.t. target object spel.

–
Affinity is computed as weighted sum of the temperature and the
temperature gradient in the neighborhood of the target spel.

–
Minimal user interaction because of dynamically assigned weights.


U of H

COSC 6397
–

Lecture 10

#
16

Segmentation (Contd..)

•
Fuzzy affinity is calculated by:


–
Spatial Adjacency:


U of H

COSC 6397
–

Lecture 10

#
17

Segmentation (Contd..)

–
Temperature homogeneity & gradient:


–
Weights:


-

Temperature of seed c


-

Temperature of seed d


-

Mean Temperature

-

Standard deviation of temperature


U of H

COSC 6397
–

Lecture 10

#
18

Problem with Single Seed

•
Temperatures on face are
different at different regions.

•
If a single seed is chosen in a particular region,
then the connectivity stretches only along this
region and the segmentation goes wrong.

U of H

COSC 6397
–

Lecture 10

#
19

Multiple Seeds

•
Solution to this problem is to choose
multiple seeds in different regions
on face and merge the resulting
segmented parts .

•
Choose a seed pixel on face wherever
there is sharp change in gradient.

•
Works well even when the subject is
wearing glasses.

•
Robust to variation of
poses.

U of H

COSC 6397
–

Lecture 10

#
20

Choosing Multiple Seeds

U of H

COSC 6397
–

Lecture 10

#
21

Assumptions

•
Merge all resultant segmented regions to
form final image.

ASSUMPTIONS

•
The center of the image contains the pixel from facial region.

•
The temperatures at all pixels are mapped between 0 and 255.

–
If this mapped temperature at a pixel is between
175
-

200
, it is classified
to be in
blue

region.

–
If this mapped temperature at a pixel is between
200
-

225
, it is classified
to be in
pink

region.

–
If this mapped temperature at a pixel is between
225
-

255
, it is classified
to be in
cyan

region.

U of H

COSC 6397
–

Lecture 10

#
22

Feature Extraction

•
The Gabor filter bank is given by:

•
The segmented facial image is divided into its spectral components
using Gabor filters
.

•
The resultant Gabor filtered images are modeled using Bessel models.


U of H

COSC 6397
–

Lecture 10

#
23

Gabor Filter Bank

•
Example Gabor filter bank with 3 scale values and 4
orientation values:


U of H

COSC 6397
–

Lecture 10

#
24

Spectral Components

U of H

COSC 6397
–

Lecture 10

#
25

Bessel Parameters

•
The filtered images are modeled using Bessel parameters:





SK
–

Sample Kurtosis


SV
–

Sample Variance





•
Each segmented image in training set is convolved with the filters in
Gabor filter bank to obtain Gabor filtered images.


U of H

COSC 6397
–

Lecture 10

#
26

Sample Variance and Kurtosis

•
Sample Variance

is the measure of the “
spread”
of the
distribution
.


•
Sample Kurtosis

is the measure of the “
peakedness”

or
“
flatness”.


Sample Kurtosis,

U of H

COSC 6397
–

Lecture 10

#
27

Bessel Model


•
Using the bessel parameters
p

and
c
, t
he filtered image I
(j
)
(x,y)

is modeled as:




(p)

is gamma function

I
v
(z)

is modified bessel function of
first kind given by:

U of H

COSC 6397
–

Lecture 10

#
28

Bessel Model

U of H

COSC 6397
–

Lecture 10

#
29

Performance of Bessel K Forms

•
Kullback
-
Leiber divergence:









KL div=0.0013 KL div=0.0027 KL div=0.0055

KL div=0.0058









–

observed marginal density


–

Estimated Bessel Form

U of H

COSC 6397
–

Lecture 10

#
30

Comparing IR Images

•
Images modeled into Bessel parameters can be compared by:







•
L
2
-
metric between two Bessel forms f(x;p
1
,c
1
) and f(x;p
2
,c
2
) in
D
:




U of H

COSC 6397
–

Lecture 10

#
31

Hypothesis Pruning

•
Applying a high
-
level classifier on entire database is
computationally very expensive.


•
Pruning of hypotheses can be achieved by using Bessel
parameters (anuj01).


•
Helps in short listing best matches.


•
Bessel parameters for images in database can be computed
offline which helps in saving a lot of computation time.

U of H

COSC 6397
–

Lecture 10

#
32

Hypothesis Pruning (Contd..)

•
Define a probability mass function on the database
A:

(p
(j)
obs
,c
(j)
obs
)
–

observed Bessel parameters for test image I
(j)

(p
(j)

,s
,c
(j)

,s
)
–

estimated Bessel parameters which can be computed offline

•
Images in database
A

with
P
1
(

|I
)

greater than a specific
threshold value are short listed as best matches.

(D=0.3 for Equinox dataset)

U of H

COSC 6397
–

Lecture 10

#
33

Hypothesis Pruning (Contd..)

•
Shortlist the subjects of

A

with P
1
(

/I) greater than a specific
threshold:


U of H

COSC 6397
–

Lecture 10

#
34

Pruning Algorithm

U of H

COSC 6397
–

Lecture 10

#
35

Classification

•
Bayesian target recognition (anuj00) searches for the target
hypothesis with largest
posterior probability

given by:

–
Likelihood
:

–
Apriori
is same for all images in database (for database of


n images, it is 1/n for each image).



: Variance of test image

d : dimension of image (2 in this case)

U of H

COSC 6397
–

Lecture 10

#
36

Experiments


•
Equinox Database:

www.equinoxsensors.com



•
Image frame sequences were
acquired at 10 frames/sec while
the subject was reciting the
vowels ‘a’,’e’,’i’,’o’,’u’.


U of H

COSC 6397
–

Lecture 10

#
37

Results
–

ROC Curves

Correct Positive

: Test image is in the
database and is correctly recognized.

False Positive

: Test image is not in the
database, but is recognized to be an image
of the database

Negatives

: Test images that are not in the
database.

U of H

COSC 6397
–

Lecture 10

#
38

Results
–

Precision & Recall

U of H

COSC 6397
–

Lecture 10

#
39

Conclusion


•
We came up with a face recognition approach which is
computationally inexpensive and at the same time good
in challenging conditions.


•
The features of all images in database can be computed
offline and stored for future use. This saves lot of
computation time.


•
We improved the performance of classifier by removing
background noise of pruned hypothesis using adaptive
fuzzy connectedness based image segmentation.



U of H

COSC 6397
–

Lecture 10

#
40

References


•
[anuj01] A. Srivastava, X. W. Liu, B. Thomasson, and C. Hesher,
"Spectral Probability Models for IR Images with Applications to IR Face
Recognition," in
Proceedings 2001 IEEE Workshop on Computer Vision
Beyond the Visible Spectrum: Methods and Applications,
Kauai, HI, Dec
14
.


•
[cutler96] R. Cutler, “Face recognition using infrared images and
eigenfaces”,
website, http://www.cs.umd.edu/rgc/face/face.htm
, 1996.

•
[anuj00] A. Srivastava, M. I. Miller, and U. Grenander, “Bayesian
automated target recognition,"
Handbook of Image and Video
Processing
, Academic Press, pp. 869
-
881, 2000.

•
[kakadiaris02] A. Pednekar, I.A. Kakadiaris, U. Kurkure. Adaptive fuzzy
connectedness
-
based medical image segmentation. In
Proc. of the Indian
Conf. on Computer Vision, Graphics, and Image Processing (ICVGIP
2002)
, pp.457
-
462, Ahmedabad, India, December 16
-
18 2002.