Average of Synthetic Exact Filters (ASEF)

chemistoddAI and Robotics

Nov 6, 2013 (3 years and 8 months ago)

89 views

MITRE Corporation

Pose Correction for Automatic
Facial Recognition

Team
: Elliot
Godzich
, Dylan
Marriner
, Emily Myers
-
Stanhope,
Emma
Taborsky

(PM), Heather Williams


Liaisons
: Josh
Klontz

’10 and Mark Burge

Advisor
: Zachary
Dodds


Fraud detection


Aid distribution


Law enforcement


National security



Algorithmic identification of faces from images


Commercial systems exist; MITRE is building a
U.S. system for flexibility and security


Unobtrusive relative to other biometric
techniques, but with similar applications
:

Automated Facial Recognition

Privacy

Concerns


Off
-
pose images are a significant challenge for
automated
facial
recognition



Many
current
algorithms
, including MITRE's,
do not include pose
correction

Pose

Correction

Pose

Correction


Our approach to pose
-
correction involves finding
and matching facial features in different images


Feature
-
finding and shape transformation, are
also useful for other image
-
processing tasks


research
:

use and extend existing approaches


implement
:

within MITRE's
existing codebase


test
:

using MITRE's test
scaffolding and databases

Problem Statement

Our goal is to
research
,
implement
, and
test

a
pose correction library

that improves
MITRE's existing facial recognition system.

Average of
Synthetic Exact
Filters

Active Shape
Model

Pose
-
correction pipeline

Pixels

F敡瑵e敳

卨慰S

ASEF

ASM


Facial
features, or
landmarks, can
support
both
recognition and
pose
-
correction



Features are
based
on spatial geometry
and/or
appearance

Features

ASEF
filter

creation

training image
(with known right
-
eye location)

human
-
designed
synthetic output

For each training image we create a
synthetic
output with
the correct position of the feature, e.g., the right eye.

ASEF
filter

creation

training image
(with known right
-
eye location)

human
-
designed
synthetic output

filter transforming
the image at left into
the image at right

We want to create
a filter that exactly transforms
a training
image into
the desired synthetic
output

*

=

ASEF
filter

creation

In the Fourier domain, we want

where
Synthetic
,
Image
, and
Filter

are the 2D Fourier
transforms of the synthetic output, image, and
filter.

Complex division thus provides the filter:

ASEF
filter

creation

We take
the average of all of the synthetic exact
filters to
define, here, a
final
right
-
eye filter

We average 517
filters like this…

ASEF
filter

creation

We take
the average of all of the synthetic exact
filters to
define, here, a
final
right
-
eye filter

We average 517
filters like this…

…to obtain the
final filter?

ASEF
filter

creation

We take
the average of all of the synthetic exact
filters to
define, here, a
final
right
-
eye filter

We average 517
filters like this…

…to obtain the
final filter.

ASEF
filter

application

The filter’s strongest
response is most
right
-
eye
-
ey

location in
the image

Unfiltered image

F
iltered image

We apply the filter in the Fourier domain; the peak in the
spatial domain is a first estimate of the feature
location

Final

output

Gallery



Error

Images

within that error

<

.01

26.3 %

<

.02

63.9 %

< .05

86.1 %

<

.1

87.7 %

ASEF
results

Many images' eyes are found quite accurately,
but there are also some dramatic outliers:

Units are fraction of
interocular

distance

Percentage of pictures

Influence of
ASEF’s

Gaussian

s

Radius,
s

= 2px

Radius,
s

=

25px

Radius,
s

=

15px

synthetic outputs

ASEF filters

Radius,
s

=

20px

ASEF
tradeoffs

Testing changes in
Gaussian radii
(
s
)

the opposite
tradeoff

more accurate
localization


and
more outliers

left eye error

(units of
interocular

distance)

Radius,
s

=

5px

left eye error

(units of
interocular

distance)

ASEF
improvements

Using spatial heuristics as weights

Unweighted

filtered image

Spatially weighted
filtered image

1.0
*

潲o杩nal

0.5
*

潲i杩gal

original

Without weighting

With weighting

ASEF
improvements

Using spatial heuristics as weights

right eye error

(units of
interocular

distance)

right eye error

(units of
interocular

distance)

left eye error

these clusters show
mis
-
identifying the
left or right eye

Average of
Synthetic Exact
Filters

Active Shape
Model

Pose
-
correction pipeline

Pixels

F敡瑵e敳

卨慰S

ASEF

ASM

Active
Shape

Models

(ASM)



Describe classes
of objects with varying
shapes

g
eometric arrangement of
facial features: eyes, nose, …



Each
shape
is
a
set of
points





ASM trains
on a
training set
of
shapes, creating a statistical
model of
the variation within
that shape
-
family.


ASM, step 1:
Procrustes

fitting

Procrustes

analysis determines a scaling, rotation,
and translation that best
align
a family of shapes.

t
raining data (hundreds of faces)

mean face

(not necessarily angry)

We
use this
approach to
align all
of the
training
faces
and extract the
mean face.

ASM, step 2:
Estimating

face
space

We use the most descriptive eigenvectors to
describe
the allowable shape domain.

ASM uses principal components analysis to build
a
model of representative transformations
of a face

s

= 0

(mean face)

-
3
s

+3
s

Independent
face
-
shape axes

ASM, step 3:
Transforming

faces

W
e
can apply realistic transformations to the
mean
face along face space’s eigenvectors.

Second semester plans

1) Multi
-
resolution and weighted ASEF feature finding


2) Adding pixel appearance to the ASM shape models


3) Implementing pose
-
correction techniques (for pixels)

shape space:
yaw

First approach:
apply ASM's transformations to
generate poses at desired values of pitch and yaw.

Project Work

Clinic Deliverables

Due Date

January

Winter break


Spring Semester
Begins: 1/17

Phase III Presentation

1/17/2012

Implement AAM,
continue improving
ASEF, research and
select pose correction
methods


Final Report & Poster

February

Begin implementing
selected pose correction
methods, combine
ASEF and
ASM

March


Spring Break: 3/9
-
18

Spring Break


Continue work on
pose
correction

April

Final Report

Draft of Poster Design

4/2/2012

Revise FR, Final Pres

Draft 1 of Final Report

4/10/2012

Final Touches

Final Report Review

4/12/2012

Feature Freeze

4/13/2012

Draft of Final Report

4/18/2012

Draft of Final
Presentation

4/23/2012

May


Finals: 5/3
-
4

Projects Day

5/1/2012

Final Report

5/4/2012

Spring Schedule

MITRE
clinic
, spring 2012
schedule

Questions
?

Average of
Synthetic Exact
Filters

Active Shape
Model

Pixels

F敡瑵e敳

卨慰S

ASEF

ASM

Gallery

Gallery



Second semester plans

The
spring term will focus on researching and
implementing landmark
-
based pose
correction
techniques.


First approach:

apply
transformations given by ASM to
generate
poses
at varying degrees of pitch and
yaw.

yaw

pitch

Error

Without

log transform

<

.01

26.3 %

<

.02

63.9 %

< .05

86.1 %

<

.1

87.7 %

ASEF
results

Comparing image pre
-
processing techniques

Error

With log transform

<

.01

25.4 %

<

.02

61.3 %

< .05

83.7 %

<

.1

85.6 %

Fraction of
interocular

distance

Percentage of pictures

AAM adds color or
grayscale

information to
ASM’s model. AAM can generate
photorealistic
faces, not just geometrically realistic ones.

Active
Appearance

Models

(AAM)

Shown here are faces generated
by varying the central face’s
appearance

parameters by
±
3
s

along two appearance axes.

from T.F.
Cootes
, G.J. Edwards, and C.J.
Taylor
,
Active Appearance Models

o
ld

pipeline

n
ew
pipeline

Face
-
recognition

pipeline

Face
detection

Recognition

Landmarking

Pose

correction

Input image

Output

ID

Fall
term’s

focus

印物湧r
term’s

focus

Next

Steps

Improving ASEF
:

We will experiment
with
image
processing
techniques and weighting based on
expected pose and image complexity



Extending
ASM:

We will implement Active
Appearance
Models to
extend face pose
-
generation to
face image
-
generation.


Implementing Pose
Correction:

ASEF
and ASM provide a
baseline approach
:
namely, transforming a query image to a
standard face pose

Pixels

Features

Shape

Automated Facial Recognition


Use
of computers to identify faces from
images


Commercial
systems
exist,
but MITRE is developing a
system specifically for the
US for flexibility and security


Unobtrusive relative to other biometric techniques, but
with similar applications:


Motivation: Uses for Biometrics



Law enforcement and national security




Fraud detection



Aid distribution



Social networking

Error

Percent of Identifications

<

.01

0.263610315186246

<

.02

0.638968481375358

< .05

0.861031518624642

<

.1

0.876790830945559

Error

Percent of Identifications

<

.01

0.253581661891117

<

.02

0.613180515759312

< .05

0.836676217765043

<

.1

0.859598853868195

Without cosine window

With cosine window

ASEF
improvements

Mapping

Last semester

This semester

Face
-
recognition

pipeline

o
ld

pipeline

n
ew
pipeline

Face
-
recognition

pipeline

Face
detection

Recognition

Landmarking

Pose

correction

Input image

Output

ID

Training Data

Average
Face

ASM, step 2:
Mean
-
face
finding

We
use this
approach to
align
all

of the training
faces
and thus find
the mean face.

We
got

this
… ?

Centered
!

ASEF’s

right
-
eye

filter

in the
spatial

domain

Face
-
recognition

pipeline

Face
-
recognition

pipeline

Pixels

Landmarks

Shape Model

Average of Synthetic Exact Filters (ASEF)

Active Appearance Model (AAM)

Landmarking

algorithms

ASEF
filter

creation

For each training image we create a
synthetic
output with
the correct position of the feature, e.g., the right eye.

training image (with
known right
-
eye location)

human
-
designed
synthetic output


Our approach to pose
-
correction involves finding
and matching facial features in different images

Pose

Correction

Average of Synthetic Exact Filters (ASEF)

Active Shape Model (ASM)

Landmarking

algorithms

Pixels

Features

Shape

o
ld

pipeline

Face
-
recognition

pipeline

Face
detection

Recognition

Input image

Output

ID

o
ld

pipeline

n
ew
pipeline

Face
-
recognition

pipeline

Face
detection

Recognition

Landmarking

Pose

correction

Input image

Output

ID

MITRE Corporation

Pose Correction for Automatic
Facial Recognition

Team
: Elliot
Godzich
, Dylan
Marriner
, Emily Myers
-
Stanhope,
Emma
Taborsky

(PM), Heather Williams


Liaisons
: Josh
Klontz

’10 and Mark Burge

Advisor
: Zachary
Dodds