Face Recognition Based on Image Sets

spraytownspeakerAI and Robotics

Oct 16, 2013 (3 years and 7 months ago)


Bryce Eggleton

Robot Vision

Fall 2011

Face Recognition Based on Image Sets


This paper stood out to me not simply because it presented a method of face recognition, but
because of the unique approach and method by which they preform their algori
thm. While normal
algorithms approach the problem from the perspective of identifying a person from a single image, the
goal of this algorithm is to recognize an individual from an entire set of input images. This can be
beneficial for a number of reason
s, such as recognition from a set of images where there are large
variations in pose, shadow, or expression, as in frames from a video taken over an extended period of
time, or where some of the input images aren't completely clear. Each known entity has
their own
collection of images that forms a set. From there, each set is modeled as a convex geometric region
where each image is represented as a point in that region. Test sets are compared against a gallery of
image sets that represent known entities.

Distances between the models can be used to determine if a
set is similar to another, and thus indicating a potential face match. Support Vector Machines as well as
Kernelization is used to make their algorithm more robust and resistant to incorrect or
dirty input
outliers that might skew test results. Implemented tests on two publicly available face data sets proved
that their approach is as good, and in many cases better than established current algorithms.


First, each image in a given set
becomes a feature vector in a feature space. A convex
approximation for each image set is then calculated to find the region occupied by the feature vectors
of that particular set. The two convex models used for their algorithm are either the affine hull

convex hull of a set's feature points. Gallery sets can then be compared using a distance of closest
approach method. Each image x in a feature space has subscripts c and i, where c is the index of the
feature space (from 1 to C image sets), and i is

the index of the image within the feature space (from 1
to n


Affine Hull Method

The first convex model that they chose is an affine hull (or affine subspace). See fig. 1 for an
image of their equation. An affine subspace is a subset of a vec
tor space (in this case the feature space
of image vectors) which is closed under affine combinations of the vectors in that space. An affine
combination is simply the normal linear combination of vectors and scalar coefficients of the form c

+ c

… + c
Vn, however, the difference is that the sum of the coefficients must be equal to 1 for it
to be considered an 'affine' combination. Affine combinations share most of the same properties as
normal linear combinations do. Based on the above definitio
ns, the affine hull of a set S can then be
described as the set of all affine combinations of the elements of set S. That is, the affine hull is the set
of all linear combinations of the elements of set S whose coefficients equal a sum of 1. Outliers in
data sets can cause the affine hulls of gallery sets to intersect each other. This can cause a test set to
have a distance of 0 to more than one set in the gallery, making a clear label indistinguishable. To
reduce this problem they impose limitation
s to the coefficients for the affine combinations (fig. 2).
These limitations control how tight, or loose the convex approximation is, with certain bounds being
special cases (fig. 3). This allowed them to tighten the hull to provide a more suitable repr
esentation of
the feature space. This can then be simplified to (fig. 4) which can be used with arg min (fig. 5) to find
which points in two feature spaces are closest, then the distance equation (fig. 6) to find the actual
distance between two hulls to u
se for comparison.

Convex Hull Method

The second convex model that they use is the convex hull method. To understand the properties
of a convex hull, it is necessary to define what a convex combination is. Similarly to how the affine
combination was a
linear combination with certain restrictions imposed, a convex combination is an
affine combination with another restriction. Thus, a convex combination is a linear combination in
which the values of all the scalar coefficients have a sum of 1, AND each c
oefficient must be non
negative. Similarly to the affine hull, the convex hull can be described as the set of all convex
combinations of the elements in set S. That is, the convex hull is the set of all linear combinations of
the elements in set S whose
coefficients are both non
negative and have a sum of 1. Applying the
constraints of L=0, and U>=1 for the equation in fig. 2 then approximates the set by the smallest
convex set containing it, or the convex hull. Distances can be calculated the same as w
ith the affine
hulls, but with L=0 and no U constraint, however they approach it by training an SVM to take inter
distances to find the distance between hulls.


They tested implementations of both the affine hull method as well as the conv
ex hull approach
against two different data sets. For each approach, they tested a linear as well as kernelized
implementation using a Gaussian kernel. The two data sets used were the Honda/UCSD data set, and
the CMU MoBo (Motion of Body) data set. Both

sets of data contained a number of different video
sequences of individuals, for which they used a Viola
Jones face detector to build the sets of images
representing different individuals. Their algorithm performed better than several other modern
thms including Mutual Subspace Method and Spectral Clustering, with their kernel convex hull
approach performing the best overall. Personally I find this to be an extremely interesting approach to
recognition of an individual from an image set rather than

singular images, and though math may not
be my strong suit, I enjoyed learning about the math logic behind it that makes it actually work.

Image Appendix

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6


"Affine Combination."
edia, the Free Encyclopedia
. Web. 3 Dec. 2011.


"Affine Hull."
Wikipedia, the Free Encyclopedia
. Web. 2 Dec. 2011.


"Affine Space."
Wikipedia, the Free Encyc
. Web. 1 Dec. 2011.


Cevikalp, Hakan. "Face Recognition Based on Image Sets." Web. 12 Oct. 2011.


"Convex Combination."
Wikipedia, the

Free Encyclopedia
. Web. 1 Dec. 2011.


"Convex Hull."
Wikipedia, the Free Encyclopedia
. Web. 3 Dec. 2011.


"An Idiot's Guide to Support Vector Machines." Web
. 16 Nov. 2011.


"Pauls Online Notes : Linear Algebra

Linear Transformations."
Pauls Online Math Notes
. Web. 13

Nov. 2011. <http://tutorial.math.lamar.edu/Classes/LinAlg/LinearTr

Weston, Jason. "Support Vector Machine (and Statistical Learning Theory) Tutorial." Web. 15 Nov.

2011. <http://www.cs.columbia.edu/~kathy/cs4701/documents/jason_svm_tutorial.pdf>.