ACAT’2002 24

28 June, Moscow, RUSSIA
Joint Institute for Nuclear Research
141980 Dubna, Moscow region, RUSSIA
Laboratory of Informative Technologies
Effective training algorithms
for RBF

networks
Gennadi A.OSOSKOV, Alexey STADNIK
e

mail: ososkov@j
inr.ru http://www.jinr.ru/~ososkov
Outline
1.
Introduction, image recognition problem
2.
Problem formulation for a security system
3.
Multilayer perceptron, arising hindrances
4.
Proposed RBF

network design and image preprocessing
Training algori
thm for RBF

nets with Mahalanobis
distance. Some examples.
5.
The first security application
6.
How to decrease neural net dimensionality for image handling
偲楮捩灡氠捯mpo湥湴nme瑨潤t
†††††††
坡v敬整猠eormag攠獣a汩湧
㜮
Applications and re
sults
8.
Conclusion
2 Problem formulation for a security system
Neural network is considered as a mean for the
fast and reliable
recognition
of any of substantial group of human faces.
Reliability requirements:
probability of ide
ntifification error = 0.01;
probability of misidentifification = 0.005.
The
only frontal views of face images
are considered further, which are
digitized (by a video

camera, for instance) and stored as 2D raster. In the
majority of cases a raster
with not less than
80x100 pixels of 8

bit grey
level
is efficient to distinguish individual features of a person.
To obtain a reliable level of recognizability the MLP must be trained before
on a training sample of digitized face images of all person
s to be recognized.
After training NN must adequately recognize any of faces from
the sample and undoubtedly indicate any case of a "stranger" face.
Network must function in real circumstances when a person can slightly
vary
its pose, have a minor chan
ges of the face expression, hair

dressing, make

ups,
be
unshaven
etc.
Such the reliability and robustness requirements can be accomplished by
including into the
training sample more than one face image (up to 10)
of the
same person.
A
rizing hindrances:
“Damnation of dimension” leading to very long back

propagation training;
Arbitrariness in choosing of the hidden layer neurons. The MLP structure
is fixed during the training;
Difficulties with selecting the training sample to be long en
ough to
guarantee the correct classification.
Such neural nets are known as RBF

nets
–
Radial Basis Function
neural networks. RBF

nets differ from MLP by two things: by their
metrics (it can be not only L
2
, but also Manhattan or Mahalanobis
metrics
) and by activation function (gaussian instead of (2)).
Our RBF

net innovations are as follows:
New sructure
New training algorithm.
4. 1 New structure
4.2 The strategy of training
The main features of the first training algorithm:
use as activation
function F(x)=1;
if
(x
)
o爠㴰r
;
楦
⡸
)
†
w楴栠a摤楴楯da氠灡牡m整敲e
,
w桩捨猠慬ho⁷楬氠扥
o
灴pmz敤
††
摵物湧
t牡楮楮
;
dynamically add neurons
to the hidden layer;
train layers
of the network
separately
. First

clasterization, second

mapping
to the desired output;
train each neuron in
the
layer also separately
(!)
Separate training of each neurons in all layers, gives high speed and
finiteness of the training procedure.
D
uring training procedure all training set
separated into three sub
sets
:
samples which are already classified by the network
(
AC
);
samples which are not classified
(
NC
);
samples which are classified by current neuron (
CC
)
The idea of training procedure is to train
a
single neuron in
NC
(not
classified samples) and th
en add
it
to the RBF

network.
Algorithm stops when
NC
becomes empty.
The strategy of training
a
single neuron is:
randomly choose one sample
in NC
.
allow threshold parameter
瑯t
g牯w.
add samples which are closer in terms of selected metric then
t
o䍃Ca湤n
牥mov攠
瑨tm
晲fm乃N
recalculate
synaptic
weights of
every
neuron as
the
center of gravity of
corresponding
samples in CC set;
keep increasing
threshold parameter
畮汥獳⁴桯獥
sam灬敳p扥汯湧⁴o瑨攠
獡m攠捬慳s
;
add a new n
euron to the network ha
ving all samples from C
C
added to the
AC.
Such t
raining procedure guarantee
s
the
finite number of training cycles and
100% correct classification
by the training set.
Therefore, as
the
result
,
we have
the
layer, which produce
s
just one
activated neuron
for each sample in training set, then the last layer to be
mapped to the desired output, can be
“
trained
”
by setting
to 1
weights
connected to such
activated
neuron
while
others
are to be set to 0.
Then we complete this first algorithm of training by
the possibility of
having an extra sample set containing
the wrong
classified
samples
(
W
C
S
).
Further we name it as the second algorithm.
Three examples of 2D classification by MLP and RBF net. Different
classes marked by different colors.
Example 1.
From left to right there are presented: (1) training set; (2) classification by
RBF

network; (3) classification by MLP.
Example 2.
demonstrates the difference in efficiency of both RBF

net
algorithms. From left to right: (1)tr
aining set; (2) RBF

network trained by
the first algorithm; (3) RBF

net trained by the second algorithm.
Example 3.
shows result of classification of well

known benchmarking
problem of separation two imbedded spirals. From le
ft to right: (1)training
set; (2) RBF

network trained by the first algorithm; (3) RBF

net trained by
the second algorithm.
5. The first security application
Now we were ready to work with frontal face images. We use, at first, as the
training sam
ple, the following set (see fig. below):
The RBF neural network with
L
2
–
metrics after training on this
small set was enabled to recognize
without errors specially distorted
faces from the same set (see the
next picture)
However
, as soon as we decided to
apply our RBF net to 400 images
set from the famous Cambridge
face database, our neural net
began to mix up faces.
The reason was foreseeable.
Let us consider a digitized image of a face as a vector
, i.e. a point in a
space of an unthinkable dimensionality like
. All these points occupy only
a very tiny part, a miserable subspace of this giant space. Therefore our
attempts to search in this whole space for an particular imag
e without taking
into account any specifics of human faces and that particular face are doomed
to be unreliable.
6. Principal component method (PCM)
PCM is a way to project our data onto this subspace extracting most adequate
features by using the inf
ormation about mutual correlations
.
There is an orthogonal transform
(named
Karhunen

Loeve transform), which converts
to its diagonal form
, where eigenvalues
of
are numbered in
their descent order. One can keep the only most essential
components
(m
<< p).
Main components as a function of their numbers
Th
us we can now express the source data
via these main components
neglecting non

important ones.
PCM computational consequences
1.
Being linear, principal component method can be easily reali
zed as the first
layer of an RBF net;
2.
It gives a considerable economy of computing resources what is important
as on the RBF

net training phase and also for improving the net reliability.
However, PCM has some
shortcomings
:
principal component
algorithm i
s NP

complete, i.e RBF

net training time
grows exponentially with the client number increase;
as soon as the number of
new
clients joining the face base exceeds
20

30% of its current value, the RBF

net must be trained afresh
completely, since the prognosis
capability of the covariance matrix
is rather restricted.
Applying PCM to the collection of frontal face images from the Cambridge
face database we found that obtained main components (see Fig. on the
right) are
too dependent from variations of the sourc
e images in
lightening, background etc.
Main components of some
faces from the Cambridge
face database without
previous wavelet
transformation
Therefore the wavelet preprocessing have been applied
It removes depending on the
lightening and
performs a scaling
of images, although some of
important face features have
been lost.
Main components of the same
face images after their
preprocessing by 2D gaussian
wavelets
A fast algorithm was developed for 2D vanishing momenta wavelets.
Applying
it to the image below we obtain the following wavelet expansion:
A face image its 2D wavelet expansion
Summarizing three wavelet
s
–
vertical, horizontal and diagonal we obtain the
wavelet transform
independent on the image variability of lightening,
background and size
.
Lower row shows results of applying
2D gaussian 2

d order wavelets
to f
ace images of the upper row
Nevertheless, after detailed studying of the efficiency and misidentification
probability of C++ program implementing our RBF

like neural network with
the second algorithm of training (RBFNN2), we had to refuse from using
wavel
ets for the present time. The reason was in above

mentioned loss of face
features for some type of faces. Besides it is easy for our security system to
provide checkpoints by uniform lightening and keep the same distance to a
photographed person.
A
fter training RBFNN2 on 250 face images we test it on 190 faces with
very promising results: efficiency
–
95% and not a single case of wrong
acceptings! 5% of inefficiency occurs due to the only case, when among 10
pictures of the same person used for trai
ning on one picture it was a photo of
this man made an unusual grimace, so the program did not accept namely
that photograph of this man.
However, we are still going to study more in details the idea of applying
wavelets for a considerable face image
compression without loosing important
features, in order to apply then principal component method to wavelet
coefficients obtained on the preprocessing stage.
Conclusion
New RBF

like neural network is proposed, which allows to process
raster inform
ation of digitized images ;
A study of the reliability of direct RBFNN2 application to frontal face
data shows the need in data preprocessing by extraction of
principal
components after scaling the data by 2D wavelet transfom;
Wavelet preprocessing result
ing in significant data compression
is still under study;
Corresponding object

oriented C++ software is developed to
work with frontal face images recorded by a video

camera. The
first results on statistics provided by Cambridge face database
are quite pr
omising.
Comments 0
Log in to post a comment