Effective training algorithms for RBF-networks

haremboingAI and Robotics

Oct 20, 2013 (3 years and 5 months ago)

70 views

ACAT’2002 24
-
28 June, Moscow, RUSSIA


Joint Institute for Nuclear Research

141980 Dubna, Moscow region, RUSSIA


Laboratory of Informative Technologies


Effective training algorithms

for RBF
-
networks


Gennadi A.OSOSKOV, Alexey STADNIK


e
-
mail: ososkov@j
inr.ru http://www.jinr.ru/~ososkov



Outline

1.

Introduction, image recognition problem

2.

Problem formulation for a security system

3.

Multilayer perceptron, arising hindrances

4.

Proposed RBF
-
network design and image preprocessing

Training algori
thm for RBF
-
nets with Mahalanobis


distance. Some examples.

5.

The first security application

6.

How to decrease neural net dimensionality for image handling


偲楮捩灡氠捯mpo湥湴nme瑨潤t

†††††††

坡v敬整猠eor⁩mag攠獣a汩湧



Applications and re
sults

8.

Conclusion
















2 Problem formulation for a security system



Neural network is considered as a mean for the
fast and reliable
recognition

of any of substantial group of human faces.

Reliability requirements:



probability of ide
ntifification error = 0.01;



probability of misidentifification = 0.005.


The
only frontal views of face images

are considered further, which are
digitized (by a video
-
camera, for instance) and stored as 2D raster. In the
majority of cases a raster

with not less than
80x100 pixels of 8
-
bit grey
level

is efficient to distinguish individual features of a person.


To obtain a reliable level of recognizability the MLP must be trained before
on a training sample of digitized face images of all person
s to be recognized.


After training NN must adequately recognize any of faces from
the sample and undoubtedly indicate any case of a "stranger" face.

Network must function in real circumstances when a person can slightly
vary
its pose, have a minor chan
ges of the face expression, hair
-
dressing, make
-
ups,
be

unshaven
etc.


Such the reliability and robustness requirements can be accomplished by
including into the
training sample more than one face image (up to 10)

of the
same person.











A
rizing hindrances:



“Damnation of dimension” leading to very long back
-
propagation training;



Arbitrariness in choosing of the hidden layer neurons. The MLP structure
is fixed during the training;



Difficulties with selecting the training sample to be long en
ough to
guarantee the correct classification.



Such neural nets are known as RBF
-
nets


Radial Basis Function

neural networks. RBF
-
nets differ from MLP by two things: by their

metrics (it can be not only L
2
, but also Manhattan or Mahalanobis

metrics
) and by activation function (gaussian instead of (2)).


Our RBF
-
net innovations are as follows:



New sructure



New training algorithm.

4. 1 New structure



4.2 The strategy of training

The main features of the first training algorithm:




use as activation

function F(x)=1;

if

(x


)

o爠㴰r
;






)

†
w楴栠a摤楴楯da氠灡牡m整敲e

,

w桩捨⁩猠慬ho⁷楬氠扥
o
灴pmz敤

††
摵物湧

t牡楮楮
;



dynamically add neurons

to the hidden layer;



train layers

of the network
separately
. First
-

clasterization, second
-

mapping
to the desired output;




train each neuron in
the
layer also separately
(!)




Separate training of each neurons in all layers, gives high speed and
finiteness of the training procedure.

D
uring training procedure all training set
separated into three sub
sets
:



samples which are already classified by the network
(
AC
);



samples which are not classified
(
NC
);



samples which are classified by current neuron (
CC
)



The idea of training procedure is to train
a
single neuron in
NC
(not
classified samples) and th
en add
it
to the RBF
-
network.

Algorithm stops when
NC
becomes empty.


The strategy of training
a
single neuron is:



randomly choose one sample

in NC
.



allow threshold parameter


瑯t
g牯w.



add samples which are closer in terms of selected metric then


t
o䍃Ca湤n
牥mov攠
瑨tm
晲fm乃N



recalculate
synaptic
weights of
every
neuron as
the
center of gravity of
corresponding
samples in CC set;



keep increasing
threshold parameter


畮汥獳⁴桯獥

sam灬敳p扥汯湧⁴o瑨攠
獡m攠捬慳s
;



add a new n
euron to the network ha
ving all samples from C
C

added to the
AC.

Such t
raining procedure guarantee
s

the
finite number of training cycles and
100% correct classification

by the training set.


Therefore, as
the
result
,

we have
the
layer, which produce
s

just one
activated neuron

for each sample in training set, then the last layer to be
mapped to the desired output, can be

trained


by setting
to 1
weights
connected to such
activated
neuron
while
others
are to be set to 0.


Then we complete this first algorithm of training by
the possibility of
having an extra sample set containing
the wrong
classified
samples

(
W
C
S
).
Further we name it as the second algorithm.


Three examples of 2D classification by MLP and RBF net. Different
classes marked by different colors.

Example 1.


From left to right there are presented: (1) training set; (2) classification by
RBF
-
network; (3) classification by MLP.



Example 2.

demonstrates the difference in efficiency of both RBF
-
net
algorithms. From left to right: (1)tr
aining set; (2) RBF
-
network trained by
the first algorithm; (3) RBF
-
net trained by the second algorithm.






Example 3.

shows result of classification of well
-
known benchmarking
problem of separation two imbedded spirals. From le
ft to right: (1)training
set; (2) RBF
-
network trained by the first algorithm; (3) RBF
-
net trained by
the second algorithm.





5. The first security application

Now we were ready to work with frontal face images. We use, at first, as the
training sam
ple, the following set (see fig. below):






The RBF neural network with

L
2


metrics after training on this
small set was enabled to recognize

without errors specially distorted
faces from the same set (see the
next picture)















However
, as soon as we decided to
apply our RBF net to 400 images
set from the famous Cambridge
face database, our neural net
began to mix up faces.









The reason was foreseeable.


Let us consider a digitized image of a face as a vector
, i.e. a point in a
space of an unthinkable dimensionality like
. All these points occupy only
a very tiny part, a miserable subspace of this giant space. Therefore our
attempts to search in this whole space for an particular imag
e without taking
into account any specifics of human faces and that particular face are doomed
to be unreliable.





6. Principal component method (PCM)


PCM is a way to project our data onto this subspace extracting most adequate
features by using the inf
ormation about mutual correlations

.
There is an orthogonal transform

(named
Karhunen
-
Loeve transform), which converts
to its diagonal form

, where eigenvalues
of
are numbered in
their descent order. One can keep the only most essential

components
(m
<< p).



Main components as a function of their numbers

Th
us we can now express the source data
via these main components



neglecting non
-
important ones.


PCM computational consequences

1.

Being linear, principal component method can be easily reali
zed as the first
layer of an RBF net;

2.

It gives a considerable economy of computing resources what is important
as on the RBF
-
net training phase and also for improving the net reliability.


However, PCM has some
shortcomings
:



principal component

algorithm i
s NP
-
complete, i.e RBF
-
net training time
grows exponentially with the client number increase;



as soon as the number of
new
clients joining the face base exceeds

20
-
30% of its current value, the RBF
-
net must be trained afresh
completely, since the prognosis

capability of the covariance matrix

is rather restricted.



Applying PCM to the collection of frontal face images from the Cambridge
face database we found that obtained main components (see Fig. on the
right) are

too dependent from variations of the sourc
e images in
lightening, background etc.








Main components of some
faces from the Cambridge
face database without
previous wavelet
transformation



Therefore the wavelet preprocessing have been applied





It removes depending on the
lightening and
performs a scaling
of images, although some of
important face features have
been lost.


Main components of the same
face images after their
preprocessing by 2D gaussian
wavelets






A fast algorithm was developed for 2D vanishing momenta wavelets.
Applying
it to the image below we obtain the following wavelet expansion:








A face image its 2D wavelet expansion





Summarizing three wavelet
s


vertical, horizontal and diagonal we obtain the
wavelet transform
independent on the image variability of lightening,
background and size
.




Lower row shows results of applying

2D gaussian 2
-
d order wavelets

to f
ace images of the upper row


Nevertheless, after detailed studying of the efficiency and misidentification
probability of C++ program implementing our RBF
-
like neural network with
the second algorithm of training (RBFNN2), we had to refuse from using
wavel
ets for the present time. The reason was in above
-
mentioned loss of face
features for some type of faces. Besides it is easy for our security system to
provide checkpoints by uniform lightening and keep the same distance to a
photographed person.


A
fter training RBFNN2 on 250 face images we test it on 190 faces with
very promising results: efficiency


95% and not a single case of wrong
acceptings! 5% of inefficiency occurs due to the only case, when among 10
pictures of the same person used for trai
ning on one picture it was a photo of
this man made an unusual grimace, so the program did not accept namely
that photograph of this man.



However, we are still going to study more in details the idea of applying
wavelets for a considerable face image

compression without loosing important
features, in order to apply then principal component method to wavelet
coefficients obtained on the preprocessing stage.





Conclusion




New RBF
-
like neural network is proposed, which allows to process
raster inform
ation of digitized images ;



A study of the reliability of direct RBFNN2 application to frontal face
data shows the need in data preprocessing by extraction of

principal
components after scaling the data by 2D wavelet transfom;



Wavelet preprocessing result
ing in significant data compression
is still under study;



Corresponding object
-
oriented C++ software is developed to
work with frontal face images recorded by a video
-
camera. The
first results on statistics provided by Cambridge face database
are quite pr
omising.