Competitive Networks - the Kohonen Self-organising Map

glibdoadingAI and Robotics

Oct 20, 2013 (3 years and 11 months ago)

101 views

Competitive Networks
-

the Kohonen Self
-
organising Map



Competitive neural networks represent a type of ANN model in which neurons in the output layer
compete with each other to determine a winner. The winner indicates which prototype pattern is most
repr
esentative of (most similar to) the input pattern.


The competition among output layer neurons is implemented by
lateral inhibition

-

a set of negative
connections between the neurons. The most well known among this paradigm of ANNs is the Self
-
organising

Map (SOM), also known as the Kohonen net.






Lea牮楮g⁩渠扩潬潧ica氠ly獴敭猠
-

瑨攠獥汦
-
潲条湩獩湧⁰牡摩gm




䅲c桩hec瑵te映瑨⁋潨湥渠⁎n瑷潲t




周T⁋潨
潮o渠ne瑷潲t⁩渠佰n牡瑩潮




呲a楮i湧⁴e⁋潨潮渠ne瑷潲t




䅰灬楣A瑩潮猠潦⁴桥⁋潨湥渠乥瑷潲t



Lea牮楮g⁩渠扩潬潧ica氠ly獴敭猠
-

瑨攠獥汦
-
潲条湩獩湧⁰牡摩gm


周T⁴ype映汥a牮楮g⁵瑩汩獥搠
楮畬u楬aye爠re牣e灴牯湳⁲e煵楲q猠瑨e⁣潲牥c琠牥獰潮獥⁴漠扥⁰牯楤e搠
摵物湧⁴a楮i湧
獵e牶楳e搠瑲楮i湧⤮n


B楯i潧楣a氠ly獴e浳⁤m獰污s瑨t猠sype映汥a牮楮gⰠ,畴⁴桥y⁡牥⁡汳漠ca灡扬e映汥a牮楮g⁢y⁴桥浳汶敳l
-

睩瑨潵⁡⁳異牶楳潲⁳桯睩湧⁴桥c潲oect

牥獰潮se
畮獵灥牶楳r搠ea牮楮g).


䄠湥畲u氠湥瑷潲欠o楴栠h獩s楬a爠ra灡扩b楴y⁩猠a汬e搠
self
-
organising system

because during training,
the network changes its weights to learn appropriate associations, without any right answers being
provided.


The pro
pagation of biological neural activation via axons can be modelled using a Mexican hat
function:




Cells close to the active cell have excitatory links. The strengths of the links drop off with distance
and then turn inh
ibitory. The Kohonen neural network also uses only locally connected neurons and
restricts the adjustment of weight values to localised "neighbourhoods".

Artificial Neural Networks

Kohonen Network



2


Architecture of the Kohonen Network


The Kohonen network consists of an input layer, which distribut
es the inputs to each node in a
second layer, the so
-
called
competitive layer
.

Each of the nodes on this layer acts as an output node.


Each neuron in the competitive layer is connected to other neurons in its neighbourhood and
feedback is restricted to n
eighbours through these lateral connections.




Neurons in the competitive layer have excitatory connections to immediate neighbours and inhibitory
connections to more distant neurons.


All neurons in the competitive la
yer receive a mixture of excitatory and inhibitory signals from the
input layer neurons and from other competitive layer neurons.



The Kohonen Network in Operation


As an input pattern is presented, some of the neurons are sufficiently activated to produ
ce outputs
which are fed back to other neurons in their neighbourhoods.


The node with the weight vector closest to the input pattern vector (the so
-
called "winning node")
produces the largest output. During training, input weights of the winning neuron a
nd its neighbours
are adjusted to make them resemble the input pattern even more closely.


At the completion of training, the winning node ends up with its weight vector aligned with the input
pattern and produces the strongest output whenever that particu
lar pattern is presented.


The nodes in the winning node's neighbourhood also have their weights modified to settle down to an
average representation of that pattern class. As a result, unseen patterns belonging to that class are
also classified correctly
(generalisation).


The
m

neighbourhoods, corresponding to the
m

possible pattern classes are said to form a
topological
map
representing the patterns.


Artificial Neural Networks

Kohonen Network



3

The initial size of the neighbourhood mentioned above and the fixed values of excitatory (positive)
and

inhibitory (negative) weights to neurons in the neighbourhood are among the design decisions to
be made.


Derivation of the learning rule for the Kohonen net


The sum squared error for pattern
p

for all output layer neurons can be written as




(1)

where x
j
p

is the
i
th

component of pattern

p

for neuron
j
. The summation is done over all
j

neurons.


Any change

w
ij

in the weight is expected to cause a reduction in error
E
p
.

Now
E
p
is a function of all the weights, so its rate of chang
e with respect to any one weight value
w
ij

has to be measured by calculating its partial derivative with respect to
w
ij
.
(That is why we have the
small delta

, instead of
d

in the following equation for the derivative)




(2)

where


is a constant of proportionality.

Now we have to calculate the partial derivative of
E
p
. Using (1):




(3)

Combining (2) and (3), we get







Artificial Neural Networks

Kohonen Network



4

Training the Kohonen Network



The Kohonen Algorithm


1.

Initia
lise weights




Initialise weights from
N
inputs to the nodes to small random
values. Set the initial radius of the neighbourhood.


2.

Present new input

x
o
(t),x
1
(t),x2(t), ..x
n
-
1
(t),
where
x
i
(t)
is
the input to node

i

at time

t.



3.

Compute distances to

all nodes



Compute distances
d
j

between the input and each output node
j
using







N
-
1





d
j

=

(x
i
(t)
-

w
ij(t))
2










i


where
x
i
(t) is the input to node
i

at time

t
and
w
ij
(t)

is the
weight from input node
i

to output node
j

at time
t.


4. Se
lect output node with minimum distance



Select output node
j
*
as the output node with minimum
d
j
.


5.

Update weights to node
j
*

and neighbours




Weights updated for node
j
*

and all nodes in the neighbourhood
defined by
N
j
*(t).

New weights are




w
ij
(
t +
1
) = w
ij
(
t
) +

(t)(x
i
(t)
-

w
ij
(
t
))




for

j

in
N
j
*, 0


i


N
-
1


The term


(t)
is a gain term

0






1.

Both



and

N
j
*(t)
decrease
with time.


6.

Repeat by going to step 2


Artificial Neural Networks

Kohonen Network



5

Learning Vector Quantisation (LVQ)


A supervised learning technique

for optimising performance of Kohonen networks, e.g., when new
vectors are to be added.

Weight adjustments
-

For a correctly classified input:



n
w
(t+1) = n
w
(t) +

(t)[x(t)
-

n
w
(t)]


For incorrect classification:



n
w
(t+1) = n
w
(t) +

(t)[x(t)
-

n
w
(t)]



Training issues in Kohonen Neural Nets


Vector normalisation

To make vector comparison independent of magnitudes and dependent on orientation only, the
vectors are normalised by dividing them by their magnitudes. This also helps to reduce training
time.


Weight initialisation

A random distribution of initial weights may not be optimal, resulting in sparsely populated trainable
nodes and poor classification performance.

Possible remedies:


a.

Initialisation of weights to same value and lumping of inp
ut vectors to similar orientation.
This increases likelihood of all nodes being closer to pattern vector. Inputs slowly returned to
original orientation with training.


b.

Addition of random noise to inputs to distribute vectors over a larger pattern space
.


c.

Using a large initial neighbourhood changing slowly.


Reducing neighbourhood size

Should be decreasing linearly with time (iterations).

Neighbourhood shape may vary to suit application
-

e.g., circular or hexagonal instead of
rectangular.



Artificial Neural Networks

Kohonen Network



6

Applicat
ions of the Kohonen Network



The Kohonen network is being used in speech and image processing and has potential for statistical
and database applications.


Speech Processing
-

Kohonen's Phonetic Typewriter




Speaker
-
independent, unlimited voca
bulary continuous speech recognition remains yet to be
achieved with conventional techniques. The problem is made difficult by the same word being
spoken with different pronunciations, levels of loudness, emphases and background noise. Apart
from analysing

individual units of sound (phonemes), the human brain uses stored speech patterns,
context and other clues to recognise speech successfully.



Kohonen's Phonetic Typewriter combines digital signal processing techniques and the use of
a rule base with a
Kohonen network to achieve 92
-
97% accuracy with multiple speaker unlimited
vocabulary Finnish and Japanese speech.











Reference:

"Neural Computing" Beale & Jackson.