2.6. How to choose a neural network structure wisely?

haremboingΤεχνίτη Νοημοσύνη και Ρομποτική

20 Οκτ 2013 (πριν από 4 χρόνια και 22 μέρες)

110 εμφανίσεις

2.6. How to choose a neural network structure wisely?

(translation by Rafał Opiał,

rafal.opial
@op.pl
)


Irrespectively of brought forward remarks showing that one can reach a goal even by teaching
a non optimally chosen neural network to solve a problem, on
e have to formalize
some

neural
network structure. And what's more, it is easy to reveal that choosing a
reasonable

structure that fits
a problem requirements at the beginning, can significantly shorten learning time and improve its
results. That is why I
want to present you some remarks although I know that it will not be a
solution for all kind of neural network construction problems. I feel indebted to give you few
advices because we all know how difficult it is sometimes to chose
whichever

solution with
out
clues. Placing a neural network constructor in situation where he can freely adopt any structure he
wants is similar to dilemma of abecedarian computer engineers that stare confused at system
message:

Press any key...

Yeah, but which is
the any

key tha
t I shall press?!

You may laugh about it but for me often similar is the question I hear from my students and
Doctoral students: okay, but
what

is this
any

structure of neural network?

I'll say now few words about common neural network structures. One impo
rtant thing is to
remember that what comes below is not everything about possibilities and more


every researcher
shall be a kind of Demiurge, a creator of new beings because neural networks with different
structures are not completely understood and thus

in this work we need every pair of... cerebral
hemispheres.

I will start here with classification of commonly used neural network structures into two
major classes: neural networks without feedback and with it. Neural networks belonging to first
mentioned

class are often called
feedforward

while the other in which signals can circuit for
unlimited time are called
recurrent
.


Fig. 2.24.
Example structure of
feedforward

type neural network. Neurons represented by yellow
circles are

connected such way, which make possible transmission signals only from input to output.



The feedforward networks are structures in which there is strictly determined direction of
signal propagation. Signals go from defined input, where data about problem

to solve is
passed into neural network, to output where a network produces result (
Fig
. 2.24). This kind
of networks is the most commonly used and the most useful. I will talk about them later in
this chapter and in few that follow.



The recurrent networks

characterize occurrence of feedbacks

(Fig. 2.25)
. In such networks
signals can circuit between neurons for a very long time before it reach a fixed state, but
there are also cases that this kind of neural network does not produce any fixed state.


Fig. 2.25.
Example structure of
recurrent
type neural network. Connections presented as red arrows
are feedbacks, causing network to be
recurrent

one.



Study of recurrent networks properties and abilities is more complex that it comes w
ith
feedforward networks, but on the other hand their computational potentials are astonishingly
different than of other types of neural networks. For instance they can solve optimization
problems


that is searching for the best possible solution, this is

almost impossible to do for
feedforward networks.



Among all recurrent networks a special place belongs to these named after John Hopfield. In
Hopfield networks the one and only kind of connection between neurons is feedback (
Fig
.
2.26).

Fig. 2.26. Hopfie
ld neural network, in which all connections are of feedback type.

Some time ago a true sensation was that a Hopfield network has solved a famous travelling
salesman problem. That circumstance opened a way for Hopfield networks to manage important
NP
-
comple
te class of computational problems, but this is something I am going to tell you later on.
Despite this sensational breakthrough Hopfield networks did not become as popular as other types
of neural networks so we will tell more about them just later, in ch
apter 11 of this book.

Since building a neural network with feedbacks is much more difficult than feedforward net
and also controlling a network in which there is a lot of simultaneous dynamic processes is also
much more difficult than controlling a networ
k where signals politely and calm go from input to
output, we start with one directional signal flow networks and then will slowly pass to recurrent
nets. If you heard before about the most famous recurrent neural network, a Hopfield network, and
want to k
now it better, you may pass over these chapters and start lecture of chapter 11, or you may
(what I definitely recommend) arm yourself with patience and successively read one chapter after
another.

If we focus on feedforward networks, we may state that the

best and commonly used way to
characterize their structure is a layer model. In this model we assume that neurons are clustered in
sets called layers. Major interlinks exist between neurons belonging to adjacent layers. This kind of
structure was already
discussed in this chapter but it is worth to take a look at it again (
Fig
. 2.27).


Fig. 2.27. Laered structure of the simplest neural network

I mentioned this in previous chapter, but it is worth to say again that links between neurons
from adjacent layer
s may be constructed in many different manners (as it wish a constructor),
although the most commonly used is all to all linkage, because we can count on, that learning
process will lead to automatic cut off unneeded connections by setting their coefficien
ts (weights) to
zero.