2.6. How to choose a neural network structure wisely?

haremboingAI and Robotics

Oct 20, 2013 (3 years and 5 months ago)


2.6. How to choose a neural network structure wisely?

(translation by Rafał Opiał,


Irrespectively of brought forward remarks showing that one can reach a goal even by teaching
a non optimally chosen neural network to solve a problem, on
e have to formalize

network structure. And what's more, it is easy to reveal that choosing a

structure that fits
a problem requirements at the beginning, can significantly shorten learning time and improve its
results. That is why I
want to present you some remarks although I know that it will not be a
solution for all kind of neural network construction problems. I feel indebted to give you few
advices because we all know how difficult it is sometimes to chose

solution with
clues. Placing a neural network constructor in situation where he can freely adopt any structure he
wants is similar to dilemma of abecedarian computer engineers that stare confused at system

Press any key...

Yeah, but which is
the any

key tha
t I shall press?!

You may laugh about it but for me often similar is the question I hear from my students and
Doctoral students: okay, but

is this

structure of neural network?

I'll say now few words about common neural network structures. One impo
rtant thing is to
remember that what comes below is not everything about possibilities and more

every researcher
shall be a kind of Demiurge, a creator of new beings because neural networks with different
structures are not completely understood and thus

in this work we need every pair of... cerebral

I will start here with classification of commonly used neural network structures into two
major classes: neural networks without feedback and with it. Neural networks belonging to first

class are often called

while the other in which signals can circuit for
unlimited time are called

Fig. 2.24.
Example structure of

type neural network. Neurons represented by yellow
circles are

connected such way, which make possible transmission signals only from input to output.

The feedforward networks are structures in which there is strictly determined direction of
signal propagation. Signals go from defined input, where data about problem

to solve is
passed into neural network, to output where a network produces result (
. 2.24). This kind
of networks is the most commonly used and the most useful. I will talk about them later in
this chapter and in few that follow.

The recurrent networks

characterize occurrence of feedbacks

(Fig. 2.25)
. In such networks
signals can circuit between neurons for a very long time before it reach a fixed state, but
there are also cases that this kind of neural network does not produce any fixed state.

Fig. 2.25.
Example structure of
type neural network. Connections presented as red arrows
are feedbacks, causing network to be


Study of recurrent networks properties and abilities is more complex that it comes w
feedforward networks, but on the other hand their computational potentials are astonishingly
different than of other types of neural networks. For instance they can solve optimization

that is searching for the best possible solution, this is

almost impossible to do for
feedforward networks.

Among all recurrent networks a special place belongs to these named after John Hopfield. In
Hopfield networks the one and only kind of connection between neurons is feedback (

Fig. 2.26. Hopfie
ld neural network, in which all connections are of feedback type.

Some time ago a true sensation was that a Hopfield network has solved a famous travelling
salesman problem. That circumstance opened a way for Hopfield networks to manage important
te class of computational problems, but this is something I am going to tell you later on.
Despite this sensational breakthrough Hopfield networks did not become as popular as other types
of neural networks so we will tell more about them just later, in ch
apter 11 of this book.

Since building a neural network with feedbacks is much more difficult than feedforward net
and also controlling a network in which there is a lot of simultaneous dynamic processes is also
much more difficult than controlling a networ
k where signals politely and calm go from input to
output, we start with one directional signal flow networks and then will slowly pass to recurrent
nets. If you heard before about the most famous recurrent neural network, a Hopfield network, and
want to k
now it better, you may pass over these chapters and start lecture of chapter 11, or you may
(what I definitely recommend) arm yourself with patience and successively read one chapter after

If we focus on feedforward networks, we may state that the

best and commonly used way to
characterize their structure is a layer model. In this model we assume that neurons are clustered in
sets called layers. Major interlinks exist between neurons belonging to adjacent layers. This kind of
structure was already
discussed in this chapter but it is worth to take a look at it again (
. 2.27).

Fig. 2.27. Laered structure of the simplest neural network

I mentioned this in previous chapter, but it is worth to say again that links between neurons
from adjacent layer
s may be constructed in many different manners (as it wish a constructor),
although the most commonly used is all to all linkage, because we can count on, that learning
process will lead to automatic cut off unneeded connections by setting their coefficien
ts (weights) to