Neural Networks

G51
IAI

Introduction to AI

Neural Networks

What is a Neural Network

Computers & Symbols versus Nets & Neurons

(Kevin Gurney)


•
An interconnected assembly of simple processing
elements,
units or nodes
, whose functionality is
loosely based on the animal neuron. The
processing ability of the network is stored in the
inter
-
unit connection strengths, or
weights
,
obtained by a process of adaptation to, or
learning

from, a set of training patterns.

Where can NN Systems Help?

•
Where we can not formulate an
algorithm for the solution.

•
Where we can get lots of examples of
the behaviour we require.

•
Where we need to pick out the structure
from existing data.

Symbolic Approach versus NN

•
What problems can
symbolic approach

solve?

•
…and can not solve?


•
Everyday tasks taken for granted

•
Visual or aural recognition (recognise
handwritten characters, someone`s face
from a new angle,…)

Neural Networks

•
McCulloch & Pitts (1943) are generally
recognised as the designers of the first neural
network

•
Many of their ideas still used today (e.g. many
simple units combine to give increased
computational power and the idea of a
threshold)

•
Hebb (1949) developed the first learning rule
(on the premise that if two neurons were
active at the same time the strength between
them should be increased)

Neural Networks

•
During the 50’s and 60’s many
researchers worked on the perceptron
amidst great excitement.

•
1969 saw the death of neural network
research for about 15 years
–

Minsky &
Papert

•
Only in the mid 80’s (Parker and LeCun)
was interest revived (in fact Werbos
discovered algorithm in 1974)


Biological Neuron

Neural Networks

•
We are born with about 100 billion neurons

•
A neuron may connect to as many as
100,000 other neurons

•
Signals “move” via complex
electrochemical reactions

•
The synapses release a chemical
transmitter
–

the sum of which can cause a
threshold to be reached
–

causing the
neuron to “fire”

•
Synapses can be inhibitory or excitatory


Modeling a Neuron

•
a
j

:Activation value of unit j

•
w
j,i

:Weight on the link from unit j to unit i

•
in
i

:Weighted sum of inputs to unit i

•
a
i

:Activation value of unit i

•
g

:Activation function

Activation Functions

•
Step
t
(x)

=

1 if x >= t, else 0

•
Sign(x)

=

+1 if x >= 0, else
–
1

•
Sigmoid(x)

=

1/(1+e
-
x
)

•
Identity Function

Simple Networks

t = 0.0

y

x

W = 1.5

W = 1

-
1

W = 1

Neural Networks
–

AND NOT

AND NOT Function

-
1

2

X
1

X
2

Y

Threshold(
Y
) = 2

Network Architectures

•
Three different classes of network
architectures

•
single
-
layer feed
-
forward

•
multi
-
layer feed
-
forward

•
recurrent





neurons are organized

in acyclic layers

Network Architectures

•
A feed
-
forward network is acyclic. There are no
cycles in the network as data passes from the inputs
to the outputs, and not vice versa. Once a feed
-
forward network has been trained, its state is fixed
and does not alter as new input data is presented to
it. In other words, it does not have
memory
.

•
A recurrent network can have connections that go
backward from output nodes to input nodes. It can
have arbitrary connections between any nodes. In
this way, a recurrent network’s internal state can
alter as sets of input data are presented to it, and it
can be said to have a memory. This is particularly
useful in solving problems where the solution
depends not just on the current inputs, but on all
previous inputs.

Perceptron

•
Synonym for
Single
-
Layer, Feed
-
Forward Network

•
First Studied in the
50’s

•
Other networks
were known about
but the perceptron
was the only one
capable of learning
and thus all
research was
concentrated in this
area

Perceptron

•
A single weight
only affects one
output so we can
restrict our
investigations to
a model as shown
on the right

•
Notation can be
simpler, i.e.


What can perceptrons
represent?

0,0

0,1

1,0

1,1

0,0

0,1

1,0

1,1

AND

XOR

•
Functions which can be separated in this way are
called
Linearly Separable


•
Only linearly separable functions can be
represented by a perceptron

What can perceptrons
represent?

Linear Separability is also possible in more than 3
dimensions
–

but it is harder to visualise

Neural Networks
-

XOR

2

2

2

2

-
1

-
1

Z
1

Z
2

Y

X
1

X
2

X
1

XOR X
2

= (X
1

AND NOT X
2
) OR (X
2

AND NOT X
1
)

XOR Function

Building a Neural Network

1.
“Select Structure”: Design the way that
the neurons are interconnected

2.
“Select weights”
–

decide the strengths
with which the neurons are
interconnected

•
weights are selected to get a “good match”
to a “training set”

•
“training set”: set of inputs and desired
outputs

•
often use a “learning algorithm”

Learning Algorithms

•
Supervised Learning

•
learn by being presented with pre
-
classified
training data, and modifying the weights of
the connections within their networks to
more accurately classify the training data.

•
Unsupervised Learning

•
learning where there’s no answer given in
the test data.

•
analyse the data in order to understand their
similarities and differences. In this way,
relationships in the data can be found that
may not have been apparent before.

Training a perceptrons

t = 0.0

y

x

-
1

W = 0.3

W =
-
0.4

W = 0.5

Learning

While epoch produces an error

Present network with next inputs
from epoch

Err = T
–

O

If Err <> 0 then

W
j

= W
j

+ LR * I
j

* Err

End If

End While



Learning

While epoch produces an error

Present network with next inputs
from epoch

Err = T
–

O

If Err <> 0 then

W
j

= W
j

+ LR * I
j

* Err

End If

End While

Epoch
:
Presentation of the entire training set to
the neural network.

In the case of the AND function an epoch
consists of four sets of inputs being
presented to the network (i.e. [0,0],
[0,1], [1,0], [1,1])

Learning

While epoch produces an error

Present network with next inputs
from epoch

Err = T
–

O

If Err <> 0 then

W
j

= W
j

+ LR * I
j

* Err

End If

End While

Training Value, T

: When we are training a
network we not only present it with the
input but also with a value that we require
the network to produce. For example, if
we present the network with [1,1] for the
AND function the training value will be 1

Learning

While epoch produces an error

Present network with next inputs
from epoch

Err = T
–

O

If Err <> 0 then

W
j

= W
j

+ LR * I
j

* Err

End If

End While

Error, Err

: The error value is the amount by
which the value output by the network
differs from the training value. For
example, if we required the network to
output 0 and it output a 1, then Err =
-
1

Learning

While epoch produces an error

Present network with next inputs
from epoch

Err = T
–

O

If Err <> 0 then

W
j

= W
j

+ LR * I
j

* Err

End If

End While

Output from Neuron, O

: The output value from the neuron

Ij
: Inputs being presented to the neuron

Wj

: Weight from input neuron (I
j
) to the output neuron

LR
: The learning rate. This dictates how quickly the network
converges. It is set by a matter of experimentation.
It is typically 0.1

Property Appraiser

Voice Recognition

Steve

David

0.01

0.99

0.99

0.01

Network Architecture


Feedforward Network


60 inputs (one for each
frequency bin);


6 hidden;


2 output (0
-
1 for
“Steve”, 1
-
0 for
“David”)

And Finally….

“If the brain were so
simple that we could
understand it then we’d
be so simple that we
couldn’t”

Lyall Watson

Summary of Neural Network

•
Definitions

•
Modelling Functions

‒
weights

‒
activation Functions

•
Simple Networks

‒
perceptron

•
Network Architecture

•
Learning Algorithm

‒
delta rule

Acknowledgements

Most of the lecture slides are

adapted from the same module

taught in Nottingham UK

by

Professor Graham Kendall,

Dr. Rong Qu

and


Dr. Andrew Parker