G51
IAI
Introduction to AI
Neural Networks
What is a Neural Network
Computers & Symbols versus Nets & Neurons
(Kevin Gurney)
•
An interconnected assembly of simple processing
elements,
units or nodes
, whose functionality is
loosely based on the animal neuron. The
processing ability of the network is stored in the
inter
-
unit connection strengths, or
weights
,
obtained by a process of adaptation to, or
learning
from, a set of training patterns.
Where can NN Systems Help?
•
Where we can not formulate an
algorithm for the solution.
•
Where we can get lots of examples of
the behaviour we require.
•
Where we need to pick out the structure
from existing data.
Symbolic Approach versus NN
•
What problems can
symbolic approach
solve?
•
…and can not solve?
•
Everyday tasks taken for granted
•
Visual or aural recognition (recognise
handwritten characters, someone`s face
from a new angle,…)
Neural Networks
•
McCulloch & Pitts (1943) are generally
recognised as the designers of the first neural
network
•
Many of their ideas still used today (e.g. many
simple units combine to give increased
computational power and the idea of a
threshold)
•
Hebb (1949) developed the first learning rule
(on the premise that if two neurons were
active at the same time the strength between
them should be increased)
Neural Networks
•
During the 50’s and 60’s many
researchers worked on the perceptron
amidst great excitement.
•
1969 saw the death of neural network
research for about 15 years
–
Minsky &
Papert
•
Only in the mid 80’s (Parker and LeCun)
was interest revived (in fact Werbos
discovered algorithm in 1974)
Biological Neuron
Neural Networks
•
We are born with about 100 billion neurons
•
A neuron may connect to as many as
100,000 other neurons
•
Signals “move” via complex
electrochemical reactions
•
The synapses release a chemical
transmitter
–
the sum of which can cause a
threshold to be reached
–
causing the
neuron to “fire”
•
Synapses can be inhibitory or excitatory
Modeling a Neuron
•
a
j
:Activation value of unit j
•
w
j,i
:Weight on the link from unit j to unit i
•
in
i
:Weighted sum of inputs to unit i
•
a
i
:Activation value of unit i
•
g
:Activation function
Activation Functions
•
Step
t
(x)
=
1 if x >= t, else 0
•
Sign(x)
=
+1 if x >= 0, else
–
1
•
Sigmoid(x)
=
1/(1+e
-
x
)
•
Identity Function
Simple Networks
t = 0.0
y
x
W = 1.5
W = 1
-
1
W = 1
Neural Networks
–
AND NOT
AND NOT Function
-
1
2
X
1
X
2
Y
Threshold(
Y
) = 2
Network Architectures
•
Three different classes of network
architectures
•
single
-
layer feed
-
forward
•
multi
-
layer feed
-
forward
•
recurrent
neurons are organized
in acyclic layers
Network Architectures
•
A feed
-
forward network is acyclic. There are no
cycles in the network as data passes from the inputs
to the outputs, and not vice versa. Once a feed
-
forward network has been trained, its state is fixed
and does not alter as new input data is presented to
it. In other words, it does not have
memory
.
•
A recurrent network can have connections that go
backward from output nodes to input nodes. It can
have arbitrary connections between any nodes. In
this way, a recurrent network’s internal state can
alter as sets of input data are presented to it, and it
can be said to have a memory. This is particularly
useful in solving problems where the solution
depends not just on the current inputs, but on all
previous inputs.
Perceptron
•
Synonym for
Single
-
Layer, Feed
-
Forward Network
•
First Studied in the
50’s
•
Other networks
were known about
but the perceptron
was the only one
capable of learning
and thus all
research was
concentrated in this
area
Perceptron
•
A single weight
only affects one
output so we can
restrict our
investigations to
a model as shown
on the right
•
Notation can be
simpler, i.e.
What can perceptrons
represent?
0,0
0,1
1,0
1,1
0,0
0,1
1,0
1,1
AND
XOR
•
Functions which can be separated in this way are
called
Linearly Separable
•
Only linearly separable functions can be
represented by a perceptron
What can perceptrons
represent?
Linear Separability is also possible in more than 3
dimensions
–
but it is harder to visualise
Neural Networks
-
XOR
2
2
2
2
-
1
-
1
Z
1
Z
2
Y
X
1
X
2
X
1
XOR X
2
= (X
1
AND NOT X
2
) OR (X
2
AND NOT X
1
)
XOR Function
Building a Neural Network
1.
“Select Structure”: Design the way that
the neurons are interconnected
2.
“Select weights”
–
decide the strengths
with which the neurons are
interconnected
•
weights are selected to get a “good match”
to a “training set”
•
“training set”: set of inputs and desired
outputs
•
often use a “learning algorithm”
Learning Algorithms
•
Supervised Learning
•
learn by being presented with pre
-
classified
training data, and modifying the weights of
the connections within their networks to
more accurately classify the training data.
•
Unsupervised Learning
•
learning where there’s no answer given in
the test data.
•
analyse the data in order to understand their
similarities and differences. In this way,
relationships in the data can be found that
may not have been apparent before.
Training a perceptrons
t = 0.0
y
x
-
1
W = 0.3
W =
-
0.4
W = 0.5
Learning
While epoch produces an error
Present network with next inputs
from epoch
Err = T
–
O
If Err <> 0 then
W
j
= W
j
+ LR * I
j
* Err
End If
End While
Learning
While epoch produces an error
Present network with next inputs
from epoch
Err = T
–
O
If Err <> 0 then
W
j
= W
j
+ LR * I
j
* Err
End If
End While
Epoch
:
Presentation of the entire training set to
the neural network.
In the case of the AND function an epoch
consists of four sets of inputs being
presented to the network (i.e. [0,0],
[0,1], [1,0], [1,1])
Learning
While epoch produces an error
Present network with next inputs
from epoch
Err = T
–
O
If Err <> 0 then
W
j
= W
j
+ LR * I
j
* Err
End If
End While
Training Value, T
: When we are training a
network we not only present it with the
input but also with a value that we require
the network to produce. For example, if
we present the network with [1,1] for the
AND function the training value will be 1
Learning
While epoch produces an error
Present network with next inputs
from epoch
Err = T
–
O
If Err <> 0 then
W
j
= W
j
+ LR * I
j
* Err
End If
End While
Error, Err
: The error value is the amount by
which the value output by the network
differs from the training value. For
example, if we required the network to
output 0 and it output a 1, then Err =
-
1
Learning
While epoch produces an error
Present network with next inputs
from epoch
Err = T
–
O
If Err <> 0 then
W
j
= W
j
+ LR * I
j
* Err
End If
End While
Output from Neuron, O
: The output value from the neuron
Ij
: Inputs being presented to the neuron
Wj
: Weight from input neuron (I
j
) to the output neuron
LR
: The learning rate. This dictates how quickly the network
converges. It is set by a matter of experimentation.
It is typically 0.1
Property Appraiser
Voice Recognition
Steve
David
0.01
0.99
0.99
0.01
Network Architecture
Feedforward Network
60 inputs (one for each
frequency bin);
6 hidden;
2 output (0
-
1 for
“Steve”, 1
-
0 for
“David”)
And Finally….
“If the brain were so
simple that we could
understand it then we’d
be so simple that we
couldn’t”
Lyall Watson
Summary of Neural Network
•
Definitions
•
Modelling Functions
‒
weights
‒
activation Functions
•
Simple Networks
‒
perceptron
•
Network Architecture
•
Learning Algorithm
‒
delta rule
Acknowledgements
Most of the lecture slides are
adapted from the same module
taught in Nottingham UK
by
Professor Graham Kendall,
Dr. Rong Qu
and
Dr. Andrew Parker
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο