Lecture 11: Neural Nets - Cwa.mdx.ac.uk

cartcletchΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 3 χρόνια και 7 μήνες)

82 εμφανίσεις

Neural Nets

Symbolic and sub
-
symbolic
artificial intelligence


The various conventional knowledge
representation techniques that have
been mentioned so far can be labelled
symbolic artificial intelligence
.

Symbolic and sub
-
symbolic
artificial intelligence


The elements in the knowledge
representation
-



production rules, frames, semantic net
nodes and arcs, objects, or whatever



-

act as symbols, with each element
corresponding to a similar element in
the real
-
world knowledge.


Manipulations of these elements
correspond to the manipulation of
elements of real
-
world knowledge.


Symbolic and sub
-
symbolic
artificial intelligence


An alternative set of approaches, which
have recently become popular, are
known as
sub
-
symbolic AI
.


Symbolic and sub
-
symbolic
artificial intelligence


Here, the real
-
world knowledge is
dispersed among the various elements
of the representation


Only by operating on the representation
as a whole can you retrieve or change
the knowledge it contains.

Symbolic and sub
-
symbolic
artificial intelligence


The two main branches of sub
-
symbolic
AI are


neural nets

(also known as
neural



networks
, or
ANNs
, standing for


artificial neural nets)


and


genetic algorithms
.


Symbolic and sub
-
symbolic
artificial intelligence


The term
connectionism

is used to mean
roughly the same as
the study of neural
nets.


The biological inspiration for
artificial neural nets


Neural networks are an attempt to mimic
the reasoning, or information
processing, to be found in the nervous
tissue of humans and other animals.

The biological inspiration for
artificial neural nets


Such nervous tissue consists of large
numbers (perhaps 100 billion in a typical
human brain) of
neurons

(nerve cells),
connected together by fibres called
axons
and
dendrites

to form networks,
which process nerve impulses.

One sort
of neuron
-

a
pyramidal
cell

A cluster of
neurons,
showing how
the axon from
one connects to
the dendrites of
others.

Apical
dendrites

Basal
dendrites

Cell
body

Axon

Synapses

The biological inspiration for
artificial neural nets


The neuron acts as a signal processing
device:


the dendrites act as inputs,


the axon acts as an output,


a connection between one of these fibres
and an adjacent cell
-

known as a
synapse

-

may be inhibitory or excitatory, i.e. may
tend to cause the next cell to 'fire', or tend
to stop it 'firing'.


The biological inspiration for
artificial neural nets


Obviously, neurones are extremely
small, and made of living tissue rather
than the metals and other inorganic
substances that make up electronic
circuits.

The biological inspiration for
artificial neural nets


The signals that pass along nerve fibres
are
electrochemical

in nature, unlike the
electrical signals in a computer.


The synapses which connect one
neuron to another use chemicals
-
neurotransmitters

-

to transmit signals.

The biological inspiration for
artificial neural nets


Drugs which affect the brain typically do
so by altering the chemistry of the
synapses, making the synapses for a
whole group of neurons either more
efficient or less efficient.


The biological inspiration for
artificial neural nets


As a result, there are some important
differences between neurons, and the
individual processing elements in computers
(transistor
-
based switches):


Neurons are far slower than artificial neurons
-

neurodes
-

but far more efficient in energy
terms.

The biological inspiration for
artificial neural nets


Brain tissue can do what it does (think,
remember, perceive, control bodies etc) partly
because of the electrochemical signals that it
processes, and partly because of the chemical
messages.


Artificial neural nets imitate the first of these, but
not the second.


The biological inspiration for
artificial neural nets


The neurons in a brain work
in parallel

to
perform their symbol processing (i.e., the
individual neurons are operating
simultaneously. This is quite unlike a
conventional computer, where the
programming steps are performed one after
the other.

The biological inspiration for
artificial neural nets


The brains of all animals of any complexity
consist of a number of these networks of
neurons, each network specialised for a
particular task.


There are many different types of neuron
(over 100) in the human brain.


Artificial neural nets


Note that neural nets are
inspired

by the
organisation of brain tissue, but the
resemblance is not necessarily very close.


Claims that a particular type of artificial
neural net has been shown to demonstrate
some property, and that this 'explains' the
working of the human brain, should be
treated with caution.

Artificial neural nets


Note that a neural net is ideally
implemented on a parallel computer (e.g.
a connection machine).


However, since these are not widely used,
most neural net research, and most
commercial neural net packages, simulate
parallel processing on a conventional
computer.


Neurodes


Neural nets are constructed out of artificial
neurones (
neurodes
). The characteristics of
these are:


each has one or more inputs (typically several).


Each input will have a
weight
, which measures
how effective
that input

is at firing the neurode
as a whole. These weights may be positive (i.e.
increasing the chance that the neurode will fire)
or negative (i.e. decreasing the chance that the
neurode will fire).


Neurodes


These weights may change as the neural
net operates.


Inputs may come from the outside
environment, or from other neurodes


each has one output (but this output may
branch, and go to several locations).


an output may go to the outside
environment, or to another neurode.


Neurodes


More properties of neurodes:


each is characterised by a
summation
function

and a
transformation function
.



Neurodes


The summation function is a technique
for finding the weighted average of all
the inputs.


These vary in complexity, according to
the type of neural net.


The simplest approach is to multiply
each input value by its weight and add
up all these figures.



Neurodes


The transformation function is a
technique for determining the output of
the neurode, given the combined
inputs.


Again, these vary in complexity, according
to the type of neural net.


The simplest approach is to have a
particular threshold value
-

but the
sigmoid
function
,

to be discussed later, is more
common.


Neurodes


"[an artificial neuron is] a unit that
accepts a bunch of numbers, and learns
to respond by producing a number of its
own."

Aleksander & Morton, 1993.


The

functions of a typical neurode
. a
i

represents the activation of the neurode. This
is also the output from the neurode

Input links

Output links

W
j,i


a
j


a
i


Transformation
function

Summation
function




Activation

a
i

Artificial neural nets


Different sorts of transformation function
are available, and are favoured for
different designs of ANN.


The three most common choices are


the step function,


the sign function,


and


the sigmoid function



Step function

Sign function

Sigmoid function

a
i

a
i

a
i

inp
i

inp
i

inp
i

+1

+1

+1

-
1

+1

-
1

Artificial neural nets


As far as networks are concerned, they
may or may not be organised into
layers
.


Usually, they are.

Artificial neural nets


Networks organised into layers may be
subdivided into those that simply have
an
input

layer

and an
output

layer
, and
those that have one or more
intermediate layers, known as
hidden
layers
.






































A neural net with one input layer and one
output layer (both containing 6 neurodes)





































A neural net with one input layer, one hidden
layer, and one output layer (each containing 6
neurodes)



















How networks are used


Each input in a network corresponds to
a single attribute of a pattern or
collection of data.


The data must be
numerical
: qualitative
aspects of the data, or graphical data,
must be pre
-
processed to convert it into
numbers before the network can deal
with it.


How networks are used


Thus, an image is converted into pixels
(a number could be converted into a 6x8
dot matrix, and provide the input to 48
input neurodes).


Similarly, a fragment of sound would
have to be digitised, and a set of
commercial decision criteria would have
to be coded before the net could deal
with them.


How networks are used


Similarly, values must be assigned to the
outputs from the output nodes before they
can be treated as 'the solution' to whatever
problem the network was given.


How networks are used


Neural nets are not programmed in the
conventional way (we do not have
techniques for 'hand
-
programming' a
net).


Instead, they go through a learning
phase, during which the weights are
modified. After which the weights are
clamped, and the system is ready to
perform.


How networks are used


Learning involves


entering examples of data as the
input,


using some appropriate algorithm to
modify the weights so that the output
changes in the desired direction,


repeating this until the desired output
is achieved.


Example of

supervised
learning in a simple neural net



Suppose we have a net consisting of a
single neurode.


The summation function is the standard
version.


The transformation function is a step
function.

Example of

supervised
learning in a simple neural net



There are two inputs and one output,


We wish to teach this net the logical
INCLUSIVE OR function, i.e.


if the values of both the inputs is 0, the
output should be 0;


if the value of either or both the inputs is
1, the output should be 1.

Example of

supervised
learning in a simple neural net


We will represent the values of the two
inputs as X
1

and X
2
, the desired output as
Z, the weights on the two inputs as W
1

and
W
2
, the actual output as Y.

Example of

supervised
learning in a simple neural net

Input Weight Desired output

X
1

X
2

W
1

W
2

Z

Actual output

Y

Example of

supervised
learning in a simple neural net


The learning process involves repeated
applying the four possible patterns of input:





X
1

X
2

Z





0

0

0




0

1

1




1

0

1




1

1

1

Example of

supervised
learning in a simple neural net



The two weights W
1

and W
2

are
initially set to random values. Each time
a set of inputs is applied, a value
D

is
calculated as



D

= Z
-

Y


(the difference between what you got
and what you wanted)


and the weights are adjusted.

Example of

supervised
learning in a simple neural net


The new weight, V for a particular input i is
given by



V
i

= W
i

+
a

*
D

* X
i


where
a

is a parameter which determines
how much the weights are allowed to
fluctuate in a particular cycle, and hence
how quickly learning takes place.


An actual learning sequence might be as
follows:


Example of

supervised
learning in a simple neural net

a

= 0.2


threshold = 0.5

Iter
-

ation

X
1

X
2

Z

W
1

W
2

Y

D

V
1

V
2

_______________________________________

1


0

0

0

0
.
1

0
.
3

0

0
.
0

0
.
1

0
.
3



0

1

1

0
.
1

0
.
3

0

1
.
0

0
.
1

0
.
5



1

0

1

0
.
1

0
.
5

0

1
.
0

0
.
3

0
.
5



1

1

1

0
.
3

0
.
5

1

0
.
0

0
.
3

0
.
5


Example of

supervised
learning in a simple neural net

a

= 0.2


threshold = 0.5

Iter
-

ation

X
1

X
2

Z

W
1

W
2

Y

D

V
1

V
2

_______________________________________

2


0

0

0

0
.
3

0
.
5

0

0
.
0

0
.
3

0
.
5



0

1

1

0
.
3

0
.
5

0

1
.
0

0
.
3

0
.
7



1

0

1

0
.
3

0
.
7

0

1
.
0

0
.
5

0
.
7



1

1

1

0
.
5

0
.
7

1

0
.
0

0
.
5

0
.
7

Example of

supervised
learning in a simple neural net

a

= 0.2


threshold = 0.5

Iter
-

ation

X
1

X
2

Z

W
1

W
2

Y

D

V
1

V
2

_______________________________________

3


0

0

0

0
.
5

0
.
7

0

0
.
0

0
.
5

0
.
7



0

1

1

0
.
5

0
.
7

1

0
.
0

0
.
5

0
.
7



1

0

1

0
.
5

0
.
7

0

1
.
0

0
.
7

0
.
7



1

1

1

0
.
7

0
.
7

1

0
.
0

0
.
7

0
.
7

Example of

supervised
learning in a simple neural net

a

= 0.2


threshold = 0.5

Iter
-

ation

X
1

X
2

Z

W
1

W
2

Y

D

V
1

V
2

_______________________________________

4


0

0

0

0
.
7

0
.
7

0

0
.
0

0
.
7

0
.
7



0

1

1

0
.
7

0
.
7

1

0
.
0

0
.
7

0
.
7



1

0

1

0
.
7

0
.
7

1

0
.
0

0
.
7

0
.
7



1

1

1

0
.
7

0
.
7

1

0
.
0

0
.
7

0
.
7

-

no errors detected for an entire iteration: learning halts.

Human
-
like features of neural
nets


Distributed representation

-

'memories'
are stored as a pattern of activation,
distributed over a set of elements.


'Memories' can be superimposed;
different memories are represented by
different patterns over the same
elements.


Human
-
like features of neural
nets


Distributed asynchronous control

-

each
element makes its own decisions, and
these add up to a global solution.


Human
-
like features of neural
nets


Content
-
addressable memory

-

a
number of patterns can be stored in a
network and, to retrieve a pattern, we
need only specify a portion of it; the
network automatically finds the best
match.


Human
-
like features of neural
nets


Fault tolerance

-

if a few processing
elements fail, the network will still
function correctly.


Human
-
like features of neural
nets


Graceful degradation

-

failure of the net
is progressive, rather than catastrophic.

Human
-
like features of neural
nets


Collectively, the network is a little like a
committee, coming to a joint decision on a
particular question.


And like a committee, the absence of one
or more members/neurodes does not
necessarily prevent the committee/network
from functioning (or even from coming to
the same decisions).

Human
-
like features of neural
nets


The failure of a small proportion of its
neurodes, or its links, does not cause a
catastrophic failure, merely a reduction in
performance.


Compare this with a conventional program,
where the loss of a vital line of code
would

cause such a failure.


Human
-
like features of neural
nets


Automatic generalisation

-

similar or
related facts are automatically stored as
related patterns of activation.

Human
-
like features of neural
nets


'Fuzzy' mapping

-

similarities (rather
than strict inclusion or exclusion) are
represented in connectionist models.
This enables human
-
like interpretation
of vague and unclear concepts.


Strengths & weaknesses of
neural nets


Connectionism seems particularly
promising for


learning
, in poorly structured and
unsupervised domains.


low
-
level tasks

such as vision, speech
recognition, handwriting recognition.


Strengths & weaknesses of
neural nets


Connectionism seems rather
unpromising for


highly
-
structured domains

such as
chess playing, theorem proving,
maths, planning.


domains where it is desirable to

understand

how the system is solving
the problem
-

expert systems, safety
-
critical systems. Neural nets are
essentially inscrutable.


Strengths & weaknesses of
neural nets


This suggests that, as a knowledge
acquisition tool, connectionism would be
useful for:


pattern recognition,


learning,


classification,


generalisation,


abstraction,


interpretation of incomplete data.


Strengths & weaknesses of
neural nets


As a basis for a decision support
systems:


optimisation and resource allocation