Introduction to Neural

clangedbivalveAI and Robotics

Oct 19, 2013 (3 years and 11 months ago)

83 views

Introduction to Neural
Networks


Andy Philippides


Centre for Computational Neuroscience and
Robotics (CCNR)

School of Cognitive and
Computing Sciences/School of Biological Sciences


andrewop@cogs.susx.ac.uk

Spring 2003


Lectures
--

2 per week


Time Day Place



12:30
-

1:20 Mon Arun
-

401


11:30
-

12:20 Wed Arun
-

401

Seminar


1 per week

Group 1 3


3.50 Mon Pev1 2D4


Group 2 4


4.50 Mon Pev1 2D4


Group 3 2


2.50 Fri Arun 404B

Group 4 3


3.50 Fri Arun 404B


Office hour: Friday12.30
-
1.30, BIOLS room 3D10

Lecture will be available online soon

Today’s Topics:






Course summary



Components of an artificial neural
network



A little bit math



Single artificial neuron




The course will introduce the theory of several variants of
artificial neural networks (ANNs) discuss how they are
used/trained in practice

Ideas will be illustrated using the example of ANNs used for
function approximation

Very common use of ANNs and also shows the major
concepts nicely. Idea:


Course Summary

Pre
-
Processing

Post
-
Processing

Neural Net model
+ training method

Course Summary

Data

Function
approx

[Will not specifically be using NNs as brain models (Computational
Neuroscience)]

Topics covered


1. Introduction to neural networks

2. Basic concepts for network training

3. Single layer perceptron

4. Probability density estimation

5+6. Multilayer perceptron

7+8. Radial Basis Function networks

9+10. Support Vector machines

11+12. Pre
-
processing + Competitve Learning

13+14. Mixtures of Experts/Committee machines

15+16. Neural networks for robot control

Assessment

3rd years: All coursework

Masters students: 50% coursework, 50 % exam
(start of next term)

Coursework is 2 programming projects first is
20% of coursework (details next week) due in
week 6, second 80% due week 10.

Coursework dealt with in seminars, some
theoretical, some practical matlab sessions
(programs can be in any language, but matlab is
useful for in
-
built functions)

This week’s seminar: light maths revision

Course Texts

1. Haykin S (1999). Neural networks. Prentice Hall
International. Excellent but quite heavily mathematical

2. Bishop C (1995). Neural networks for pattern
recognition. Oxford: Clarendon Press (good but a bit
statistical, not enough dynamical theory)

3. Pattern Classification, John Wiley, 2001

R.O. Duda and P.E. Hart and D.G. Stork

4. Hertz J., Krogh A., and Palmer R.G. Introduction to
the theory of neural computation (nice, but somewhat
out of date)

5. Pattern Recognition and Neural Networks

by Brian D. Ripley. Cambridge University Press. Jan
1996. ISBN 0 521 46086 7.

6. Neural Networks. An Introduction, Springer
-
Verlag
Berlin, 1991 B. Mueller and J. Reinhardt

As its quite a mathematical subject good to find the book
that best suits your level


Also for algorithms/mathematical detail see Numerical
Recipe’s, Press et al.

And appendices of Duda, Hart and Stork and Bishop


Uses of NNs

Neural Networks
Are For

Applications

Science


Character recognition Neuroscience


Optimization Physics,
mathematics statistics


Financial prediction Computer science


Automatic driving Psychology


.............................. ...........................




What are biological NNs?


UNITs
: nerve cells called neurons, many different
types and are extremely complex



around
10
11

neurons in the brain (depending on
counting technique) each with 10
3

connections



INTERACTIONs
: signal is conveyed by action
potentials, interactions could be chemical (release or
receive neurotransmitters) or electrical at the synapse



STRUCTUREs:

feedforward and

feedback and
self
-
activation recurrent



The nerve fibre is clearly a signalling mechanism of limited scope.

It can only transmit a succession of brief explosive waves, and the

message can only be varied by changes in the frequency and in the

total number of these waves. … But this limitation is really a small
matter, for in the body the nervous units do not act in isolation as

they do in our experiments. A sensory stimulus will usually affect a

number of receptor organs, and its result will depend on the

composite message in many nerve fibres
.”
Lord Adrian, Nobel
Acceptance Speech, 1932.


We now know it’s not quite that simple


Single neurons are highly complex
electrochemical devices


Synaptically connected networks are only
part of the story


Many forms of interneuron communication
now known


acting over many different
spatial and temporal scales

The complexity of a
neuronal system can be

partly seen from a picture
in a book on computational
neuroscience

edited by Jianfeng that I am
writing a chapter for

How do we go from real neurons to artificial ones?


Hillock

input

output


Single neuron activity





Membrane potential

is the voltage difference between a neuron
and its surroundings (0 mV)




Cell

Cell

Cell

Cell

0 Mv


Membrane potential


Single neuron activity




If you measure the
membrane potential

of a neuron and print it out


on the screen, it looks like:

spike


Single neuron activity




A spike is generated when the
membrane potential

is greater than


its threshold




Abstraction


So we can forget all sub
-
threshold activity and concentrate on
spikes (action potentials)
, which are the signals sent to other
neurons

Spikes



Only spikes are important since other neurons receive them


(signals)




Neurons communicate with spikes




Information is coded by

spikes



So if we can manage to measure the spiking
time, we decipher how the brain works ….





Again its not quite
that simple










spiking time in the cortex is random













With identical input


for the identical neuron


spike patterns are similar, but not identical

Recording from a real neuron: membrane potential


Single spiking time is meaningless


To extract useful information, we have to average








to obtain the
firing rate
r



for a group of neurons in a local circuit where neuron


codes the same information



over a time window

Local circuit

=

Time window = 1 sec

r =

= 6

Hz

So we can have a network of these
local groups

w
1:
synaptic strength

w
n

r
1

r
n

Hence we have firing rate of a group of neurons


r
i
is the firing rate of input
local circuit


The neurons at output local circuits receives signals in the form





The output firing rate of the output local circuit is then given by


R




where f is the activation function, generally a Sigmoidal
function of some sort


w
i
weight, (synaptic strength) measuring the strength of the
interaction between neurons
.

Artificial Neural networks




Local circuits (average to get firing rates)




Single neuron (send out spikes)

Artificial Neural Networks (ANNs)

A network with interactions, an attempt to mimic the brain



UNITs
: artificial neuron (linear or nonlinear input
-
output unit), small numbers, typically less than a few
hundred



INTERACTIONs
: encoded by weights, how strong a
neuron affects others



STRUCTUREs
: can be feedforward, feedback or
recurrent

It is still far too naïve as a brain model and an information
processing device and the development of the field relies
on all of us

x
n

x
1

x
2

Input


(visual

input)


Output





(
Motor


output)

Four
-
layer networks

Hidden layers

The general artificial neuron model has five components, shown
in the following list. (The subscript i indicates the i
-
th input
or weight.)

1.

A set of
inputs
, x
i
.


2.

A set of
weights
, w
i
.


3.

A
bias
, u.


4.

An
activation function
, f.


5.

Neuron output
, y


Thus the key to understanding ANNs is to
understand/generate the local input
-
output relationship