Neural Networks I

cartcletchAI and Robotics

Oct 19, 2013 (3 years and 10 months ago)

80 views

Neural Networks I

Karel Berkovec

karel.berkovec (at) seznam.cz

Neural Networks I


Karel Berkovec, 2007

Artificial Intelligence


Artificial Intelligence

Neural Networks I


Karel Berkovec, 2007


Stochastic methodes


Expert systems,
mathematical logic,
production systems,
bayesian networks …


Regression,
interpolation, frequency
analysis ..


Neural Networks

Symbolic approach

Connectionist approach

Adaptive approach

Analytic approach

Is it really working?




Is it a standard mechanism?



What is it good for?



Use it someone for
real

applications?



Can I grasp how it works?



Can I use it?




Neural Networks I


Karel Berkovec, 2007

This presentation




Basic introduction



Small history window



Model of neuron and neural network



Supervised learning (backpropagation)




No
biology, mathematical fundaments, unsupervised
learning, stochastic models, neurocomputers, etc.

Neural Networks I


Karel Berkovec, 2007

History I



20s


von Neumann computer model



1943


Warren McCulloch and Walter Pitts


matematical model of neuron



1946


Eniac



1949


Donald Hebb


The Organization of
Behaviour



1951


1
st

Czechoslovak computer SAPO



1951


1
st

neurocomputer
Snark



1957


Frank Rosenblatt


perceptron

+ learning
algorithm



1958


Rosenblatt and Charless Wightman


1
st

really used neurocomputer
Mark I Perceptron

Neural Networks I


Karel Berkovec, 2007

History II



60s ADALINE



1
st

company oriented on neurocomputing



Exhausting of potential



1967 Marvin Minsky & Seymour Papert


Perceptrons


XOR problem can’t be solved by 1 perceptron


Neural Networks I


Karel Berkovec, 2007

History III



1983


DARPA



1982, 1984
-

John Hopfield


physical models & NN



1986


David Rumehart, Geoffrey Hinton, Ronald
Williams


Backpropagation



1969 Arthur Bryson & Paul Webos



1974 Paul Werbos



1985 David Parker




1987


IEEE International Conference on Neural
Networks



Since 90’ NN boom of NNs



ART, BAM, RBF, spiking neurons

Neural Networks I


Karel Berkovec, 2007

Present



Many models of neuron


Perceptron, RBF, spiking
neuron …



Many approaches


backpropagation, hopfield
learning, correlations, competitive learning, stochastic
learning, …



Many libraries and modules


for Matlab, Statistica,
Excel …



Many applications


forecasting, smoothing,
recognition, classification, datamining, compression …


Neural Networks I


Karel Berkovec, 2007

Pros and cons


+ Simple to use


+ Very good results


+ Fast results


+ Robust against incomplete or corrupted inputs


+ Generalization


+/
-

Mathematical background



-

Not transparent and traceable


-

Hard to tune parameters (sometimes hair
-
triggered)


-

Sometimes a long time for learning needed


-

Some tasks are hard to formulate for NNs


Neural Networks I


Karel Berkovec, 2007

Formal neuron
-

perceptron

Neural Networks I


Karel Berkovec, 2007
















-

potential

-

threshold

-

weights

AB problem

Neural Networks I


Karel Berkovec, 2007

XOR problem

Neural Networks I


Karel Berkovec, 2007

XOR problem

Neural Networks I


Karel Berkovec, 2007

1

1

XOR problem

Neural Networks I


Karel Berkovec, 2007

2

2

XOR problem

Neural Networks I


Karel Berkovec, 2007

1

2

AND

Feed
-
forward layered network

Neural Networks I


Karel Berkovec, 2007

Output layer


2
nd

hidden layer


1
st

hidden layer



Input layer


Activating function

Neural Networks I


Karel Berkovec, 2007
















Heaviside function

Saturated linear function

Standard sigmoidal function

Hyperbolical tangents

NN function



NN maps input on output




Feed
-
forward NN with one hidden layer and with
sigmoidal activation function can approximate arbitrary
closely any continuous function




The question is how to set up parameters of the
network.

Neural Networks I


Karel Berkovec, 2007









NN learning



Error function





Perceptron adaptation rule:











Algorithm with this learning rule convergates in finite
time (if A and B separatable)

Neural Networks I


Karel Berkovec, 2007









y=0 d=1

y=1 d=0

AB problem

Neural Networks I


Karel Berkovec, 2007

Backpropagation




The most often used learning algorithm for NNs


cca 80%




Fast convergation




Good results




Many modifications


Neural Networks I


Karel Berkovec, 2007

Energetic function



How to adapt weights of neurons in hidden layers?



We would like to find a minimum of the error function


-

why not use a derivation?








Neural Networks I


Karel Berkovec, 2007

Error gradient


Adaptation rule:




Neural Networks I


Karel Berkovec, 2007

Output layer






Neural Networks I


Karel Berkovec, 2007

Hidden layer






Neural Networks I


Karel Berkovec, 2007

Implementation BP




initialize network






repeat



update weights



for all patterns



count the result



count error



count








until error is not small enough


Neural Networks I


Karel Berkovec, 2007















Improvements of BP




Momentum





Adaptive learning parameters



Other variants of BP: SuperSAB, QuickProp,
Levenberg
-
Marquart alg.




Neural Networks I


Karel Berkovec, 2007

Overfitting






Neural Networks I


Karel Berkovec, 2007