History of Neural Computing
•
McCulloch

Pitts 1943

showed that a ”neural network” with
simple logical units computes any
computable function

beginning of Neural Computing,
Artificial Intelligence, and Automaton
Theory
•
Wiener 1948

Cybernetics, first time statistical
mechanics model for computing


compare Hopfield 1982
•
Hebb 1949

physiological learning rule based on the
synaptic modification, Hebbian learning


repeated synaptic activity strenghen
synaptic response
•
Marvin Minsky 1954

”Neural

anlog system & brain model”
Ph.D. thesis at Princeton

An article ”Toward AI” in 1961, a chapter
”Neural Computing”

The book ”Computation: Finite and
infinite machines” transform McCulloch

Pitts results into Automaton theory
•
Gabor 1954

nonlinear adaptive filter
•
Taylor 1956

associative memory

> learning matrix

also early works for correlation matrix
memory (Anderson 1972, Kohonen 1972,
Nakano 1972)
PERCEPTRON
PERCEPTRON
•
Rosenblatt 1958

a new method for
supervised learning
”perceptron convergence theorem”
•
Widrow

Hoff 1960

LMS

algorithm for learning
Adaline
•
Widrow 1962

Madaline: leyered neural networks
•
Amari 1967

stochastic gradient method
•
Nilsson 1965

linearly separable sets
During golden era of Perceptrons, in
60’s, it was believed that they solve all
problems.
PERCEPTRON
•
Minsky

Papert 1969

the book ”Perceptrons”

showed mathematically the restrictions of
1

leyer perceptrons

they doubted that more leyers do not
bring essentially more power
Neural Network research went into
”HALT” state
•
The research was low about ten years

reasons: low computing power
psychologically math results
•
research was continued in neurosciences
and in psychology
Self

Organizing Maps
•
This reseach was continued during
”Perceptron

halt”
•
von der Malsburg 1973

first demonstration of self

organization

first paper was inspired by topological
maps in brain
•
Grossberg 1980

a new form of self

organization; ART
•
Kohonen 1982

1 and 2 dimensional lattice, different to
von der Malsburg

nowadays a benchmark SOM
Self

Organizing Maps
Hopfield networks
•
Hopfield 1982

formulation of an energy function for
understanding how attraction network work

popular in 80’s:
feedback Neural Net = Hopfield Net

no neurophysiologically adequate, but
interesting since information could be stored
into a stable net
•
Paper triggered a new era of Neural
Networks
•
Paper caused much controversy, there were
similar ideas in the literature: Cragg

Tamperley (1954), Cowan (1967),
Grossberg (1967)
New rise of NN
•
Kirkpatrick

Gelatt

Vecchi 1983

Simulated annealing for combinatorial
optimization problem

idea from statistical mechanics model for
cooling in crystal formation
•
Ackley

Hinton

Sejnowski 1985

Bolzmann machine, first succeeded
realization of multileyer network

> earlier psychological barrier was broken
•
Barto

Sutton

Anderson 1983

reinforcement learning, balance of a
broomstick
MULTILEYER PERCEPTRON
(Error)
Back Propagation
•
Problem in multileyer perceptron network:
How to update the weights?
•
Rumelhart

Hinton

Williams 1986

The book:
Parallel Distributed Processing

back propagation algorithm solve problem

most popular learning algorithm for MLP’s
•
found also by Parker 1985, LeCun 1985
•
earlier by
Werbos 1974
(Bryson

Ho 1969)
MULTILEYER PERCEPTRON
Latest additions
•
Broomhead

Lowe 1988

Radial basis functions (RBF)

input leyer : nonlinear hidden leyer : linear
output leyer

link neural networks to numerical analysis
•
Linsker 1988

self organization in perceptual networks

triggered again interest of information
theorists
•
Bell

Sejnowski 1995

blind source separation
Comments 0
Log in to post a comment