History of Neural Computing

clangedbivalveAI and Robotics

Oct 19, 2013 (3 years and 5 months ago)

54 views

History of Neural Computing


McCulloch
-

Pitts 1943


-

showed that a ”neural network” with
simple logical units computes any
computable function


-

beginning of Neural Computing,
Artificial Intelligence, and Automaton
Theory


Wiener 1948


-

Cybernetics, first time statistical
mechanics model for computing


-

-

compare Hopfield 1982


Hebb 1949


-

physiological learning rule based on the
synaptic modification, Hebbian learning


-

-

repeated synaptic activity strenghen
synaptic response


Marvin Minsky 1954


-

”Neural
-

anlog system & brain model”
Ph.D. thesis at Princeton


-

An article ”Toward AI” in 1961, a chapter
”Neural Computing”


-

The book ”Computation: Finite and
infinite machines” transform McCulloch
-

Pitts results into Automaton theory


Gabor 1954


-

nonlinear adaptive filter


Taylor 1956


-

associative memory
-
> learning matrix


-

also early works for correlation matrix
memory (Anderson 1972, Kohonen 1972,
Nakano 1972)


PERCEPTRON

PERCEPTRON


Rosenblatt 1958


-

a new method for
supervised learning
”perceptron convergence theorem”


Widrow
-

Hoff 1960


-

LMS
-
algorithm for learning
Adaline


Widrow 1962


-

Madaline: leyered neural networks


Amari 1967


-

stochastic gradient method


Nilsson 1965


-

linearly separable sets



During golden era of Perceptrons, in
60’s, it was believed that they solve all
problems.


PERCEPTRON


Minsky
-

Papert 1969


-

the book ”Perceptrons”


-

showed mathematically the restrictions of
1
-
leyer perceptrons


-

they doubted that more leyers do not
bring essentially more power


Neural Network research went into
”HALT” state



The research was low about ten years


-

reasons: low computing power



psychologically math results


research was continued in neurosciences
and in psychology

Self
-
Organizing Maps


This reseach was continued during
”Perceptron
-
halt”


von der Malsburg 1973


-

first demonstration of self
-
organization


-

first paper was inspired by topological
maps in brain


Grossberg 1980


-

a new form of self
-
organization; ART


Kohonen 1982


-

1 and 2 dimensional lattice, different to
von der Malsburg


-

nowadays a benchmark SOM


Self
-
Organizing Maps

Hopfield networks


Hopfield 1982


-

formulation of an energy function for
understanding how attraction network work


-

popular in 80’s:






feedback Neural Net = Hopfield Net


-

no neurophysiologically adequate, but
interesting since information could be stored
into a stable net



Paper triggered a new era of Neural
Networks



Paper caused much controversy, there were
similar ideas in the literature: Cragg
-
Tamperley (1954), Cowan (1967),
Grossberg (1967)

New rise of NN


Kirkpatrick
-

Gelatt
-

Vecchi 1983


-

Simulated annealing for combinatorial
optimization problem


-

idea from statistical mechanics model for
cooling in crystal formation


Ackley
-

Hinton
-

Sejnowski 1985


-

Bolzmann machine, first succeeded
realization of multileyer network



--
> earlier psychological barrier was broken



Barto
-

Sutton
-

Anderson 1983


-

reinforcement learning, balance of a
broomstick


MULTILEYER PERCEPTRON

(Error)
Back Propagation


Problem in multileyer perceptron network:

How to update the weights?



Rumelhart
-

Hinton
-

Williams 1986


-

The book:
Parallel Distributed Processing


-

back propagation algorithm solve problem


-

most popular learning algorithm for MLP’s


found also by Parker 1985, LeCun 1985


earlier by
Werbos 1974

(Bryson
-
Ho 1969)


MULTILEYER PERCEPTRON

Latest additions


Broomhead
-

Lowe 1988


-

Radial basis functions (RBF)


-

input leyer : nonlinear hidden leyer : linear
output leyer


-

link neural networks to numerical analysis


Linsker 1988


-

self organization in perceptual networks


-

triggered again interest of information
theorists


Bell
-

Sejnowski 1995


-

blind source separation