2806 Neural Computation Introduction Lecture 1

cracklegulleyΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 3 χρόνια και 8 μήνες)

132 εμφανίσεις

2806 Neural Computation



Introduction




Lecture 1

2005 Ari Visa


Agenda


Some historical notes


Biological background


What neural networks are?


Properties of neural network


Compositions of neural network


Relation to artificial intelligence

Overview

The human brain computes in an entirely
different way from the conventional digital
computer.

The brain routinely accomplishes perceptual
recognition in approximately 100
-
200 ms.

How does a human brain do it?

Some Expected Benefits


Nonlinearity


Input
-
Output Mapping


Adaptivity


Evidential Response


Contextual
Information



Fault Tolerance


VLSI
Implementability


Uniformity of
Analysis and Design


Neurobiological
Analogy

Definition


A neural network is a massive parallel
distributed processor made up of simple
processing units, which has a natural
propensity for storing experimental
knowledge and making it available for use.
It resembles the brain in two respects:

Definition


1) Knowledge is acquired by the network
from its environment through a learning
process.


2) Interneuron connection strengths, known
as synaptic weights, are used to store the
acquired knowledge.

Some historical notes

Lot of activities concerning automatas,
communication, computation,
understanding of nervous system during
1930s and 1940s

McCulloch and Pitts 1943

von Neumann EDVAC (Electronic Discrete
Variable Automatic Computer)

Hebb: The Organization of Behavior, 1949

Some historical notes


Some historical notes


Minsky: Theory of Neural
-
Analog
Reinforcement Systems and Its Application
to the Brain
-
Model Problem, 1954


Gabor: Nonlinear adaptive filter, 1954


Uttley: leaky integrate and fire neuron, 1956


Rosenblatt: the perceptron, 1958

Biological Background


The human nervous system may be viewed
as a three stage system (Arbib 1987): The
brain continually receives information,
perceives it, and makes appropriate
decisions.

Biological Background


Axons = the transmission lines


Dendrites = the receptive zones


Action potentials, spikes originate at the cell
body of neurons and then propagate

across
the individual neurons at constant velocity
and amplitude.


Biological Background


Synapses are
elementary structural
and functional units
that mediate the
interactions between
neurons.


Excitation or
inhibition

Biological Background


Note, that the
structural levels of
organization are a
unique characteristic
of the brain

Biological Background

Properties of Neural Network


A model of a neuron:


synapses (=connecting
links)


adder (=a linear
combiner)


an activation function

Properties of Neural Network


Another formulation
of a neuron model

Properties of Neural Network


Types of Activation
Function:


Threshold Function


Piecewise
-
Linear
Function


Sigmoid Function
(signum fuction or
hyperbolic tangent
function)

Properties of Neural Network


Stochastic Model of a Neuron

The activation function of the McCulloch
-
Pitts
model is given a probabilistic interpretation, a

neuron is permitted to reside in only one of two
states: +1 or

1. The decision for a neuron to fire
is probabilistic.

A standard choice for P(v) is the sigmoid
-
shaped
function = 1/(1+exp(
-
v/T)), where T is a
pseudotemperature.

Properties of Neural Network


The model of an artificial neuron may also be
represented as a signal
-
flow graph.


A signal
-
flow graph is a network of directed links
that are interconnected at certain points called
nodes. A typical node j has an associated node
signal x
j
. A typical directed link originates at node
j and terminates on node k. It has an associated
transfer function (transmittance) that specifies the
manner in which the signal y
k

at node k depends
on the signal x
j

at node j.

Properties of Neural Network


Rule 1: A signal flows
along a link in the
direction defined by
the arrow


Synaptic links (a
linear input
-
output
relation, 1.19a)


Activation links (a
nonlinear input
-
output
relation, 1.19b)

Properties of Neural Network


Rule 2: A node signal
equals the algebraic
sum of all signals
entering the pertinent
node via the incoming
links (1.19c)

Properties of Neural Network

Rule 3: The signal at a node
is transmitted to each
outgoing link originating
from that node, with the
transmission being
entirely independent of the
transfer functions of the
outgoing links, synaptic
divergence or fan
-
out
(1.9d)


Properties of Neural Network


A neural network is a
directed graph consisting
of nodes with
interconnecting synaptic
and activation links, and is
characterized by four
properties:


1. Each neuron is
represented by a set of
linear synaptic links, an
externally applied bias,
and a possibly nonlinear
activation link, This bias
is represented by a
synaptic link connected to
an input fixed at +1.


Properties of Neural Network



2. The synaptic links of a
neuron weight their
respective input signals.


3. The weighted sum of
the input signals defines
the induced local field of a
neuron in question.


4. The activation link
squashes the induced local
field of the neuron to
produce an output.

Properties of Neural Network


Complete graph


Partially complete
graph = architectural
graph

Properties of Neural Network


Feedback is said to exist
in a dynamic system
whenever the output of an
element in the system
influences in part the input
applied to the particular
element, thereby giving
rise to one or more closed
paths for the transmission
of signals around the
system (1.12)

Properties of Neural Network


y
k
(n) = A[x’
j
(n)]


x’
j
(n) = x
j
(n)+B[y
k
(n)]


y
k
(n)=A/(1
-
AB)[x
j
(n)]


the closed
-
loop
operator A/(1
-
AB)


the open
-
loop operator

AB

In general AB

BA

Properties of Neural Network


A/(1
-
AB)


w/1
-
wz
-
1
)



y
k
(n) is convergent
(=stable), if |w| < 1
(1.14a)


y
k
(n) is divergent
(=unstable), if |w| < 1



Properties of Neural Network


A/(1
-
AB)


w/1
-
wz
-
1
)



y
k
(n) is convergent
(=stable), if |w| < 1 (1.14a)


y
k
(n) is divergent
(=unstable), if |w|


1,

if |w| = 1 the divergence is
linear (1.14.b)

if |w| >1 the divergence is
exponential (1.14c)


Compositions of Neural Network


The manner in which
the neurons of a neural
network are structured
is intimately linked
with the learning
algorithm used to train
the network.


Single
-
Layer
Feedforward
Networks

Compositions of Neural Network


Multilayer
Feedforward
Networks (1.16)


Hidden layers, hidden
neurons or hidden
units
-
> enabled to
extract higher
-
order
statistics

Compositions of Neural Network



Recurrent Neural
Network (1.17)


It has at least one
feedback loop.

Knowledge Representation


Knowledge refers to stored information or
models used by a person or machine to
interpret, predict, and appropriately respond
to the outside world (Fishler and Firschein,
1987)


A major task for neural network is to learn a
model of the world


Knowledge Representation


Knowledge of the world consists of two
kind of information


1) The known world state, prior information


2) Observations of the world, obtained by
means of sensor.


Obtained observations provide a pool of
information from which the examples used
to train the neural network are drawn.

Knowledge Representation


The examples can be labelled or unlabelled.


In labelled examples, each example representing
an input signal is paired with a corresponding
desired response. Note, both positive and negative
examples are possible.


A set of input
-
output pairs, with each pair
consisting of an input signal and the
corresponding desired response, is referred to as a
set of training data or training sample.

Knowledge Representation


Selection of an appropriate architecture


A subset of examples is used to train the
network by means of a suitable algorithm
(=learning).


The performance of the trained network is
tested with data not seen before (=testing).


Generalization

Knowledge Representation


Rule 1: Similar inputs from similar classes
should usually produce similar
representations inside the network, and
should therefore be classified as belonging
to the same category.


Rule 2: Items to be categorized as separate
classes should be given widely different
representations in the network.

Knowledge Representation


Rule 3: If a particular feature is important,
then there should be a large number of
neurons involved in the representation of
that item in the network


Rule 4: Prior information and invariances
should be built into the design of a neural
network, thereby simplifying the network
design by not having to learn them.

Knowledge Representation


How to Build Prior Information Into Neural
Network Design?


1) Restricting the network architecture
through the use of local connections known
as receptive fields.


2) Constraining the choice of synaptic
weights through the use of weight
-
sharing.

Knowledge Representation


How to Build Invariances into Neural
Network Design?


1) Invariance by Structure


2) Invariance by Training


3) Invariant Feature Space

Relation to Artificial Intelligence


The goal of artificial intelligence (AI) is the
development of paradigms or algorithms that
require machines to perform cognitive tasks (Sage
1990).


An AI system must be capable of doing three
things:


1) Store knowledge


2) Apply the knowledge stored to solve problems.


3) Acquire new knowledge through experience.

Relation to Artificial Intelligence


Representation: The use of
a language of symbol
structures to represent
both general knowledge
about a problem domain
of interest and specific
knowledge about the
solution to the problem

Declarative knowledge

Procedural knowledge


Reasoning: The ability to solve problems


The system must be able to express and solve a broad
range of problems and problem types.


The system must be able to make explicit and implicit
information known to it.


The system must have control mechanism that
determines which operations to apply to a particular
problem.


Relation to Artificial Intelligence



Relation to Artificial Intelligence


Learning: The
environment supplies
some information to a
learning element. The
learning element then uses
this information to make
improvements in a
knowledge base, and
finally the performance
element uses the
knowledge base to
perform its task.

Summary



A major task for neural network is to learn a
model of the world


It is not a totally new approach but it has
differences to AI, matematical modeling,
Pattern Recognition and so on.