Dr.

Ing. Erwin Sitompul
President University
Lecture 2
Introduction to Neural Networks
and Fuzzy Logic
President University
Erwin Sitompul
NNFL
2/
1
http://zitompul.wordpress.com
President University
Erwin Sitompul
NNFL
2/
2
Weights, need to
be determined
Biological neuron
Artificial neuron
Bias, need to
be determined
Learning Processes
Neural Networks
Biological and Artificial Neuron
President University
Erwin Sitompul
NNFL
2/
3
Learning Processes
Neural Networks
Application of Neural Networks
Function approximation and prediction
Pattern recognition
Signal processing
Modeling and control
Machine learning
President University
Erwin Sitompul
NNFL
2/
4
Building a Neural Network
Select Structure
: design the way that the neurons are
interconnected.
Select weights
: decide the strengths with which the
neurons are interconnected.
Weights are selected to get a “good match” of
network output to the output of a training set.
Training set is a set of inputs and desired outputs.
The weight selection is conducted by the use of a
learning algorithm.
Learning Processes
Neural Networks
President University
Erwin Sitompul
NNFL
2/
5
Stage 1: Network Training
Training Data
Stage 2: Network Validation
Artificial neural network
Input and output
sets, adequate
coverage
Learning
Process
In the form of a set
of optimized
synaptic weights
and biases
Unseen Data
From the same
range as the
training data
Artificial neural network
Implementation
Phase
Learning Processes
Neural Networks
Learning Process
Knowledge
Output
Prediction
President University
Erwin Sitompul
NNFL
2/
6
Learning Process
Learning
is a process
by which the free
parameters of a neural
network are adapted
through a process of
stimulation by the
environment in which
the network is
embedded.
In most cases, due to
complex optimization
plane, the optimized
weights and biases are
obtained as a result of
a number of learning
iterations.
[
w
,
b
]
x
y
[
w,b
]
0
x
y
(0)
Initialize: Iteration (0)
[
w,b
]
1
x
y
(1)
Iteration (1)
[
w,b
]
n
x
y
(
n
)
≈
d
Iteration (
n
)
ANN
d
: desired output
…
Learning Processes
Neural Networks
President University
Erwin Sitompul
NNFL
2/
7
Learning Rules
Learning Processes
Neural Networks
Error Correction Learning
Delta Rule or Widrow

Hoff Rule
Memory Based Learning
Nearest Neighbor Rule
Hebbian Learning
Synchronous activation increases the synaptic
strength
Asynchronous activation decreases the synaptic
strength
Competitive Learning
Boltzmann Learning
President University
Erwin Sitompul
NNFL
2/
8
w
k
1
(
n
)
x
1
x
2
x
m
Inputs
Synaptic
weights
Bias
Activation
function
w
k
2
(
n
)
w
km
(
n
)
Output
y
k
(
n
)
Desired output
d
k
(
n
)
e
k
(
n
)
+
f
(.)
b
k
(
n
)
1

Error
signal
Learning Processes
Neural Networks
Error

Correction Learning
Learning
Rule
President University
Erwin Sitompul
NNFL
2/
9
Learning Processes
Neural Networks
Delta Rule (Widrow

Hoff Rule)
Minimization of a cost function
(or performance index)
President University
Erwin Sitompul
NNFL
2/
10
w
kj
(0) = 0
y
k
(
n
) =
[
w
kj
(n
)
x
j
(
n
)]
w
kj
(
n
+1) =
w
kj
(
n
) +
h
[
d
k
(
n
)
–
y
k
(
n
)]
x
j
(
n
)
h
: learning rate, [0…1]
n
=
n
+1
n
= 0
Least Means Squares Rule
Learning Processes
Neural Networks
Delta Rule (Widrow

Hoff Rule)
President University
Erwin Sitompul
NNFL
2/
11
Learning Processes
Neural Networks
Learning Paradigm
ANN
Error
Desired
Actual
+

Environment
(Data)
Teacher
(Expert)
Supervised
Unsupervised
Environment
(Data)
Delay
ANN
Delayed
Reinforcement
Learning
Cost
Function
President University
Erwin Sitompul
NNFL
2/
12
Single Layer Perceptrons
Neural Networks
Single Layer Perceptrons
•
Output unit is independent
of the others.
•
Analysis can be limited to
single output perceptron.
Single

layer perceptron network is a network with
all
the inputs connected
directly
to the output(s).
President University
Erwin Sitompul
NNFL
2/
13
Single Layer Perceptrons
Neural Networks
Derivation of a Learning Rule for Perceptrons
w
1
w
2
E
(
w
)
Key idea
: Learning is performed by adjusting the
weights in order to minimize the sum of squared errors
on a training.
Weights
are
u
pdated repeatedly (in each
epoch/iteration).
Sum of squared errors is a classical error measure (
e
.
g
.
commonly used in linear regression).
Learning can be
viewed as an
optimization search
problem in weight
space
.
President University
Erwin Sitompul
NNFL
2/
14
Single Layer Perceptrons
Neural Networks
Derivation of a Learning Rule for Perceptrons
The learning rule performs a search within the
solution's vector space towards a
global minimum
.
T
he error surface itself is a
hyper

paraboloid
but is
seldom as smooth as is
depicted below.
In most problems, the solution
space is quite irregular with
numerous pits and hills which
may cause the network to settle
down in a
local minimum
(not
the best overall solution).
Epochs
are repeated until
stopping criterion is reached
(error magnitude, number of
iterations, change of weights,
etc).
President University
Erwin Sitompul
NNFL
2/
15
Single Layer Perceptrons
Neural Networks
Derivation of a Learning Rule for Perceptrons
Widrow [1962]
x
1
x
2
x
m
w
k
1
w
k
2
w
km
.
.
.
Adaline
(Adaptive Linear Element)
Goal:
President University
Erwin Sitompul
NNFL
2/
16
Least Mean Squares (LMS)
Single Layer Perceptrons
Neural Networks
The following
cost
function (
error
function) should be
minimized:
President University
Erwin Sitompul
NNFL
2/
17
Single Layer Perceptrons
Neural Networks
Least Mean Squares (LMS)
Letting
f
(
w
k
) =
f
(
w
k
1
,
w
k
2
,
…
,
w
km
)
be a function over
R
m
,
then
Defining
President University
Erwin Sitompul
NNFL
2/
18
f
w
f
w
df
:
positive
df
:
zero
df
:
negative
go uphill
plain
go downhill
f
w
To minimize
f
, we choose
Single Layer Perceptrons
Neural Networks
Gradient Operator
President University
Erwin Sitompul
NNFL
2/
19
Single Layer Perceptrons
Neural Networks
Adaline Learning Rule
With
then
As already obtained before,
Weight Modification Rule
Defining
we can write
President University
Erwin Sitompul
NNFL
2/
20
Single Layer Perceptrons
Neural Networks
Adaline Learning Modes
Batch
Learning Mode
Incremental
Learning Mode
President University
Erwin Sitompul
NNFL
2/
21

Learning Rule
LMS Algorithm
Widrow

Hoff Learning Rule
Single Layer Perceptrons
Neural Networks
Adaline Learning Rule
President University
Erwin Sitompul
NNFL
2/
22
Single Layer Perceptrons
Neural Networks
Generalization and Early Stopping
By proper training, a neural
network may produce
reasonable output for inputs
not seen
during training
G
eneralization
Generalization is particularly
useful for the analysis of a
“noisy” data (e.g. time
–
series
)
“Overtraining” will not improve
the ability of a neural network
to produce good output.
On the contrary, it will try to
take noise as the real data
and lost its generality.
President University
Erwin Sitompul
NNFL
2/
23
Generalization and Early Stopping
Single Layer Perceptrons
Neural Networks
Overfitting vs
Generalization
President University
Erwin Sitompul
NNFL
2/
24
Homework 1
Single Layer Perceptrons
Neural Networks
Given a function y = 4
x
2
, you are required to find the
value of
x
that will result
y
= 2 by using the Least Mean
Squares method.
Use initial estimate
x
0
= 1 and learning rate
η
= 0.01.
Write down the results of the first 10
epochs/iterations.
Give conclusion about your result.
Note
: Calculation can be done manually or using
Matlab.
Comments 0
Log in to post a comment