and Fuzzy Logic

runmidgeAI and Robotics

Oct 20, 2013 (4 years and 20 days ago)

95 views

Dr.
-
Ing. Erwin Sitompul

President University

Lecture 2

Introduction to Neural Networks

and Fuzzy Logic

President University

Erwin Sitompul

NNFL

2/
1

http://zitompul.wordpress.com

President University

Erwin Sitompul

NNFL

2/
2

Weights, need to
be determined

Biological neuron

Artificial neuron

Bias, need to
be determined

Learning Processes

Neural Networks

Biological and Artificial Neuron

President University

Erwin Sitompul

NNFL

2/
3

Learning Processes

Neural Networks

Application of Neural Networks


Function approximation and prediction


Pattern recognition


Signal processing


Modeling and control


Machine learning

President University

Erwin Sitompul

NNFL

2/
4

Building a Neural Network


Select Structure
: design the way that the neurons are
interconnected.


Select weights
: decide the strengths with which the
neurons are interconnected.


Weights are selected to get a “good match” of
network output to the output of a training set.


Training set is a set of inputs and desired outputs.


The weight selection is conducted by the use of a
learning algorithm.

Learning Processes

Neural Networks

President University

Erwin Sitompul

NNFL

2/
5

Stage 1: Network Training

Training Data

Stage 2: Network Validation

Artificial neural network

Input and output
sets, adequate
coverage

Learning

Process

In the form of a set
of optimized
synaptic weights
and biases

Unseen Data

From the same
range as the
training data

Artificial neural network

Implementation

Phase

Learning Processes

Neural Networks

Learning Process

Knowledge

Output
Prediction

President University

Erwin Sitompul

NNFL

2/
6

Learning Process


Learning

is a process
by which the free
parameters of a neural
network are adapted
through a process of
stimulation by the
environment in which
the network is
embedded.


In most cases, due to
complex optimization
plane, the optimized
weights and biases are
obtained as a result of
a number of learning
iterations.

[
w
,
b
]

x

y

[
w,b
]
0

x

y
(0)

Initialize: Iteration (0)

[
w,b
]
1

x

y
(1)

Iteration (1)

[
w,b
]
n

x

y
(
n
)


d

Iteration (
n
)

ANN

d

: desired output



Learning Processes

Neural Networks

President University

Erwin Sitompul

NNFL

2/
7

Learning Rules

Learning Processes

Neural Networks


Error Correction Learning


Delta Rule or Widrow
-
Hoff Rule


Memory Based Learning


Nearest Neighbor Rule


Hebbian Learning


Synchronous activation increases the synaptic
strength


Asynchronous activation decreases the synaptic
strength


Competitive Learning


Boltzmann Learning

President University

Erwin Sitompul

NNFL

2/
8

w
k
1
(
n
)

x
1

x
2

x
m

Inputs

Synaptic

weights

Bias

Activation
function

w
k
2
(
n
)

w
km
(
n
)



Output

y
k

(
n
)

Desired output

d
k

(
n
)

e
k

(
n
)

+

f
(.)

b
k
(
n
)

1

-

Error

signal

Learning Processes

Neural Networks

Error
-
Correction Learning



Learning

Rule

President University

Erwin Sitompul

NNFL

2/
9

Learning Processes

Neural Networks

Delta Rule (Widrow
-
Hoff Rule)

Minimization of a cost function
(or performance index)

President University

Erwin Sitompul

NNFL

2/
10


w
kj
(0) = 0



y
k
(
n
) =

[
w
kj
(n
)
x
j
(
n
)]



w
kj
(
n
+1) =
w
kj
(
n
) +
h
[
d
k
(
n
)


y
k
(
n
)]
x
j
(
n
)


h

: learning rate, [0…1]


n

=
n
+1



n

= 0


Least Means Squares Rule

Learning Processes

Neural Networks

Delta Rule (Widrow
-
Hoff Rule)

President University

Erwin Sitompul

NNFL

2/
11

Learning Processes

Neural Networks

Learning Paradigm



ANN

Error

Desired

Actual

+

-

Environment

(Data)

Teacher

(Expert)

Supervised

Unsupervised

Environment

(Data)

Delay

ANN

Delayed

Reinforcement

Learning

Cost

Function

President University

Erwin Sitompul

NNFL

2/
12

Single Layer Perceptrons

Neural Networks

Single Layer Perceptrons


Output unit is independent
of the others.


Analysis can be limited to
single output perceptron.


Single
-
layer perceptron network is a network with
all

the inputs connected
directly

to the output(s).


President University

Erwin Sitompul

NNFL

2/
13

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons

w
1

w
2

E
(
w
)


Key idea
: Learning is performed by adjusting the
weights in order to minimize the sum of squared errors
on a training.


Weights
are
u
pdated repeatedly (in each
epoch/iteration).


Sum of squared errors is a classical error measure (
e
.
g
.
commonly used in linear regression).


Learning can be
viewed as an
optimization search
problem in weight
space
.


President University

Erwin Sitompul

NNFL

2/
14

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons


The learning rule performs a search within the
solution's vector space towards a
global minimum
.



T
he error surface itself is a
hyper
-
paraboloid

but is
seldom as smooth as is
depicted below.


In most problems, the solution
space is quite irregular with
numerous pits and hills which
may cause the network to settle
down in a
local minimum

(not
the best overall solution).


Epochs
are repeated until
stopping criterion is reached
(error magnitude, number of
iterations, change of weights,
etc).


President University

Erwin Sitompul

NNFL

2/
15

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons

Widrow [1962]

x
1

x
2

x
m

w
k
1

w
k
2

w
km

.

.

.



Adaline

(Adaptive Linear Element)

Goal:

President University

Erwin Sitompul

NNFL

2/
16

Least Mean Squares (LMS)

Single Layer Perceptrons

Neural Networks


The following
cost

function (
error

function) should be
minimized:

President University

Erwin Sitompul

NNFL

2/
17

Single Layer Perceptrons

Neural Networks

Least Mean Squares (LMS)


Letting

f
(
w
k
) =
f
(
w
k
1
,
w
k
2
,

,
w
km
)
be a function over

R
m
,
then


Defining

President University

Erwin Sitompul

NNFL

2/
18


f


w


f


w

df
:
positive

df

:
zero

df

:
negative

go uphill

plain

go downhill


f


w

To minimize
f
, we choose

Single Layer Perceptrons

Neural Networks

Gradient Operator

President University

Erwin Sitompul

NNFL

2/
19

Single Layer Perceptrons

Neural Networks

Adaline Learning Rule


With


then


As already obtained before,

Weight Modification Rule


Defining


we can write

President University

Erwin Sitompul

NNFL

2/
20

Single Layer Perceptrons

Neural Networks

Adaline Learning Modes


Batch

Learning Mode


Incremental
Learning Mode

President University

Erwin Sitompul

NNFL

2/
21



-
Learning Rule


LMS Algorithm


Widrow
-
Hoff Learning Rule

Single Layer Perceptrons

Neural Networks

Adaline Learning Rule

President University

Erwin Sitompul

NNFL

2/
22

Single Layer Perceptrons

Neural Networks

Generalization and Early Stopping


By proper training, a neural
network may produce
reasonable output for inputs
not seen

during training


G
eneralization


Generalization is particularly
useful for the analysis of a
“noisy” data (e.g. time

series
)



“Overtraining” will not improve
the ability of a neural network
to produce good output.


On the contrary, it will try to
take noise as the real data
and lost its generality.

President University

Erwin Sitompul

NNFL

2/
23

Generalization and Early Stopping

Single Layer Perceptrons

Neural Networks

Overfitting vs

Generalization

President University

Erwin Sitompul

NNFL

2/
24

Homework 1

Single Layer Perceptrons

Neural Networks

Given a function y = 4
x
2
, you are required to find the
value of
x

that will result
y

= 2 by using the Least Mean
Squares method.


Use initial estimate
x
0

= 1 and learning rate
η

= 0.01.


Write down the results of the first 10
epochs/iterations.


Give conclusion about your result.


Note
: Calculation can be done manually or using
Matlab.