# and Fuzzy Logic

AI and Robotics

Oct 20, 2013 (4 years and 7 months ago)

107 views

Dr.
-
Ing. Erwin Sitompul

President University

Lecture 2

Introduction to Neural Networks

and Fuzzy Logic

President University

Erwin Sitompul

NNFL

2/
1

http://zitompul.wordpress.com

President University

Erwin Sitompul

NNFL

2/
2

Weights, need to
be determined

Biological neuron

Artificial neuron

Bias, need to
be determined

Learning Processes

Neural Networks

Biological and Artificial Neuron

President University

Erwin Sitompul

NNFL

2/
3

Learning Processes

Neural Networks

Application of Neural Networks

Function approximation and prediction

Pattern recognition

Signal processing

Modeling and control

Machine learning

President University

Erwin Sitompul

NNFL

2/
4

Building a Neural Network

Select Structure
: design the way that the neurons are
interconnected.

Select weights
: decide the strengths with which the
neurons are interconnected.

Weights are selected to get a “good match” of
network output to the output of a training set.

Training set is a set of inputs and desired outputs.

The weight selection is conducted by the use of a
learning algorithm.

Learning Processes

Neural Networks

President University

Erwin Sitompul

NNFL

2/
5

Stage 1: Network Training

Training Data

Stage 2: Network Validation

Artificial neural network

Input and output
coverage

Learning

Process

In the form of a set
of optimized
synaptic weights
and biases

Unseen Data

From the same
range as the
training data

Artificial neural network

Implementation

Phase

Learning Processes

Neural Networks

Learning Process

Knowledge

Output
Prediction

President University

Erwin Sitompul

NNFL

2/
6

Learning Process

Learning

is a process
by which the free
parameters of a neural
through a process of
stimulation by the
environment in which
the network is
embedded.

In most cases, due to
complex optimization
plane, the optimized
weights and biases are
obtained as a result of
a number of learning
iterations.

[
w
,
b
]

x

y

[
w,b
]
0

x

y
(0)

Initialize: Iteration (0)

[
w,b
]
1

x

y
(1)

Iteration (1)

[
w,b
]
n

x

y
(
n
)

d

Iteration (
n
)

ANN

d

: desired output

Learning Processes

Neural Networks

President University

Erwin Sitompul

NNFL

2/
7

Learning Rules

Learning Processes

Neural Networks

Error Correction Learning

Delta Rule or Widrow
-
Hoff Rule

Memory Based Learning

Nearest Neighbor Rule

Hebbian Learning

Synchronous activation increases the synaptic
strength

Asynchronous activation decreases the synaptic
strength

Competitive Learning

Boltzmann Learning

President University

Erwin Sitompul

NNFL

2/
8

w
k
1
(
n
)

x
1

x
2

x
m

Inputs

Synaptic

weights

Bias

Activation
function

w
k
2
(
n
)

w
km
(
n
)

Output

y
k

(
n
)

Desired output

d
k

(
n
)

e
k

(
n
)

+

f
(.)

b
k
(
n
)

1

-

Error

signal

Learning Processes

Neural Networks

Error
-
Correction Learning

Learning

Rule

President University

Erwin Sitompul

NNFL

2/
9

Learning Processes

Neural Networks

Delta Rule (Widrow
-
Hoff Rule)

Minimization of a cost function
(or performance index)

President University

Erwin Sitompul

NNFL

2/
10

w
kj
(0) = 0

y
k
(
n
) =

[
w
kj
(n
)
x
j
(
n
)]

w
kj
(
n
+1) =
w
kj
(
n
) +
h
[
d
k
(
n
)

y
k
(
n
)]
x
j
(
n
)

h

: learning rate, [0…1]

n

=
n
+1

n

= 0

Least Means Squares Rule

Learning Processes

Neural Networks

Delta Rule (Widrow
-
Hoff Rule)

President University

Erwin Sitompul

NNFL

2/
11

Learning Processes

Neural Networks

ANN

Error

Desired

Actual

+

-

Environment

(Data)

Teacher

(Expert)

Supervised

Unsupervised

Environment

(Data)

Delay

ANN

Delayed

Reinforcement

Learning

Cost

Function

President University

Erwin Sitompul

NNFL

2/
12

Single Layer Perceptrons

Neural Networks

Single Layer Perceptrons

Output unit is independent
of the others.

Analysis can be limited to
single output perceptron.

Single
-
layer perceptron network is a network with
all

the inputs connected
directly

to the output(s).

President University

Erwin Sitompul

NNFL

2/
13

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons

w
1

w
2

E
(
w
)

Key idea
: Learning is performed by adjusting the
weights in order to minimize the sum of squared errors
on a training.

Weights
are
u
pdated repeatedly (in each
epoch/iteration).

Sum of squared errors is a classical error measure (
e
.
g
.
commonly used in linear regression).

Learning can be
viewed as an
optimization search
problem in weight
space
.

President University

Erwin Sitompul

NNFL

2/
14

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons

The learning rule performs a search within the
solution's vector space towards a
global minimum
.

T
he error surface itself is a
hyper
-
paraboloid

but is
seldom as smooth as is
depicted below.

In most problems, the solution
space is quite irregular with
numerous pits and hills which
may cause the network to settle
down in a
local minimum

(not
the best overall solution).

Epochs
are repeated until
stopping criterion is reached
(error magnitude, number of
iterations, change of weights,
etc).

President University

Erwin Sitompul

NNFL

2/
15

Single Layer Perceptrons

Neural Networks

Derivation of a Learning Rule for Perceptrons

Widrow [1962]

x
1

x
2

x
m

w
k
1

w
k
2

w
km

.

.

.

Goal:

President University

Erwin Sitompul

NNFL

2/
16

Least Mean Squares (LMS)

Single Layer Perceptrons

Neural Networks

The following
cost

function (
error

function) should be
minimized:

President University

Erwin Sitompul

NNFL

2/
17

Single Layer Perceptrons

Neural Networks

Least Mean Squares (LMS)

Letting

f
(
w
k
) =
f
(
w
k
1
,
w
k
2
,

,
w
km
)
be a function over

R
m
,
then

Defining

President University

Erwin Sitompul

NNFL

2/
18

f

w

f

w

df
:
positive

df

:
zero

df

:
negative

go uphill

plain

go downhill

f

w

To minimize
f
, we choose

Single Layer Perceptrons

Neural Networks

President University

Erwin Sitompul

NNFL

2/
19

Single Layer Perceptrons

Neural Networks

With

then

Weight Modification Rule

Defining

we can write

President University

Erwin Sitompul

NNFL

2/
20

Single Layer Perceptrons

Neural Networks

Batch

Learning Mode

Incremental
Learning Mode

President University

Erwin Sitompul

NNFL

2/
21

-
Learning Rule

LMS Algorithm

Widrow
-
Hoff Learning Rule

Single Layer Perceptrons

Neural Networks

President University

Erwin Sitompul

NNFL

2/
22

Single Layer Perceptrons

Neural Networks

Generalization and Early Stopping

By proper training, a neural
network may produce
reasonable output for inputs
not seen

during training

G
eneralization

Generalization is particularly
useful for the analysis of a
“noisy” data (e.g. time

series
)

“Overtraining” will not improve
the ability of a neural network
to produce good output.

On the contrary, it will try to
take noise as the real data
and lost its generality.

President University

Erwin Sitompul

NNFL

2/
23

Generalization and Early Stopping

Single Layer Perceptrons

Neural Networks

Overfitting vs

Generalization

President University

Erwin Sitompul

NNFL

2/
24

Homework 1

Single Layer Perceptrons

Neural Networks

Given a function y = 4
x
2
, you are required to find the
value of
x

that will result
y

= 2 by using the Least Mean
Squares method.

Use initial estimate
x
0

= 1 and learning rate
η

= 0.01.

Write down the results of the first 10
epochs/iterations.

Note
: Calculation can be done manually or using
Matlab.