Supervised
Learning
Networks
Supervised Learning Networks
•
Linear perceptron networks
•
Multi

layer perceptrons
•
Mixture of experts
•
Decision

based neural networks
•
Hierarchical neural networks
Two

Level:
(b) Linear perceptron networks
(c) decision

based neural network.
(d) mixture of experts network.
Hierarchical Neural Network Structures
Three

Level:
(e) experts

in

class network.
(f) classes

in

expert network.
One

Level:
(a) multi

layer perceptrons.
Hierarchical Structure of NN
•
1

level hierarchy:
BP
•
2

level hierarchy
: MOE,DBNN
•
3

level hierarchy:
PDBNN
“Synergistic Modeling and Applications of
Hierarchical Fuzzy
Neural Networks”,
by S.Y. Kung, et al., Proceedings of the IEEE, Special Issue on
Computational Intelligence, Sept. 1999
All Classes in One Net
multi

layer perceptron
Divide

and

conquer principle: divide the task into
modules and then integrate the individual results
into a collective decision.
Modular Structures (two

level)
Two typical modular networks:
(1) mixture

of

experts (MOE) which
utilizes the expert

level modules,
(2) decision

based neural networks
(DBNN) based on the class

level modules.
Each expert serves the function of
(1)
extracting local features and
(2)
making local recommendations.
The rules in the gating network are used to decide how to
combine recommendations from several local experts, with
corresponding degree of confidences.
Expert

level (Rule

level) Modules:
mixture of experts network
Class

level modules are natural basic
partitioning units, where each module
specializes in distinguishing its own class
from the others.
Class

level modules:
In contrast to expert

level partitioning, this OCON structure
facilitates a global (or mutual) supervised training scheme.
In global inter

class supervised learning, any dispute over a
pattern region by (two or more) competing classes may be
effectively resolved by resorting to the teacher's guidance.
Decision Based Neural Network
Depending on the order used, two kinds of
hierarchical networks:
•
one has an experts

in

class construct and
•
another a classes

in

expert Construct.
Three

level hierarchical structures:
Apply the divide

and

conquer principle twice:
one time on the expert

level and another on the class

level.
Classes

in

Expert Network
Experts

in

Class Network
Multilayer Back

Propagation Networks
A BP Multi

Layer Perceptron(MLP) possesses adaptive
learning abilities to estimate sampled functions, represent
these samples, encode structural knowledge, and inference
inputs to outputs via association.
Its main strength lies in its (sufficiently large number of )
hidden units, thus a large number of interconnections.
The MLP neural networks enhance the ability to learn and
generalize from training data. Namely, MLP can
approximate almost any function.
BP Multi

Layer Perceptron(MLP)
A 3

Layer Network
Neuron Units: Activation Function
Linear Basis Function (LBF)
RBF NN is More Suitable for
Probabilistic Pattern Classification
MLP
RBF
Hyperplane
Kernel function
The probability density function (also called conditional density
function or likelihood) of the k

th class is defined as
The centers and widths of the RBF Gaussian kernels are
deterministic functions of the training data;
RBF BP Neural Network
•
According to Bays’ theorem, the posterior prob. is
where
P
(
C
k
) is the prior prob. and
RBF Output as Probability Function
MLPs
are
highly
non

linear
in
the
parameter
space
杲慤楥湴
摥獣d湴
汯捡l
m楮ima
RBF
networks
solve
this
problem
by
dividing
the
learning
into
two
independent
processes
.
1.
Use the
K

mean algorithm to find
c
i
and determine
weights
w
using the least square method
2.
RBF
learning
by
gradient
descent
Comparison of RBF and MLP
x
p
K

means
K

Nearest
Neighbor
Basis
Functions
Linear
Regression
c
i
c
i
i
A
w
RBF
learning
process
RBF
networks
implement
the
function
w
i
i
and
c
i
can
be
determined
separately
䙡獴
汥慲湩湧
慬杯物a桭
Basis
function
types
Finding
the
RBF
Parameters
(1 ) Use the
K

mean algorithm to find
c
i
Centers and widths found by K

means and K

NN
Use
K
nearest
neighbor
rule
to
find
the
function
width
k

th nearest neighbor of
c
i
The
objective
is
to
cover
the
training
points
so
that
a
smooth
fit
of
the
training
samples
can
be
achieved
For
Gaussian
basis
functions
Assume
the
variance
慣a潳
e慣a
摩浥湳楯d
慲e
敱畡e
To
write
in
matrix
form,
let
Determining
weights
w
using
the
least
square
method
where
d
p
is
the
desired
output
for
pattern
p
(
2
)
RBF
learning
by
gradient
descent
we
have
Apply
we
have
the
following
update
equations
Elliptical Basis Function networks
: function centers
: covariance matrix
EBF Vs. RBF networks
RBFN with 4 centers
EBFN with 4 centers
MatLab Assignment #3: RBF BP Network to separate 2 classes
RBF BP with 4 hidden units
EBF BP with 4 hidden units
ratio=2:1
Comments 0
Log in to post a comment