30supervised-BP

Supervised
Learning
Networks

Supervised Learning Networks

•
Linear perceptron networks


•
Multi
-
layer perceptrons

•
Mixture of experts

•
Decision
-
based neural networks

•
Hierarchical neural networks

Two
-
Level:

(b) Linear perceptron networks

(c) decision
-
based neural network.

(d) mixture of experts network.

Hierarchical Neural Network Structures

Three
-
Level:

(e) experts
-
in
-
class network.

(f) classes
-
in
-
expert network.

One
-
Level:

(a) multi
-
layer perceptrons.

Hierarchical Structure of NN

•
1
-
level hierarchy:

BP


•
2
-
level hierarchy
: MOE,DBNN


•
3
-
level hierarchy:

PDBNN

“Synergistic Modeling and Applications of
Hierarchical Fuzzy

Neural Networks”,

by S.Y. Kung, et al., Proceedings of the IEEE, Special Issue on
Computational Intelligence, Sept. 1999

All Classes in One Net

multi
-
layer perceptron

Divide
-
and
-
conquer principle: divide the task into
modules and then integrate the individual results
into a collective decision.

Modular Structures (two
-
level)

Two typical modular networks:

(1) mixture
-
of
-
experts (MOE) which
utilizes the expert
-
level modules,

(2) decision
-
based neural networks
(DBNN) based on the class
-
level modules.

Each expert serves the function of

(1)

extracting local features and

(2)

making local recommendations.


The rules in the gating network are used to decide how to
combine recommendations from several local experts, with
corresponding degree of confidences.

Expert
-
level (Rule
-
level) Modules:

mixture of experts network

Class
-
level modules are natural basic
partitioning units, where each module
specializes in distinguishing its own class
from the others.

Class
-
level modules:

In contrast to expert
-
level partitioning, this OCON structure
facilitates a global (or mutual) supervised training scheme.
In global inter
-
class supervised learning, any dispute over a
pattern region by (two or more) competing classes may be
effectively resolved by resorting to the teacher's guidance.

Decision Based Neural Network

Depending on the order used, two kinds of
hierarchical networks:

•
one has an experts
-
in
-
class construct and

•
another a classes
-
in
-
expert Construct.


Three
-
level hierarchical structures:

Apply the divide
-
and
-
conquer principle twice:

one time on the expert
-
level and another on the class
-
level.

Classes
-
in
-
Expert Network

Experts
-
in
-
Class Network

Multilayer Back
-
Propagation Networks

A BP Multi
-
Layer Perceptron(MLP) possesses adaptive
learning abilities to estimate sampled functions, represent
these samples, encode structural knowledge, and inference
inputs to outputs via association.

Its main strength lies in its (sufficiently large number of )
hidden units, thus a large number of interconnections.

The MLP neural networks enhance the ability to learn and
generalize from training data. Namely, MLP can
approximate almost any function.

BP Multi
-
Layer Perceptron(MLP)

A 3
-
Layer Network

Neuron Units: Activation Function

Linear Basis Function (LBF)

RBF NN is More Suitable for
Probabilistic Pattern Classification

MLP

RBF

Hyperplane

Kernel function

The probability density function (also called conditional density
function or likelihood) of the k
-
th class is defined as

The centers and widths of the RBF Gaussian kernels are


deterministic functions of the training data;

RBF BP Neural Network

•
According to Bays’ theorem, the posterior prob. is

where
P
(
C
k
) is the prior prob. and

RBF Output as Probability Function


MLPs

are

highly

non
-
linear

in

the

parameter

space


杲慤楥湴

摥獣d湴



汯捡l

m楮ima


RBF

networks

solve

this

problem

by

dividing

the

learning

into

two

independent

processes
.


1.
Use the
K
-
mean algorithm to find
c
i
and determine
weights
w

using the least square method

2.
RBF

learning

by

gradient

descent



Comparison of RBF and MLP

x
p

K
-
means

K
-
Nearest

Neighbor

Basis

Functions

Linear

Regression

c
i

c
i


i

A

w


RBF

learning

process


RBF

networks

implement

the

function


w
i



i

and

c
i

can

be

determined

separately





䙡獴

汥慲湩湧

慬杯物a桭


Basis

function

types


Finding

the

RBF

Parameters

(1 ) Use the
K
-
mean algorithm to find
c
i

Centers and widths found by K
-
means and K
-
NN


Use

K

nearest

neighbor

rule

to

find

the

function

width






k
-
th nearest neighbor of
c
i


The

objective

is

to

cover

the

training

points

so

that

a

smooth

fit

of

the

training

samples

can

be

achieved


For

Gaussian

basis

functions




Assume

the

variance



慣a潳

e慣a

摩浥湳楯d

慲e

敱畡e



To

write

in

matrix

form,

let


Determining

weights

w

using

the

least

square

method


where

d
p

is

the

desired

output

for

pattern

p

(
2
)

RBF

learning

by

gradient

descent

we

have

Apply

we

have

the

following

update

equations


Elliptical Basis Function networks

: function centers

: covariance matrix


EBF Vs. RBF networks

RBFN with 4 centers

EBFN with 4 centers

MatLab Assignment #3: RBF BP Network to separate 2 classes

RBF BP with 4 hidden units

EBF BP with 4 hidden units

ratio=2:1




