Linear System Estimation

gilamonsterbirdsElectronics - Devices

Nov 24, 2013 (3 years and 6 months ago)

111 views

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Neural Networks Models for
Pattern Classification and
Linear System Estimation

Docent Xiao
-
Zhi Gao

Department of Automation and Systems Technology

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Structure of Human Brain

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Human Neurons

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Neural Networks (NN)


Neural networks are highly simplified models of
human brain to deal with specific tasks


massively connected neurons


Several kinds of neural networks are proposed


feedforward neural networks


recurrent neural networks


supervised neural networks


unsupervised neural networks


Applications of neural networks are intensive


Institute of Intelligent Power Electronics


IPE

Page

‹#›

Neural Networks (Feedforward)

Back
-
Propagation
Neural Networks

(Multiple Layer
Perceptron)

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Neural Networks (Recurrent)

Hopfield Neural Networks

(Hopfield [1982] [1986])

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Neural Networks Models


Perceptron


Pattern classification


Adaptive Linear Element (Adaline)


Linear system identification


Back
-
Propagation neural network (BP)


Nonlinear system identification


Time series prediction

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Perceptron


First and simplest neural network model


firstly proposed by Rosenblatt in 1958


pattern classicification applications

»
AND and OR classification


Feature input
vs

Binary output


Learning algorithm is based on
Hebb

principle


Perceptron
cannot

solve non
-
linearly
separable classification problems


XOR problem

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Perceptron

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Perceptron Learning Algorithm


Perceptron output:


Desired output:


Based on Hebb principle (Hebb 1940s)


Learning algorithm





d
(
0
,
1


)

w
(
k

1
)

w
(
k
)


(
d

y
)
x

y

f(
w
i
i

0
n

x
i
)

x
0

1
,
w
0



Institute of Intelligent Power Electronics


IPE

Page

‹#›

XOR Truth Table

X
Y
C
la
s
s
0
0
0
0
1
1
1
0
1
1
1
0
Institute of Intelligent Power Electronics


IPE

Page

‹#›

XOR Classification Problem

-0.5
0
0.5
1
1.5
-0.5
0
0.5
1
1.5
input 1
input 2
TextEnd
o: class 1, x: class 2
Cannot

be
solved by
Perceptron

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Linearly Separable and Non
-
linearly Separable Patterns

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Adaptive Linear Element (Adaline)


Linear

network firstly proposed by
Widrow (1960s)


Analog

input and output


Learning algorithm is based on Least
Mean Square (LMS) principle or
Widrow
-
Hoff rule


Linear system identification with Adaline

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Bernard Widrow (1929
-
)

Professor

Electrical Engineering Department

Stanford University

Institute of Intelligent Power Electronics


IPE

Page

‹#›

B. Widrow and S.D. Stearns,
Adaptive Signal Processing,
Prentice Hall, Englewood Cliffs,
NJ, 1985.

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Adaptive Linear Element (Adaline)

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Adaline Learning Algorithm


Output:


Approximation error:


Gradient descent



Weight update



O

w
i
x
i
i

0
n


E

1
2
(
d

O
)
2


E

w
i


E

O

O

w
i


(
d

O
)

O

w
i


(
d

O
)
x
i


w
i




E

w
i


(
d

O
)
x
i
)

,
1
(
0
0




w
x
Institute of Intelligent Power Electronics


IPE

Page

‹#›

Multilayer Adaline

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Approximation Error Surface of Adaline

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Linear System Identification Using Adaline

5
.
1
5
.
0
1
)
(
)
(
2



z
z
z
x
z
y
Institute of Intelligent Power Electronics


IPE

Page

‹#›

Identification Signal



)
sin(
10
sin
)
(
t
t
t
x

Institute of Intelligent Power Electronics


IPE

Page

‹#›

System Output

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Testing Signal

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Identification Output

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Identification Error

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Conclusions


Perceptron and Adaline are introduced
for pattern classification and linear
system modeling


Perceptron and Adaline are both
supervised learning neural network
models


They are suitable for simple data mining
tasks

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Computer Exercises


Solve AND and OR classification
problems using perceptron


Design a perceptron to solve the two
-
input
AND classification problem


Design a perceptron to solve the two
-
input
OR classification problem


Demonstrate perceptron cannot solve
the XOR classification problem


Institute of Intelligent Power Electronics


IPE

Page

‹#›

AND Classification Problem

Institute of Intelligent Power Electronics


IPE

Page

‹#›

OR Classification Problem

Institute of Intelligent Power Electronics


IPE

Page

‹#›

XOR Classification Problem

Generalize your results of three cases to three inputs!

Institute of Intelligent Power Electronics


IPE

Page

‹#›

Linear System Modeling Using Adaline


Design an Adaline to identify a discrete
linear system



Show your identification results with
identification and testing input signals: