y - Engineering Computer Systems Support - University of Iowa

puffautomaticBiotechnology

Dec 10, 2012 (4 years and 6 months ago)

154 views

2/21/2008
1
Neural Networks
Andrew Kusiak
I t lli t S t L b t
The University of Iowa
Intelligent Systems Laboratory
I
n
t
e
lli
gen
t

S
ys
t
ems
L
a
b
ora
t
ory
2139 Seamans Center
The University of Iowa
Iowa City, IA 52242 - 1527
andrew-kusiak@uiowa.edu
http://www.icaen.uiowa.edu/~ankusiak
Tel. 319-335 5934
The University of Iowa
Intelligent Systems Laboratory
Neural Network
Neuron 1
Age = 51
Input
Output
The University of Iowa
Intelligent Systems Laboratory
Neuron 2
Neuron 3
Temperature = 40C
Disease A = Yes
Neural Network
Neuron 1
Age = .70
Weight 1 = .6
The University of Iowa
Intelligent Systems Laboratory
Neuron 2
Neuron 3
Temperature = .85
Disease A = .59 (= .7.6 + .85.2)
Weight 2 = .2
Neural Network
1
. Design a neural network structure.
2. Assign weights to the connectors.
The University of Iowa
Intelligent Systems Laboratory
3. Train the neural network.
4. Check for stopping criterion, e.g., training error
or the network cross-validation. If the stopping
criterion is not met, go to Step 3; otherwise
go to Step 5.
5. Use the trained neural network for decision-making.
2/21/2008
2
When NNs can be Used?
• In
p
uts are well understood
The University of Iowa
Intelligent Systems Laboratory
p
• Outputs are well understood
• Historical data is available
Neural Networks
• Applicable to a wide range of problems
• Good results even for complex domains
The University of Iowa
Intelligent Systems Laboratory
• Handling of categorical and continuous
variables
• Off-the-shelf software is available
Neural Networks
• Inputs in the range 0 to 1
• Outputs are produced in the range 0 to 1
The University of Iowa
Intelligent Systems Laboratory
• Result are not explained
• May generate local optimum
• Solutions with errors
Application Examples 1
• Banking: Credit Application Evaluation
• Image Signal Processing: Data
Compression
The University of Iowa
Intelligent Systems Laboratory
• Financial: Real Estate Appraisal
• Manufacturing: Manufacturing Process
Control, Chemical Product Design Analysis
• Robotics: Vision Systems
2/21/2008
3
Application Examples 2
• Medicine:
- Predict diagnosis
Determine diagnostic tests
The University of Iowa
Intelligent Systems Laboratory
-
Determine

diagnostic

tests
- Predict length of stay in a hospital
- Predict treatment cost
Biological Neuron vs Processing
Element
Σ
y


W
2
W
1
nucleus
synapse
axon
The University of Iowa
Intelligent Systems Laboratory
Σ
θ
x

W
n
axon
cell body
dendrites
Combination
function
Activation
function
Neural Networks
• Processing Elements
• Weighted Connections

Activation Functions
Σ
θ
y
x



W
n
W
2
W
1
The University of Iowa
Intelligent Systems Laboratory

Activation

Functions
– threshold function:
– sigmoid function:
– complex function:
or
– linear function:
+ −
1 (1 exp( x))
− − + −(exp(x) exp( x)) (exp(x) exp( x))
(2/(1
+
exp(

2x)))

1
x
else
f ( x )
=
1,x
>
0
0,



Neural Network Types
• Architecture
– Feedforward

Feedback (Loops)
The University of Iowa
Intelligent Systems Laboratory
Feedback

(Loops)
• Learning Algorithm
– Supervised
– Unsupervised
2/21/2008
4
Neural Network Types
Supervised Feedforward
• Perceptron
• Multilayer Perceptron
• Cascade Correlation
• ART
(
Ada
p
tive
Supervised & Unsupervised
Feedforward
• RBF (Radial Basis
Function)
• Counter Propagation
The University of Iowa
Intelligent Systems Laboratory
( p
Resonance Theory) Map
• ART 1&2 - Adaptive
Resonance Theory 1&2
• SOM (Self Organizing Map)
Unsupervised Feedforward
Supervised Feedback
• Hopfield
• BAM (Bidirective Associate
Memory)
• Boltzmann-Machine
Feedforward NN
The University of Iowa
Intelligent Systems Laboratory
Example 1
i1
i2
o
The University of Iowa
Intelligent Systems Laboratory
i3
Inputs
Output
Analogous to
regression
analysis
Example 2
i1
i2
o
The University of Iowa
Intelligent Systems Laboratory
i3
Inputs
Output
Intermediate
layer
More powerful
network
2/21/2008
5
Example 3
i1
i2
i3
o
Output
The University of Iowa
Intelligent Systems Laboratory
More neurons in the
intermediate layer
i3
Inputs
Output
More powerful network
with the increased risk
of overfitting
Example 4
i1
i2
o
The University of Iowa
Intelligent Systems Laboratory
i3
Inputs
Outputs
Multiple
output
values
Simple NN Application
Single
i di
$ spent/
month
Age
Apparel
Furniture
The University of Iowa
Intelligent Systems Laboratory
i
nterme
di
ate
layer NN
Gender
Income
Furniture
Entertainment
Propensity of making the next purchase
Illustrative Learning Rules
• Hebbian (Hebb 1949)
• Winner-Takes-All (Competitive learning)
(Kohonnen 1982)

S
im
p
l
e

e
rr
o
r -
co
rr
ect
in
g
r
u
l
e

The University of Iowa
Intelligent Systems Laboratory
S p e e o
co ect g u e
(Rosenblatt 1958)
• Backpropagation error - correcting rule
(Werbos 1974)
• Radial Basis Function (RBF) Network
2/21/2008
6
Learning
Hebbian
x1
x2
x3
y
Output
wij
yj
The University of Iowa
Intelligent Systems Laboratory
wij(t+1) = wij(t) + yj(t)xi(t)
η
0
< <
1
η
learning rate
x3
Inputs
Output
Learning
Winner-Takes-All
x1
x2
3
y
O t t
wi
yj
The University of Iowa
Intelligent Systems Laboratory
x
3
Inputs
O
u
t
pu
t
wi(t+1) = wi(t) + (t) (X - wi(t))
η
X = input vector
The winning neuron’s weight
Learning
Simple error - correcting rule
x1
x2
y
wi
yj
d
The University of Iowa
Intelligent Systems Laboratory
wij(t+1) = wij(t) + (d - y) xi
η
x3
Inputs
Output
Desired
output
Learning
Backpropagation error - correcting rule
x1
x2
x3
y
wi
yj
d
The University of Iowa
Intelligent Systems Laboratory
x3
Inputs
Outpu
t
Desired
output
w(t+1) = w(t) + Δf(x(t+1), w(t)) (d(t +1)) -
- f(x(t+1), w(t)))
η
Δ
= gradient
2/21/2008
7
Learning
Radial Basis Function (RBF) Network
Features:

Based on approximation and regularization theory
The University of Iowa
Intelligent Systems Laboratory

Based

on

approximation

and

regularization

theory
• Global optimum is more likely to be determined
• Short training time
• Simple topology
Mathematical Model of a Neural
Network
Training Patterns , for some
unknown function
in a space of possible functions
x
,
f
(
x
)
f
Φ
The University of Iowa
Intelligent Systems Laboratory
in

a

space

of

possible

functions

and an error function
Function that minimizes
Φ
E
(
ˆ
f
),
ˆ
f
∈Φ
ˆ
f
E
(
ˆ
f
)
Generalization
Problems:
NN is to generalize (examples)
not to memorize
The University of Iowa
Intelligent Systems Laboratory
• Overfitting
• Underfitting
• Network Size
Overfitting
Training
Prediction
The University of Iowa
Intelligent Systems Laboratory
Good Generalization
Overfitting
2/21/2008
8
Number of Training Examples
Given
N
nodes,
W
weights
The number of training examples
p
The University of Iowa
Intelligent Systems Laboratory
• Vapnik-Chervonenkis (VC) Dimension
• Widrow’s rule of thumb
p

O(
W
ε
log
N
ε
)
p

10W
Steps for Building a Neural Network
Model
• Identify Problem

Collect Data (Choose the training set)
The University of Iowa
Intelligent Systems Laboratory

Collect

Data

(Choose

the

training

set)
• Preprocess Data
• Implement a Neural Network
Choosing the Training Set
• The number of features
- NN training time depends on the number
of features
N b f l i i h h b
The University of Iowa
Intelligent Systems Laboratory
-
N
um
b
er o
f
examp
l
e
i
ncreases w
i
t
h
t
h
e num
b
e
r
of NN inputs
• Coverage of the range values of all features
• Number of outputs
- Sufficient coverage in examples
NN Implementation
• What kind of architecture is best for modeling
the underlying problem?

Which learning algorithmcan achieve the best
The University of Iowa
Intelligent Systems Laboratory
Which

learning

algorithm

can

achieve

the

best

generalization?
• What size network gives the best generalization?
• How many training instances are required for good
generalization?
2/21/2008
9
Implementation Sequence
1. Normalize Data
2. Design a Network, e.g., #Layers, #Neurons and
Transfer Function for Each Layer
3 S t T i i P t
The University of Iowa
Intelligent Systems Laboratory
3
.
S
e
t

T
ra
i
n
i
ng
P
arame
t
ers
4. Provide Initial Weights and Save the Weights if
Necessary
5. Training with a Learning Algorithm
6. Testing
Matlab Implementation of NNs
• Getting Started and Quit
% matlab
>> ...
....
>> quit
The University of Iowa
Intelligent Systems Laboratory
>>

quit
• Online Help
>> help
>> help which
>> help lookfor
>> help diary
Variables
>> a=1
>> a=1;
>> A = 2
>> x = 4\1 % x = 1/4
>> C = ‘aStrin
g
’ % ‘Don”t for
g
et’
The University of Iowa
Intelligent Systems Laboratory
g
g
>> b = [1 2 3]
>> d = 1:3
>> e = 3:-1:1 % e = [3 2 1]
>> g = [1 2 3 ...
4 5] % g = [ 1 2 3 4 5]
Matrices
>> B = [ 1 2 3; 4 5 6
7 8 9]
>> P = [10 11 12], B=[B; P]
>> zeros(3, 2)
The University of Iowa
Intelligent Systems Laboratory
>> ones(2, 3)
>> B(1, 2)
>> B(:, 3)
>> B(2, :)
>> B(:, [1 3])
2/21/2008
10
Matrix Operations and Manipulation
>> C = A*B
>> B = B’
>> D = B + 2
>> E = B + B % E = 2 * B
The University of Iowa
Intelligent Systems Laboratory
>> X = [1 2 3], Y=[4 5 6], Z = X.*Y
>> X = A\B % A*X = B
>> X = B/A % X*A = B
>> Y = []
>> Y(:, [1 3])=[]
NN and Genetic Algorithm
Neuron 1
Neuron 3
Input 1
Output
Weight 1
The University of Iowa
Intelligent Systems Laboratory
Neuron 2
Neuron

3
Input 2
Weight 2
Weight 1 Weight 2
11011000 11001111
NN and Evolutionary
Computation
Output
layer
Input
layer
Intermediate
layer
Evolutionary
computation
algorithm
The University of Iowa
Intelligent Systems Laboratory