Neural Networks
Objectives
To improve our understanding of
Neural Networks and how they work
To see how to implement Neural
Network Models using iDA
To understand strengths and
weaknesses of Neural Network
models
Neural Networks
: Phisiological
Motivation
Human brain has approximately 100
billion neurons (10
11
)
Each neuron has approximately 1000
dendrite, 10
14
synapses
Each neuron operates at
a
pproximately 100 Hz.
Neurons act in parallel
10
16
interconnections/sec.
i
1
i
2
i
n
o
1
o
2
Brain’s structure can be captured in
electronic circuitry.
Therefore the same structure can be
captured by a mathematical network
model.
The mathematical model can be
captured in a special software.
Figure 8.1 A fully connected feed

forward neural network
A
Feed

Forward Neural Network
Equation 8.1
Neural Network Input Format
Inputs have to be numeric and in the
interval [0,1]
Neural Network Output Format
A Neural Network model may have
multiple outputs
The output field is numeric, and in
the interval [0,1]
Handling Categorical Data
Inputs
Value discretization
E.g. {green, yellow, blue} <

>{0,0.5,1}
Binary coding
E.g. {green, yellow, blue} <

>{(0,0),
(1,0),(1,1)}
Outputs
Interpret as probabilities
Convert into categorical by assigning ranges
Challenge: dealing with “inconclusive” values
Equation 8.2
The Sigmoid Function
Figure 8.2 The sigmoid function
Neural Network Training: A
Conceptual View
Supervised Learning with Feed

Forward Networks
Backpropagation Learning
Genetic Learning
Unsupervised Clustering with Self

Organizing Maps
Figure 8.3 A 3x3 Kohonen network
with two input layer nodes
Equation 8.3
Backpropagation Error
Output Layer
Equation 8.4
Backpropagation Error
Output Layer
Equation 8.5
Backpropagation Error
Hidden Layer
Equations 8.6 and 8.7
The Delta Rule
Equation 8.8
Mean Squared Error
The Backpropogation Algorithm
1.
Initialize the network
Create the network topology (input,
hidden, output layers)
Initialize the weights
Cho
o
se a value for learning parameter
Cho
o
se a termination condition
The Backpropogation Algorithm
2. For all training set instances
Feed the instance through the network
Determine the output error
Update the network weights
Repeat step 2 until the termination
condition is met:
Number of epochs (passes/trials)
RMSE convergence (stop if <0.10)
The Backpropogation Algorithm
3. Evaluation
Test the accuracy of the network on a
test set.
If not satisfactory, re

initialize and start
over.
Figure 8.4 Connections for two
output layer nodes
Kohonen Self

Organizing Maps: An
Example
Equation 8.9
Classifying a New Instance
Output Node = j
Equation 8.10
Adjusting the Weight Vectors
Output Node = j
Neural Network Explanation
Generating rules based on the results
Sensitivity Analysis
General Considerations
What input attributes will be used to build the
network?
How will the network output be represented?
How many hidden layers should the network contain?
How many nodes should there be in each hidden
layer?
What condition will terminate network training?
All difficult questions with no straight

forward answers
Neural Network Strengths
Work well with noisy data.
Can process numeric and categorical data.
Appropriate for applications requiring a
time element.
Have performed well in several domains.
Appropriate for supervised learning and
unsupervised clustering.
Neural Network Weaknesses
Lack explanation capabilities
May not provide optimal solutions to
problems
Too much experimentation might be time
consuming
Overtraining can be a problem
Building Neural Networks with
iDA
A Four

Step Approach
Backpropagation Learning
(supervised)
Unsupervised clustering
1.
Prepare the data to be mined.
2.
Define the network architecture.
3.
Watch the network train.
4.
Read and interpret summary results.
Example 1: Modeling the
Exclusive

OR Function
Figure 9.1A graph of the XOR
function
XOR: Not Linearly Separable
Figure 9.2 XOR training data
Step 1: Prepare The Data To Be
Mined
Figure 9.3 Dialog box for
supervised learning
Step 2: Define The Network
Architecture
Figure 9.4 Training options for
backpropagation learning
Number of
nodes in
hidden layers
Algorithm
parameter
settings
Epochs:
number of
passes
Figure 9.5 Neural network
execution window
Step 3: Watch The Network Train
Figure 9.6 XOR output file for
Experiment 1
Step 4: Read and Interpret
Summary Results
Example 2: Cardiology Numerical
Dataset
CardiologyNumericalNN.xls
A Four

Step Approach for
Neural Network Clustering
1.
Prepare the data to be mined.
The Deer Hunter Dataset
2.
Define the network architecture.
3.
Watch the network train.
4.
Read and interpret summary results.
Figure 9.12 Learning parameters
for unsupervised clustering
Step 2: Define The Network
Architecture
Figure 9.13 Network execution
window
Step 3: Watch The Network Train
Figure 9.14 Deer hunter data:
Unsupervised summary statistics
Step 4: Read And Interpret
Summary Results
Figure 9.15 Output clusters for the
deer hunter dataset
Comments 0
Log in to post a comment