Neural Networks

Neural Networks

Objectives


To improve our understanding of
Neural Networks and how they work


To see how to implement Neural
Network Models using iDA


To understand strengths and
weaknesses of Neural Network
models

Neural Networks
: Phisiological
Motivation


Human brain has approximately 100
billion neurons (10
11
)


Each neuron has approximately 1000
dendrite, 10
14

synapses


Each neuron operates at
a
pproximately 100 Hz.


Neurons act in parallel


10
16

interconnections/sec.



i
1

i
2

i
n

o
1

o
2



Brain’s structure can be captured in
electronic circuitry.


Therefore the same structure can be
captured by a mathematical network
model.


The mathematical model can be
captured in a special software.

Figure 8.1 A fully connected feed
-
forward neural network

A
Feed
-
Forward Neural Network


Equation 8.1

Neural Network Input Format


Inputs have to be numeric and in the
interval [0,1]

Neural Network Output Format


A Neural Network model may have
multiple outputs


The output field is numeric, and in
the interval [0,1]

Handling Categorical Data


Inputs


Value discretization


E.g. {green, yellow, blue} <
-
>{0,0.5,1}


Binary coding


E.g. {green, yellow, blue} <
-
>{(0,0),
(1,0),(1,1)}


Outputs


Interpret as probabilities


Convert into categorical by assigning ranges


Challenge: dealing with “inconclusive” values

Equation 8.2

The Sigmoid Function


Figure 8.2 The sigmoid function

Neural Network Training: A
Conceptual View


Supervised Learning with Feed
-
Forward Networks


Backpropagation Learning



Genetic Learning


Unsupervised Clustering with Self
-
Organizing Maps

Figure 8.3 A 3x3 Kohonen network
with two input layer nodes

Equation 8.3

Backpropagation Error

Output Layer


Equation 8.4

Backpropagation Error

Output Layer


Equation 8.5

Backpropagation Error

Hidden Layer


Equations 8.6 and 8.7

The Delta Rule


Equation 8.8

Mean Squared Error


The Backpropogation Algorithm

1.
Initialize the network


Create the network topology (input,
hidden, output layers)


Initialize the weights


Cho
o
se a value for learning parameter


Cho
o
se a termination condition


The Backpropogation Algorithm

2. For all training set instances


Feed the instance through the network


Determine the output error


Update the network weights


Repeat step 2 until the termination
condition is met:


Number of epochs (passes/trials)


RMSE convergence (stop if <0.10)


The Backpropogation Algorithm


3. Evaluation


Test the accuracy of the network on a
test set.


If not satisfactory, re
-
initialize and start
over.

Figure 8.4 Connections for two
output layer nodes

Kohonen Self
-
Organizing Maps: An
Example


Equation 8.9

Classifying a New Instance


Output Node = j


Equation 8.10

Adjusting the Weight Vectors

Output Node = j


Neural Network Explanation


Generating rules based on the results



Sensitivity Analysis



General Considerations


What input attributes will be used to build the
network?


How will the network output be represented?


How many hidden layers should the network contain?


How many nodes should there be in each hidden
layer?


What condition will terminate network training?




All difficult questions with no straight
-
forward answers

Neural Network Strengths



Work well with noisy data.



Can process numeric and categorical data.



Appropriate for applications requiring a
time element.



Have performed well in several domains.



Appropriate for supervised learning and
unsupervised clustering.


Neural Network Weaknesses



Lack explanation capabilities



May not provide optimal solutions to
problems


Too much experimentation might be time
consuming



Overtraining can be a problem




Building Neural Networks with
iDA


A Four
-
Step Approach


Backpropagation Learning

(supervised)


Unsupervised clustering



1.
Prepare the data to be mined.

2.
Define the network architecture.

3.
Watch the network train.

4.
Read and interpret summary results.


Example 1: Modeling the
Exclusive
-
OR Function

Figure 9.1A graph of the XOR
function

XOR: Not Linearly Separable


Figure 9.2 XOR training data

Step 1: Prepare The Data To Be
Mined


Figure 9.3 Dialog box for
supervised learning

Step 2: Define The Network
Architecture


Figure 9.4 Training options for
backpropagation learning


Number of
nodes in
hidden layers


Algorithm
parameter
settings


Epochs:
number of
passes

Figure 9.5 Neural network
execution window

Step 3: Watch The Network Train

Figure 9.6 XOR output file for
Experiment 1

Step 4: Read and Interpret
Summary Results

Example 2: Cardiology Numerical
Dataset


CardiologyNumericalNN.xls

A Four
-
Step Approach for
Neural Network Clustering

1.
Prepare the data to be mined.


The Deer Hunter Dataset

2.
Define the network architecture.

3.
Watch the network train.

4.
Read and interpret summary results.



Figure 9.12 Learning parameters
for unsupervised clustering

Step 2: Define The Network
Architecture

Figure 9.13 Network execution
window

Step 3: Watch The Network Train

Figure 9.14 Deer hunter data:
Unsupervised summary statistics

Step 4: Read And Interpret
Summary Results


Figure 9.15 Output clusters for the
deer hunter dataset