Neural Networks

cartcletchAI and Robotics

Oct 19, 2013 (3 years and 9 months ago)

73 views

Neural Networks

Objectives


To improve our understanding of
Neural Networks and how they work


To see how to implement Neural
Network Models using iDA


To understand strengths and
weaknesses of Neural Network
models

Neural Networks
: Phisiological
Motivation


Human brain has approximately 100
billion neurons (10
11
)


Each neuron has approximately 1000
dendrite, 10
14

synapses


Each neuron operates at
a
pproximately 100 Hz.


Neurons act in parallel


10
16

interconnections/sec.



i
1

i
2

i
n

o
1

o
2



Brain’s structure can be captured in
electronic circuitry.


Therefore the same structure can be
captured by a mathematical network
model.


The mathematical model can be
captured in a special software.

Figure 8.1 A fully connected feed
-
forward neural network

A
Feed
-
Forward Neural Network


Equation 8.1

Neural Network Input Format


Inputs have to be numeric and in the
interval [0,1]

Neural Network Output Format


A Neural Network model may have
multiple outputs


The output field is numeric, and in
the interval [0,1]

Handling Categorical Data


Inputs


Value discretization


E.g. {green, yellow, blue} <
-
>{0,0.5,1}


Binary coding


E.g. {green, yellow, blue} <
-
>{(0,0),
(1,0),(1,1)}


Outputs


Interpret as probabilities


Convert into categorical by assigning ranges


Challenge: dealing with “inconclusive” values

Equation 8.2

The Sigmoid Function


Figure 8.2 The sigmoid function

Neural Network Training: A
Conceptual View


Supervised Learning with Feed
-
Forward Networks


Backpropagation Learning



Genetic Learning


Unsupervised Clustering with Self
-
Organizing Maps

Figure 8.3 A 3x3 Kohonen network
with two input layer nodes

Equation 8.3

Backpropagation Error

Output Layer


Equation 8.4

Backpropagation Error

Output Layer


Equation 8.5

Backpropagation Error

Hidden Layer


Equations 8.6 and 8.7

The Delta Rule


Equation 8.8

Mean Squared Error


The Backpropogation Algorithm

1.
Initialize the network


Create the network topology (input,
hidden, output layers)


Initialize the weights


Cho
o
se a value for learning parameter


Cho
o
se a termination condition


The Backpropogation Algorithm

2. For all training set instances


Feed the instance through the network


Determine the output error


Update the network weights


Repeat step 2 until the termination
condition is met:


Number of epochs (passes/trials)


RMSE convergence (stop if <0.10)


The Backpropogation Algorithm


3. Evaluation


Test the accuracy of the network on a
test set.


If not satisfactory, re
-
initialize and start
over.

Figure 8.4 Connections for two
output layer nodes

Kohonen Self
-
Organizing Maps: An
Example


Equation 8.9

Classifying a New Instance


Output Node = j


Equation 8.10

Adjusting the Weight Vectors

Output Node = j


Neural Network Explanation


Generating rules based on the results



Sensitivity Analysis



General Considerations


What input attributes will be used to build the
network?


How will the network output be represented?


How many hidden layers should the network contain?


How many nodes should there be in each hidden
layer?


What condition will terminate network training?




All difficult questions with no straight
-
forward answers

Neural Network Strengths



Work well with noisy data.



Can process numeric and categorical data.



Appropriate for applications requiring a
time element.



Have performed well in several domains.



Appropriate for supervised learning and
unsupervised clustering.


Neural Network Weaknesses



Lack explanation capabilities



May not provide optimal solutions to
problems


Too much experimentation might be time
consuming



Overtraining can be a problem




Building Neural Networks with
iDA


A Four
-
Step Approach


Backpropagation Learning

(supervised)


Unsupervised clustering



1.
Prepare the data to be mined.

2.
Define the network architecture.

3.
Watch the network train.

4.
Read and interpret summary results.


Example 1: Modeling the
Exclusive
-
OR Function

Figure 9.1A graph of the XOR
function

XOR: Not Linearly Separable


Figure 9.2 XOR training data

Step 1: Prepare The Data To Be
Mined


Figure 9.3 Dialog box for
supervised learning

Step 2: Define The Network
Architecture


Figure 9.4 Training options for
backpropagation learning


Number of
nodes in
hidden layers


Algorithm
parameter
settings


Epochs:
number of
passes

Figure 9.5 Neural network
execution window

Step 3: Watch The Network Train

Figure 9.6 XOR output file for
Experiment 1

Step 4: Read and Interpret
Summary Results

Example 2: Cardiology Numerical
Dataset


CardiologyNumericalNN.xls

A Four
-
Step Approach for
Neural Network Clustering

1.
Prepare the data to be mined.


The Deer Hunter Dataset

2.
Define the network architecture.

3.
Watch the network train.

4.
Read and interpret summary results.



Figure 9.12 Learning parameters
for unsupervised clustering

Step 2: Define The Network
Architecture

Figure 9.13 Network execution
window

Step 3: Watch The Network Train

Figure 9.14 Deer hunter data:
Unsupervised summary statistics

Step 4: Read And Interpret
Summary Results


Figure 9.15 Output clusters for the
deer hunter dataset