# Neural Networks

AI and Robotics

Oct 19, 2013 (4 years and 6 months ago)

85 views

Neural Networks

Objectives

To improve our understanding of
Neural Networks and how they work

To see how to implement Neural
Network Models using iDA

To understand strengths and
weaknesses of Neural Network
models

Neural Networks
: Phisiological
Motivation

Human brain has approximately 100
billion neurons (10
11
)

Each neuron has approximately 1000
dendrite, 10
14

synapses

Each neuron operates at
a
pproximately 100 Hz.

Neurons act in parallel

10
16

interconnections/sec.

i
1

i
2

i
n

o
1

o
2

Brain’s structure can be captured in
electronic circuitry.

Therefore the same structure can be
captured by a mathematical network
model.

The mathematical model can be
captured in a special software.

Figure 8.1 A fully connected feed
-
forward neural network

A
Feed
-
Forward Neural Network

Equation 8.1

Neural Network Input Format

Inputs have to be numeric and in the
interval [0,1]

Neural Network Output Format

A Neural Network model may have
multiple outputs

The output field is numeric, and in
the interval [0,1]

Handling Categorical Data

Inputs

Value discretization

E.g. {green, yellow, blue} <
-
>{0,0.5,1}

Binary coding

E.g. {green, yellow, blue} <
-
>{(0,0),
(1,0),(1,1)}

Outputs

Interpret as probabilities

Convert into categorical by assigning ranges

Challenge: dealing with “inconclusive” values

Equation 8.2

The Sigmoid Function

Figure 8.2 The sigmoid function

Neural Network Training: A
Conceptual View

Supervised Learning with Feed
-
Forward Networks

Backpropagation Learning

Genetic Learning

Unsupervised Clustering with Self
-
Organizing Maps

Figure 8.3 A 3x3 Kohonen network
with two input layer nodes

Equation 8.3

Backpropagation Error

Output Layer

Equation 8.4

Backpropagation Error

Output Layer

Equation 8.5

Backpropagation Error

Hidden Layer

Equations 8.6 and 8.7

The Delta Rule

Equation 8.8

Mean Squared Error

The Backpropogation Algorithm

1.
Initialize the network

Create the network topology (input,
hidden, output layers)

Initialize the weights

Cho
o
se a value for learning parameter

Cho
o
se a termination condition

The Backpropogation Algorithm

2. For all training set instances

Feed the instance through the network

Determine the output error

Update the network weights

Repeat step 2 until the termination
condition is met:

Number of epochs (passes/trials)

RMSE convergence (stop if <0.10)

The Backpropogation Algorithm

3. Evaluation

Test the accuracy of the network on a
test set.

If not satisfactory, re
-
initialize and start
over.

Figure 8.4 Connections for two
output layer nodes

Kohonen Self
-
Organizing Maps: An
Example

Equation 8.9

Classifying a New Instance

Output Node = j

Equation 8.10

Output Node = j

Neural Network Explanation

Generating rules based on the results

Sensitivity Analysis

General Considerations

What input attributes will be used to build the
network?

How will the network output be represented?

How many hidden layers should the network contain?

How many nodes should there be in each hidden
layer?

What condition will terminate network training?

All difficult questions with no straight
-

Neural Network Strengths

Work well with noisy data.

Can process numeric and categorical data.

Appropriate for applications requiring a
time element.

Have performed well in several domains.

Appropriate for supervised learning and
unsupervised clustering.

Neural Network Weaknesses

Lack explanation capabilities

May not provide optimal solutions to
problems

Too much experimentation might be time
consuming

Overtraining can be a problem

Building Neural Networks with
iDA

A Four
-
Step Approach

Backpropagation Learning

(supervised)

Unsupervised clustering

1.
Prepare the data to be mined.

2.
Define the network architecture.

3.
Watch the network train.

4.

Example 1: Modeling the
Exclusive
-
OR Function

Figure 9.1A graph of the XOR
function

XOR: Not Linearly Separable

Figure 9.2 XOR training data

Step 1: Prepare The Data To Be
Mined

Figure 9.3 Dialog box for
supervised learning

Step 2: Define The Network
Architecture

Figure 9.4 Training options for
backpropagation learning

Number of
nodes in
hidden layers

Algorithm
parameter
settings

Epochs:
number of
passes

Figure 9.5 Neural network
execution window

Step 3: Watch The Network Train

Figure 9.6 XOR output file for
Experiment 1

Summary Results

Example 2: Cardiology Numerical
Dataset

CardiologyNumericalNN.xls

A Four
-
Step Approach for
Neural Network Clustering

1.
Prepare the data to be mined.

The Deer Hunter Dataset

2.
Define the network architecture.

3.
Watch the network train.

4.

Figure 9.12 Learning parameters
for unsupervised clustering

Step 2: Define The Network
Architecture

Figure 9.13 Network execution
window

Step 3: Watch The Network Train

Figure 9.14 Deer hunter data:
Unsupervised summary statistics