Neural Networks -I

[Artificial] Neural Networks

•Computation based on Biological Neural Net

–humans can generalize from experience

–Can ANNs do the same

•A class of powerful, general-purpose tools

–Prediction

–Classification

–Clustering

•Computerized Neural Nets attempt to bridge the gap

–Predicting time-series in financial world

–Diagnosing medical conditions

–Identifying clusters of valuable customers

–Fraud detection

–etc…

Neural Networks

•‘When applied in well-defined domains, their ability to

generalize and learn from data “mimics”a human’s

ability to learn from experience.’

(!!!)

•Very useful in Data Mining…

better results are the hope

Adequately designed and trained NN can capture varied

patterns

•Drawback –models tend to be difficult to understand

(Is it necessary to ‘understand’?)

History

•Early years 1930-1950

•AI vs. NN

–‘Perceptrons’and the XOR problem

•1980s

–Back-propagation, Kohonennets

–Applications

–Availability of computing power

•More ‘intelligent’NNs?

–“On Intelligence”–Jeff Hawkins

Real Estate Appraiser…

??

“…is not applying some set formula, but balancing her experience

and knowledge of sale prices of similar houses…her knowledge about

housing prices is not static...fine tuning her calculation to fit the latest data”

Loan Prospector –HNC/Fair Isaac

A Neural Network is like a black box that knows how to

process inputs to create a useful output. The calculation is

quite complex and difficult to understand, yet the results are

often useful

Is it a black box ?

Garbage in ?

Garbage in ?

Garbage out ?

Garbage out ?

Neural Net Limitations

•Neural Nets are good for prediction and

estimation when:

–Inputs are well understood

–Output is well understood

–There are adequate examples to “train”the neural net

•Neural Nets are only as good as the training set

used to generate it. The resulting model is static

and must be updated with more recent

examples and retraining for it to stay relevant

Feed-Forward Neural Net Examples

•One-way flow through the network from inputs to outputs

Feed-Forward Neural Net Examples

Biological neuron

Artificial Neuron

The loan appraiser

Model defined by

interconnection weights

NN can have multiple output neurons

Neural Network Training

•Training -process of setting the best weights on the

edges connecting all the units in the network

•Use the training set to calculate weights such that NN

output is as close as possible to the desired output for as

many of the examples in the training set as possible

•Back propagationhas been used since the 1980s to

adjust the weights (other methods are now available):

–Calculates the error by taking the difference between the

calculated result and the actual result

–The error is fed back through the network and the weights are

adjusted to minimize the error

How does it work ?

∑

=

i

ii

xwNet

)(Netfy

=

w1

w2

w3

w4

x1

x2

x3

x4

Transfer function

Hard limiter

Linear threshold

Squashing function

(Sigmoid)

Net

e

y

−

+

=

1

1

Common transfer functions

Single ‘layer’NN

W: weight matrix

w11

w12

…w1n

w21

w22

…w2n

……

wm1

wm2

…wmn

x: input vector

Net = x.W (dot product)

Assume no transfer function

Let only one neuron with

highest output ‘fire’

yk

= w1kx1 + w2kx2 +…+w

mkxm

= x.w

k

w11

x2

xm

y1

y2

yn

x1

wmn

Inputs

m

Weights

m xn

Outputs

n

w1

w2

w3

w4

w5

x

wk

x

Net = |wk||x| (cos A)

NN classifies inputs into one of 5 classes.

How do we get the weights? –NN Training

Multi-layer NN

x2

xm

y1

y2

yp

x1

Inputs

m

Outputs

p

Weights

W1

Weights

W2

Consider no transfer function

Then Y = (x W1) W2

= x (W1W2)

= x ( W)

ie. equivalent to single layer

Advantage arises from transfer

function (non-linearity)

Perceptrons and XOR

Rosenblatt (1962): Perceptron can learn

anything that it can represent

Representation –ability of a NN to simulate

a particular function

If Net >= threshold, output 1, else output 0

x1

x2

(0,0)

(0,1)

(1,1)

(1,0)

∑

=

i

ii

xwNet

w1

w2

w3

x1

wm

xm

t

Can we linearly separate the red from yellow?

Linear separability limitation of single layer

perceptron

Overcoming linear seperability

x1

x2

w11

w21

w21

w22

0.5

0.5

t=0.75

t

In layer-2, output is 1 only when

both s1=1 and s2=1 (AND)

s1

s2

In layer 1, each neuron implements

a linear separator

Output is 1

only is this

region

Training a NN

•Adjust weights such that the application of inputs produce desired

outputs (as close as possible)

•Input data is continuously applied, actual outputs calculated, and

weights are adjusted

•Weights should converge to some value after many rounds of training

•Supervised training

–Adjust weights such that differences between desired and actual outputs

are minimized

–Desired output: dependent variable in training data

–Each training example specifies

{independent variables, dependent variable}

•Unsupervised training

–No dependent variable specified in training data

–Train the NN such that similar input data should generate same output

## Comments 0

Log in to post a comment