Neural Networks - I

prudencewooshAI and Robotics

Oct 19, 2013 (3 years and 9 months ago)

65 views

Neural Networks -I
[Artificial] Neural Networks
•Computation based on Biological Neural Net
–humans can generalize from experience
–Can ANNs do the same
•A class of powerful, general-purpose tools
–Prediction
–Classification
–Clustering
•Computerized Neural Nets attempt to bridge the gap
–Predicting time-series in financial world
–Diagnosing medical conditions
–Identifying clusters of valuable customers
–Fraud detection
–etc…
Neural Networks
•‘When applied in well-defined domains, their ability to
generalize and learn from data “mimics”a human’s
ability to learn from experience.’
(!!!)
•Very useful in Data Mining…
better results are the hope
Adequately designed and trained NN can capture varied
patterns
•Drawback –models tend to be difficult to understand
(Is it necessary to ‘understand’?)
History
•Early years 1930-1950
•AI vs. NN
–‘Perceptrons’and the XOR problem
•1980s
–Back-propagation, Kohonennets
–Applications
–Availability of computing power
•More ‘intelligent’NNs?
–“On Intelligence”–Jeff Hawkins
Real Estate Appraiser…
??
“…is not applying some set formula, but balancing her experience
and knowledge of sale prices of similar houses…her knowledge about
housing prices is not static...fine tuning her calculation to fit the latest data”
Loan Prospector –HNC/Fair Isaac
A Neural Network is like a black box that knows how to
process inputs to create a useful output. The calculation is
quite complex and difficult to understand, yet the results are
often useful
Is it a black box ?
Garbage in ?
Garbage in ?
Garbage out ?
Garbage out ?
Neural Net Limitations
•Neural Nets are good for prediction and
estimation when:
–Inputs are well understood
–Output is well understood
–There are adequate examples to “train”the neural net
•Neural Nets are only as good as the training set
used to generate it. The resulting model is static
and must be updated with more recent
examples and retraining for it to stay relevant
Feed-Forward Neural Net Examples
•One-way flow through the network from inputs to outputs
Feed-Forward Neural Net Examples
Biological neuron
Artificial Neuron
The loan appraiser
Model defined by
interconnection weights
NN can have multiple output neurons
Neural Network Training
•Training -process of setting the best weights on the
edges connecting all the units in the network
•Use the training set to calculate weights such that NN
output is as close as possible to the desired output for as
many of the examples in the training set as possible
•Back propagationhas been used since the 1980s to
adjust the weights (other methods are now available):
–Calculates the error by taking the difference between the
calculated result and the actual result
–The error is fed back through the network and the weights are
adjusted to minimize the error
How does it work ?

=
i
ii
xwNet
)(Netfy
=
w1
w2
w3
w4
x1
x2
x3
x4
Transfer function
Hard limiter
Linear threshold
Squashing function
(Sigmoid)
Net
e
y

+
=
1
1
Common transfer functions
Single ‘layer’NN
W: weight matrix
w11
w12
…w1n
w21
w22
…w2n
……
wm1
wm2
…wmn
x: input vector
Net = x.W (dot product)
Assume no transfer function
Let only one neuron with
highest output ‘fire’
yk
= w1kx1 + w2kx2 +…+w
mkxm
= x.w
k
w11
x2
xm
y1
y2
yn
x1
wmn
Inputs
m
Weights
m xn
Outputs
n
w1
w2
w3
w4
w5
x
wk
x
Net = |wk||x| (cos A)
NN classifies inputs into one of 5 classes.
How do we get the weights? –NN Training
Multi-layer NN
x2
xm
y1
y2
yp
x1
Inputs
m
Outputs
p
Weights
W1
Weights
W2
Consider no transfer function
Then Y = (x W1) W2
= x (W1W2)
= x ( W)
ie. equivalent to single layer
Advantage arises from transfer
function (non-linearity)
Perceptrons and XOR
Rosenblatt (1962): Perceptron can learn
anything that it can represent
Representation –ability of a NN to simulate
a particular function
If Net >= threshold, output 1, else output 0
x1
x2
(0,0)
(0,1)
(1,1)
(1,0)

=
i
ii
xwNet
w1
w2
w3
x1
wm
xm
t
Can we linearly separate the red from yellow?
Linear separability limitation of single layer
perceptron
Overcoming linear seperability
x1
x2
w11
w21
w21
w22
0.5
0.5
t=0.75
t
In layer-2, output is 1 only when
both s1=1 and s2=1 (AND)
s1
s2
In layer 1, each neuron implements
a linear separator
Output is 1
only is this
region
Training a NN
•Adjust weights such that the application of inputs produce desired
outputs (as close as possible)
•Input data is continuously applied, actual outputs calculated, and
weights are adjusted
•Weights should converge to some value after many rounds of training
•Supervised training
–Adjust weights such that differences between desired and actual outputs
are minimized
–Desired output: dependent variable in training data
–Each training example specifies
{independent variables, dependent variable}
•Unsupervised training
–No dependent variable specified in training data
–Train the NN such that similar input data should generate same output