Neural Networks and Classical Linear
Regression
Szu Hua Huang Jianjun Luo
Texas Tech University
10/19/2013
Contents
A.
Neural Networks and Classical Linear
Regression
1.
Introduction Neural Networks
2.
Neural Networks VS Classical Linear Regression
B.
Case study
Neural Network
-
Neuron
STRUCTURE
OF A NEURAL NETWORK
Multilayer
feedforward
networks
A simple MLP (multilayer perceptron)
1
2
3
4
5
6
Input Layer
Hidden Layer
Output Layer
W
13
W
14
W
15
W
23
W
24
W
25
W
36
W
46
W
56
Updating (Learning)
Node
Weights(Regression Coefficients)
Input
Output
Data from True Function
Transfer Function, g(s)
Back
-
propagation Model
Errors
Transfer Function
Neural Network VS Classical Regression Model
Neural Network
•
Non
-
linear
Classical Regression
•
Linear
•
Normality
•
Constant variability of error
terms
•
Independence Assumption
Neural Network VS Classical Regression Model
Neural Network
•
Weight estimates(regression
coefficient) do not tell you
the effect
•
No guarantee the best linear
combination of parameter
estimates
Classical Regression
•
Regression coefficient
shows the effect
Neural Network VS Classical Regression Model
Multi
-
collinearity
•
No effect to
Neural
Network ?
•
It does hurt the
Classical
Regression
Outlier or Influential
•
No effect to
Neural
Network ?
•
It does hurt the
Classical
Regression
Overfitting
Neural Network
•
Sensitive to the given data, too
much flexibility to the
underlying distribution of
data.
•
Big Sample size can help to
solve the problem of
overfitting.
Classical Regression
•
Not very sensitive to the
given data.
Assessment
Neural Network
•
Optimization plot based on
the updated weight
estimates at each iteration of
the iterative grip search
routine.
•
Using valid data
Classical Regression
•
R square
0
5
15
20
validation
training
ASE
Iteration
10
Optimization plot
•
1. When the new
weights are only
incrementally different
from those of the
preceding iteration
•
2. When the
misclassification rate
reaches a required
threshold
•
3. When the limit on
the number of runs is
reached
Supplement
-
Other Optimization Algorithms:
•
Newton
•
Quasi
-
Newton method
•
Levenberg
-
Marquardt
•
Gauss
-
Newton Method
•
etc
Stanford Open Course
–
Machine Learning
•
Dataset
•
The School Children Data Set from Lewis & Taylor
“Introduction to Experimental Ecology” (1967)
•
Includes 126 male records
•
Variables:
1)
Age
(months)
2)
height
(inches)
3)
weight
(pounds)
•
Purpose
•
Predicting the weight of male school children based
on their age and height.
•
Comparing neural networks with OLS
B: Case Study
Exploration of the dataset
Classical Linear Regression Model
proc
reg
data=men;
model weight=height age;
output out=regout p=pred r=resid;
run
;
Output of OLS
18
INPUT
HIDDEN
OUTPUT
COMBINATION
w
7
+
w
8
A+
w
9
B=Weight
COMBINATION
w
1
+
w
2
S_Height+
w
3
S_Age = H11
TRANSFORMATION
tanh
(H11)) =A
COMBINATION
w
4
+
w
5
S_Height+
w
6
S_Age = H12
TRANSFORMATION
tanh
(H12) =B
H11
H12
Weight
Height
S_Height
Age
S_Age
Standardization
Standardization
Neural Network Model
Neural Network in SAS
•
The SAS neural network procedure
–
PROC NEURAL
•
SAS Enterprise Miner
–
A visual programming with a GUI interface
Neural Network Modeling using SAS Enterprise Miner
•
To save time, I recorded the following video to show
how to build the Neural Network Model with SAS
Enterprise Miner.
•
In case you are interested
, I uploaded this video to
YouTube:
http://www.youtube.com/watch?v=Bb3K7xAcJbk&feature=youtu.be
Neural Network Weight Estimates
Variables
Variable Definition
Weights
Weight
Estimate
age_H11
AGE : Input Layer Weights for 1st hidden unit
ŵ
3
1.731949
height_H11
HEIGHT: Input Layer Weights for 1 st hidden unit
ŵ
2
1.394462
age_H12
AGE : Input Layer Weights for 2nd hidden unit
ŵ
6
-
0.225332
height_H12
HEIGHT: Input Layer Weights for 2ndhidden unit
ŵ
5
1.164043
BIAS_H11
Input
-
to
-
Hidden Layer Bias for 1st hidden unit
ŵ
1
-
5.645913
BIAS_H12
Input
-
to
-
Hidden Layer Bias for 2nd hidden unit
ŵ
4
0.548446
H11_weight
Hidden
-
to
-
Target Layer Weight for 1st hidden unit
ŵ
8
29.703755
H12_weight
Hidden
-
to
-
Target Layer Weight for 2nd hidden unit
ŵ
9
21.240443
BIAS_weight
Hidden
-
to
-
Target Layer Bias
ŵ
7
125.950303
Neural network or classical linear regression?
Comparing Neural Network and Classical
Linear Regression predicted values
Output:
Observed and Predicted Values of Male's Weight against Age
References
1.
Eric Roberts. Neural networks. Available online at: http://www
-
cs
-
faculty.stanford.edu/~eroberts/courses/soco/projects/neural
-
networks/
2.
Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS
Institute Inc.
3.
Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc.
4.
Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner.
AuthorHouse
5.
SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at:
http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm
6.
Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes.
SAS Institute Inc.
7.
Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network
Thank You!
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment