Neural Networks and Classical Linear

Neural Networks and Classical Linear
Regression

Szu Hua Huang Jianjun Luo


Texas Tech University

10/19/2013

Contents

A.
Neural Networks and Classical Linear
Regression

1.
Introduction Neural Networks



2.
Neural Networks VS Classical Linear Regression

B.
Case study





Neural Network
-

Neuron

STRUCTURE
OF A NEURAL NETWORK

Multilayer
feedforward

networks

A simple MLP (multilayer perceptron)

1

2

3

4

5

6

Input Layer

Hidden Layer

Output Layer

W
13

W
14

W
15

W
23

W
24

W
25

W
36

W
46

W
56

Updating (Learning)

Node

Weights(Regression Coefficients)

Input

Output

Data from True Function

Transfer Function, g(s)

Back
-
propagation Model

Errors

Transfer Function

Neural Network VS Classical Regression Model

Neural Network


•
Non
-
linear


Classical Regression

•
Linear

•
Normality

•
Constant variability of error
terms

•
Independence Assumption


Neural Network VS Classical Regression Model

Neural Network

•
Weight estimates(regression
coefficient) do not tell you
the effect

•
No guarantee the best linear
combination of parameter
estimates

Classical Regression

•
Regression coefficient
shows the effect

Neural Network VS Classical Regression Model

Multi
-
collinearity


•
No effect to
Neural
Network ?

•

It does hurt the
Classical
Regression


Outlier or Influential

•
No effect to
Neural
Network ?

•
It does hurt the
Classical
Regression



Overfitting

Neural Network


•
Sensitive to the given data, too
much flexibility to the
underlying distribution of
data.

•
Big Sample size can help to
solve the problem of
overfitting.



Classical Regression

•
Not very sensitive to the
given data.

Assessment

Neural Network



•
Optimization plot based on
the updated weight
estimates at each iteration of
the iterative grip search
routine.

•
Using valid data

Classical Regression

•
R square

0

5

15

20

validation

training

ASE

Iteration

10

Optimization plot

•
1. When the new
weights are only
incrementally different
from those of the
preceding iteration

•
2. When the
misclassification rate
reaches a required
threshold

•
3. When the limit on
the number of runs is
reached

Supplement
-
Other Optimization Algorithms:


•
Newton

•
Quasi
-
Newton method

•
Levenberg
-
Marquardt

•
Gauss
-
Newton Method

•
etc

Stanford Open Course
–

Machine Learning

•
Dataset

•
The School Children Data Set from Lewis & Taylor
“Introduction to Experimental Ecology” (1967)

•
Includes 126 male records

•
Variables:

1)

Age

(months)

2)

height

(inches)

3)

weight

(pounds)

•
Purpose

•
Predicting the weight of male school children based
on their age and height.

•
Comparing neural networks with OLS

B: Case Study

Exploration of the dataset

Classical Linear Regression Model

proc

reg

data=men;


model weight=height age;


output out=regout p=pred r=resid;

run
;

Output of OLS

18

INPUT


HIDDEN

OUTPUT

COMBINATION


w
7
+
w
8
A+

w
9
B=Weight


COMBINATION

w
1
+

w
2
S_Height+

w
3
S_Age = H11

TRANSFORMATION


tanh
(H11)) =A

COMBINATION


w
4
+

w
5
S_Height+

w
6
S_Age = H12

TRANSFORMATION


tanh
(H12) =B


H11

H12

Weight

Height

S_Height

Age

S_Age

Standardization

Standardization

Neural Network Model

Neural Network in SAS

•
The SAS neural network procedure

–
PROC NEURAL

•
SAS Enterprise Miner

–
A visual programming with a GUI interface

Neural Network Modeling using SAS Enterprise Miner

•
To save time, I recorded the following video to show
how to build the Neural Network Model with SAS
Enterprise Miner.


•
In case you are interested
, I uploaded this video to
YouTube:


http://www.youtube.com/watch?v=Bb3K7xAcJbk&feature=youtu.be

Neural Network Weight Estimates

Variables

Variable Definition

Weights

Weight
Estimate

age_H11

AGE : Input Layer Weights for 1st hidden unit

ŵ
3

1.731949

height_H11

HEIGHT: Input Layer Weights for 1 st hidden unit

ŵ
2

1.394462

age_H12

AGE : Input Layer Weights for 2nd hidden unit

ŵ
6

-
0.225332

height_H12

HEIGHT: Input Layer Weights for 2ndhidden unit

ŵ
5

1.164043

BIAS_H11

Input
-
to
-
Hidden Layer Bias for 1st hidden unit

ŵ
1

-
5.645913

BIAS_H12

Input
-
to
-
Hidden Layer Bias for 2nd hidden unit

ŵ
4

0.548446

H11_weight

Hidden
-
to
-
Target Layer Weight for 1st hidden unit

ŵ
8

29.703755

H12_weight

Hidden
-
to
-
Target Layer Weight for 2nd hidden unit

ŵ
9

21.240443

BIAS_weight

Hidden
-
to
-
Target Layer Bias

ŵ
7

125.950303

Neural network or classical linear regression?


Comparing Neural Network and Classical
Linear Regression predicted values

Output:
Observed and Predicted Values of Male's Weight against Age

References

1.
Eric Roberts. Neural networks. Available online at: http://www
-
cs
-
faculty.stanford.edu/~eroberts/courses/soco/projects/neural
-
networks/

2.
Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS
Institute Inc.

3.
Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc.

4.
Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner.
AuthorHouse

5.
SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at:
http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm

6.
Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes.
SAS Institute Inc.

7.
Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network

Thank You!