# Neural Networks and Classical Linear

AI and Robotics

Oct 19, 2013 (4 years and 8 months ago)

325 views

Neural Networks and Classical Linear
Regression

Szu Hua Huang Jianjun Luo

Texas Tech University

10/19/2013

Contents

A.
Neural Networks and Classical Linear
Regression

1.
Introduction Neural Networks

2.
Neural Networks VS Classical Linear Regression

B.
Case study

Neural Network
-

Neuron

STRUCTURE
OF A NEURAL NETWORK

Multilayer
feedforward

networks

A simple MLP (multilayer perceptron)

1

2

3

4

5

6

Input Layer

Hidden Layer

Output Layer

W
13

W
14

W
15

W
23

W
24

W
25

W
36

W
46

W
56

Updating (Learning)

Node

Weights(Regression Coefficients)

Input

Output

Data from True Function

Transfer Function, g(s)

Back
-
propagation Model

Errors

Transfer Function

Neural Network VS Classical Regression Model

Neural Network

Non
-
linear

Classical Regression

Linear

Normality

Constant variability of error
terms

Independence Assumption

Neural Network VS Classical Regression Model

Neural Network

Weight estimates(regression
coefficient) do not tell you
the effect

No guarantee the best linear
combination of parameter
estimates

Classical Regression

Regression coefficient
shows the effect

Neural Network VS Classical Regression Model

Multi
-
collinearity

No effect to
Neural
Network ?

It does hurt the
Classical
Regression

Outlier or Influential

No effect to
Neural
Network ?

It does hurt the
Classical
Regression

Overfitting

Neural Network

Sensitive to the given data, too
much flexibility to the
underlying distribution of
data.

Big Sample size can help to
solve the problem of
overfitting.

Classical Regression

Not very sensitive to the
given data.

Assessment

Neural Network

Optimization plot based on
the updated weight
estimates at each iteration of
the iterative grip search
routine.

Using valid data

Classical Regression

R square

0

5

15

20

validation

training

ASE

Iteration

10

Optimization plot

1. When the new
weights are only
incrementally different
from those of the
preceding iteration

2. When the
misclassification rate
reaches a required
threshold

3. When the limit on
the number of runs is
reached

Supplement
-
Other Optimization Algorithms:

Newton

Quasi
-
Newton method

Levenberg
-
Marquardt

Gauss
-
Newton Method

etc

Stanford Open Course

Machine Learning

Dataset

The School Children Data Set from Lewis & Taylor
“Introduction to Experimental Ecology” (1967)

Includes 126 male records

Variables:

1)

Age

(months)

2)

height

(inches)

3)

weight

(pounds)

Purpose

Predicting the weight of male school children based
on their age and height.

Comparing neural networks with OLS

B: Case Study

Exploration of the dataset

Classical Linear Regression Model

proc

reg

data=men;

model weight=height age;

output out=regout p=pred r=resid;

run
;

Output of OLS

18

INPUT

HIDDEN

OUTPUT

COMBINATION

w
7
+
w
8
A+

w
9
B=Weight

COMBINATION

w
1
+

w
2
S_Height+

w
3
S_Age = H11

TRANSFORMATION

tanh
(H11)) =A

COMBINATION

w
4
+

w
5
S_Height+

w
6
S_Age = H12

TRANSFORMATION

tanh
(H12) =B

H11

H12

Weight

Height

S_Height

Age

S_Age

Standardization

Standardization

Neural Network Model

Neural Network in SAS

The SAS neural network procedure

PROC NEURAL

SAS Enterprise Miner

A visual programming with a GUI interface

Neural Network Modeling using SAS Enterprise Miner

To save time, I recorded the following video to show
how to build the Neural Network Model with SAS
Enterprise Miner.

In case you are interested
, I uploaded this video to

Neural Network Weight Estimates

Variables

Variable Definition

Weights

Weight
Estimate

age_H11

AGE : Input Layer Weights for 1st hidden unit

ŵ
3

1.731949

height_H11

HEIGHT: Input Layer Weights for 1 st hidden unit

ŵ
2

1.394462

age_H12

AGE : Input Layer Weights for 2nd hidden unit

ŵ
6

-
0.225332

height_H12

HEIGHT: Input Layer Weights for 2ndhidden unit

ŵ
5

1.164043

BIAS_H11

Input
-
to
-
Hidden Layer Bias for 1st hidden unit

ŵ
1

-
5.645913

BIAS_H12

Input
-
to
-
Hidden Layer Bias for 2nd hidden unit

ŵ
4

0.548446

H11_weight

Hidden
-
to
-
Target Layer Weight for 1st hidden unit

ŵ
8

29.703755

H12_weight

Hidden
-
to
-
Target Layer Weight for 2nd hidden unit

ŵ
9

21.240443

BIAS_weight

Hidden
-
to
-
Target Layer Bias

ŵ
7

125.950303

Neural network or classical linear regression?

Comparing Neural Network and Classical
Linear Regression predicted values

Output:
Observed and Predicted Values of Male's Weight against Age

References

1.
Eric Roberts. Neural networks. Available online at: http://www
-
cs
-
faculty.stanford.edu/~eroberts/courses/soco/projects/neural
-
networks/

2.
Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS
Institute Inc.

3.
Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc.

4.
Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner.
AuthorHouse

5.
SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at:
http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm

6.
Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes.
SAS Institute Inc.

7.
Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network

Thank You!