Reformulated Neural Network (ReNN)

muscleblouseAI and Robotics

Oct 19, 2013 (4 years and 21 days ago)

71 views

Reformulated Neural Network (
ReNN
)



A
New Alternative
for Data
-
driven
Modelling

in
Hydrology
and Water
Resources Engineering

Saman Razavi
1
, Bryan Tolson
1
, Donald Burn
1
, and Frank Seglenieks
2





1

Department of Civil and Environmental Engineering,



University of Waterloo, Waterloo, Ontario, Canada




2

Environment Canada, Burlington, Ontario, Canada


Outline of the Presentation

Introduction to Reformulated Neural Network


Application 1


Metamodelling


Application 2


Rainfall
-
Runoff Modelling


A new Measure of Regularization


Summary


2


ReNN

is:


Essentially a single
-
hidden
-
layer
neural network


Defined on a new set of
variables based on the network’s
internal geometry


Main Features:


ReNN

is more efficient in training


ReNN

variables are interpretable


ReNN

is more predictable in
generalization

Multilayer
Perceptron

(Traditional Neural Network)


3

1

Hw
i

.Ho
i

x
1

Hw
i

.Ho
i

Height

Location

Slope

Hw
i

d
i

=
-
Hb
i

/

Iw
i,
1

s
i

=
Hw
i

. Iw
i,
1

ReNN

variables in 1
-
input problems

4

a
sigmoidal

unit

x
2

Hw
1

Ho
1

Height

Location

Angle

Slope

Directional
Slopes

Height


Location
&


Slope


Angle

New Variables:

Directional Slope 1


Directional Slope 2

ReNN

variables in
2
-
input problems

5

Details include:


Generalized geometry & revised neural network
formulation with respect to the new variables


Derive the partial derivatives of the network error function
with respect to the new variables for back
-
propagation
training algorithms



Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

ReNN

in
n
-
input problems is non
-
trivial but for
details see

Razavi

and
Tolson

(2011)

Example Applications …

6

Test Function

SWAT2000 Hydrologic Model

Cannonsville

Reservoir Watershed, NY

Saving

Example Application in
Metamodelling

Neural networks are frequently used to model (emulate) computationally
expensive models (e.g., in optimization, model calibration, real
-
time/
operational settings)


Network Training efficiency is very important.

Averaged over
50
trials

Averaged over 50 trials

7

trained with
Standard Back
-
propagation Alg.

trained with
Standard Back
-
propagation Alg.

Precipitation
Gauge 1(
t
)

Precipitation
Gauge 2(
t
)

Precipitation
Gauge 3(
t
)

Precipitation
Gauge 4(
t
)

Average
Precipitation
(
t
-
1
)

Average
Temperature (
t
)

Runoff (
t
)

Precipitation

Gauge 3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge
2

Precipitation

Gauge
4

Cannonsville

Reservoir Watershed


New York
(area = 1200 km
2
)

Example Application in Rainfall
-
Runoff Modelling

(monthly)

Input
-
output data
are scaled to [
-
1 +1]

8

Sigmoidal

Unit 1

0.62

Sigmoidal

Unit 2

-
0.04

Sigmoidal

Unit 3

0.81

Sigmoidal

Unit 4

-
8.65

Sigmoidal

Unit 5

-
0.77

25.92

-
1.42

8.70

-
6.60

-
17.55

-
0.05

0.25

-
0.02

-
2.65

-
0.68

Heights

Overall
Slopes

Locations

Output Bias

8.56

-
8.30

-
5.72

-
6.70

-
4.16

-
22.14

4.25

0.52

0.55

0.69

0.07

-
0.74

-
0.65

3.43

1.77

2.62

1.53

6.76

-
2.44

1.39

2.61

-
1.31

-
0.80

5.46

1.65

1.30

1.85

2.41

1
.93

-
14.95

-
5.07

Precipitation

Gauge
3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge 2

Precipitation

Gauge 4

Cannonsville

Reservoir Watershed


New York
(area = 1200 km
2
)

PR1(t)

PR2(t)

PR3(t)

PR4(t)

PR
(
t
-
1)

TP(
t
)

Directional Slopes

1
6

-
1
6

1

-
1

0.8

9

Sigmoidal

Unit 1

0.62

Sigmoidal

Unit 2

-
0.04

Sigmoidal

Unit 3

0.81

Sigmoidal

Unit 4

-
8.65

Sigmoidal

Unit 5

-
0.77

25.92

-
1.42

8.70

-
6.60

-
17.55

-
0.05

0.25

-
0.02

-
2.65

-
0.68

Heights

Overall
Slopes

Locations

Output Bias

8.56

-
8.30

-
5.72

-
6.70

-
4.16

-
22.14

4.25

0.52

0.55

0.69

0.07

-
0.74

-
0.65

3.43

1.77

2.62

1.53

6.76

-
2.44

1.39

2.61

-
1.31

-
0.80

5.46

1.65

1.30

1.85

2.41

1
.93

-
14.95

-
5.07

Precipitation

Gauge 3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge
2

Precipitation

Gauge
4

Cannonsville

Reservoir Watershed


New York
(area = 1200 km
2
)

PR1(t)

PR2(t)

PR3(t)

PR4(t)

PR
(
t
-
1)

TP(
t
)

Directional Slopes

8.65

8.56

Output Bias

2.65

10

1
6

-
1
6

1

-
1

reg
new

=

Performance function with regularization:

Conventional measure of regularization:

reg
conventional

=

Performance Function =
*

mse

+

*
reg
new

w

(
1
-

w
)

This measure directly quantifies the smoothness of the network response.


Among two networks with the same accuracy on the training data, the one with
smoother response (more regularized) is expected to have better
generalizability


11

This measure is applicable to both
ReNN

and traditional neural networks


Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

Reformulated Neural Network (
ReNN
) is an equivalent reformulation of
multilayer
perceptron

(MLP) neural networks with the following benefits:



ReNN

is trained faster,



ReNN

has an interpretable Internal Geometry


e.g. useful for


Sensitivity Analysis, and



ReNN

has a direct measure of regularization (smoothness).



ReNN

can turn into traditional neural network.



For full information on
ReNN

formulation and derivations,

please refer to:

12

Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

For full information on
ReNN

formulation and derivations,

please refer to: