Reformulated Neural Network (ReNN)

AI and Robotics

Oct 19, 2013 (4 years and 8 months ago)

80 views

Reformulated Neural Network (
ReNN
)

A
New Alternative
for Data
-
driven
Modelling

in
Hydrology
and Water
Resources Engineering

Saman Razavi
1
, Bryan Tolson
1
, Donald Burn
1
, and Frank Seglenieks
2

1

Department of Civil and Environmental Engineering,

University of Waterloo, Waterloo, Ontario, Canada

2

Outline of the Presentation

Introduction to Reformulated Neural Network

Application 1

Metamodelling

Application 2

Rainfall
-
Runoff Modelling

A new Measure of Regularization

Summary

2

ReNN

is:

Essentially a single
-
hidden
-
layer
neural network

Defined on a new set of
variables based on the network’s
internal geometry

Main Features:

ReNN

is more efficient in training

ReNN

variables are interpretable

ReNN

is more predictable in
generalization

Multilayer
Perceptron

3

1

Hw
i

.Ho
i

x
1

Hw
i

.Ho
i

Height

Location

Slope

Hw
i

d
i

=
-
Hb
i

/

Iw
i,
1

s
i

=
Hw
i

. Iw
i,
1

ReNN

variables in 1
-
input problems

4

a
sigmoidal

unit

x
2

Hw
1

Ho
1

Height

Location

Angle

Slope

Directional
Slopes

Height

Location
&

Slope

Angle

New Variables:

Directional Slope 1

Directional Slope 2

ReNN

variables in
2
-
input problems

5

Details include:

Generalized geometry & revised neural network
formulation with respect to the new variables

Derive the partial derivatives of the network error function
with respect to the new variables for back
-
propagation
training algorithms

Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

ReNN

in
n
-
input problems is non
-
trivial but for
details see

Razavi

and
Tolson

(2011)

Example Applications …

6

Test Function

SWAT2000 Hydrologic Model

Cannonsville

Reservoir Watershed, NY

Saving

Example Application in
Metamodelling

Neural networks are frequently used to model (emulate) computationally
expensive models (e.g., in optimization, model calibration, real
-
time/
operational settings)

Network Training efficiency is very important.

Averaged over
50
trials

Averaged over 50 trials

7

trained with
Standard Back
-
propagation Alg.

trained with
Standard Back
-
propagation Alg.

Precipitation
Gauge 1(
t
)

Precipitation
Gauge 2(
t
)

Precipitation
Gauge 3(
t
)

Precipitation
Gauge 4(
t
)

Average
Precipitation
(
t
-
1
)

Average
Temperature (
t
)

Runoff (
t
)

Precipitation

Gauge 3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge
2

Precipitation

Gauge
4

Cannonsville

Reservoir Watershed

New York
(area = 1200 km
2
)

Example Application in Rainfall
-
Runoff Modelling

(monthly)

Input
-
output data
are scaled to [
-
1 +1]

8

Sigmoidal

Unit 1

0.62

Sigmoidal

Unit 2

-
0.04

Sigmoidal

Unit 3

0.81

Sigmoidal

Unit 4

-
8.65

Sigmoidal

Unit 5

-
0.77

25.92

-
1.42

8.70

-
6.60

-
17.55

-
0.05

0.25

-
0.02

-
2.65

-
0.68

Heights

Overall
Slopes

Locations

Output Bias

8.56

-
8.30

-
5.72

-
6.70

-
4.16

-
22.14

4.25

0.52

0.55

0.69

0.07

-
0.74

-
0.65

3.43

1.77

2.62

1.53

6.76

-
2.44

1.39

2.61

-
1.31

-
0.80

5.46

1.65

1.30

1.85

2.41

1
.93

-
14.95

-
5.07

Precipitation

Gauge
3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge 2

Precipitation

Gauge 4

Cannonsville

Reservoir Watershed

New York
(area = 1200 km
2
)

PR1(t)

PR2(t)

PR3(t)

PR4(t)

PR
(
t
-
1)

TP(
t
)

Directional Slopes

1
6

-
1
6

1

-
1

0.8

9

Sigmoidal

Unit 1

0.62

Sigmoidal

Unit 2

-
0.04

Sigmoidal

Unit 3

0.81

Sigmoidal

Unit 4

-
8.65

Sigmoidal

Unit 5

-
0.77

25.92

-
1.42

8.70

-
6.60

-
17.55

-
0.05

0.25

-
0.02

-
2.65

-
0.68

Heights

Overall
Slopes

Locations

Output Bias

8.56

-
8.30

-
5.72

-
6.70

-
4.16

-
22.14

4.25

0.52

0.55

0.69

0.07

-
0.74

-
0.65

3.43

1.77

2.62

1.53

6.76

-
2.44

1.39

2.61

-
1.31

-
0.80

5.46

1.65

1.30

1.85

2.41

1
.93

-
14.95

-
5.07

Precipitation

Gauge 3

Walton

Runoff Gauge

Precipitation

Gauge 1

Precipitation

Gauge
2

Precipitation

Gauge
4

Cannonsville

Reservoir Watershed

New York
(area = 1200 km
2
)

PR1(t)

PR2(t)

PR3(t)

PR4(t)

PR
(
t
-
1)

TP(
t
)

Directional Slopes

8.65

8.56

Output Bias

2.65

10

1
6

-
1
6

1

-
1

reg
new

=

Performance function with regularization:

Conventional measure of regularization:

reg
conventional

=

Performance Function =
*

mse

+

*
reg
new

w

(
1
-

w
)

This measure directly quantifies the smoothness of the network response.

Among two networks with the same accuracy on the training data, the one with
smoother response (more regularized) is expected to have better
generalizability

11

This measure is applicable to both
ReNN

and traditional neural networks

Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

Reformulated Neural Network (
ReNN
) is an equivalent reformulation of
multilayer
perceptron

(MLP) neural networks with the following benefits:

ReNN

is trained faster,

ReNN

has an interpretable Internal Geometry

e.g. useful for

Sensitivity Analysis, and

ReNN

has a direct measure of regularization (smoothness).

ReNN

can turn into traditional neural network.

For full information on
ReNN

formulation and derivations,

12

Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward

neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588
-
1598, DOI:
1510.1109/TNN.2011.2163169.

For full information on
ReNN

formulation and derivations,