Reformulated Neural Network (
ReNN
)
A
New Alternative
for Data

driven
Modelling
in
Hydrology
and Water
Resources Engineering
Saman Razavi
1
, Bryan Tolson
1
, Donald Burn
1
, and Frank Seglenieks
2
1
Department of Civil and Environmental Engineering,
University of Waterloo, Waterloo, Ontario, Canada
2
Environment Canada, Burlington, Ontario, Canada
Outline of the Presentation
Introduction to Reformulated Neural Network
Application 1
–
Metamodelling
Application 2
–
Rainfall

Runoff Modelling
A new Measure of Regularization
Summary
2
ReNN
is:
–
Essentially a single

hidden

layer
neural network
–
Defined on a new set of
variables based on the network’s
internal geometry
Main Features:
–
ReNN
is more efficient in training
–
ReNN
variables are interpretable
–
ReNN
is more predictable in
generalization
Multilayer
Perceptron
(Traditional Neural Network)
3
1
Hw
i
.Ho
i
x
1
Hw
i
.Ho
i
Height
Location
Slope
Hw
i
d
i
=

Hb
i
/
Iw
i,
1
s
i
=
Hw
i
. Iw
i,
1
ReNN
variables in 1

input problems
4
a
sigmoidal
unit
x
2
Hw
1
Ho
1
Height
Location
Angle
Slope
Directional
Slopes
Height
Location
&
Slope
Angle
New Variables:
Directional Slope 1
Directional Slope 2
ReNN
variables in
2

input problems
5
Details include:
Generalized geometry & revised neural network
formulation with respect to the new variables
Derive the partial derivatives of the network error function
with respect to the new variables for back

propagation
training algorithms
Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward
neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588

1598, DOI:
1510.1109/TNN.2011.2163169.
ReNN
in
n

input problems is non

trivial but for
details see
Razavi
and
Tolson
(2011)
Example Applications …
6
Test Function
SWAT2000 Hydrologic Model
Cannonsville
Reservoir Watershed, NY
Saving
Example Application in
Metamodelling
Neural networks are frequently used to model (emulate) computationally
expensive models (e.g., in optimization, model calibration, real

time/
operational settings)
Network Training efficiency is very important.
Averaged over
50
trials
Averaged over 50 trials
7
trained with
Standard Back

propagation Alg.
trained with
Standard Back

propagation Alg.
Precipitation
Gauge 1(
t
)
Precipitation
Gauge 2(
t
)
Precipitation
Gauge 3(
t
)
Precipitation
Gauge 4(
t
)
Average
Precipitation
(
t

1
)
Average
Temperature (
t
)
Runoff (
t
)
Precipitation
Gauge 3
Walton
Runoff Gauge
Precipitation
Gauge 1
Precipitation
Gauge
2
Precipitation
Gauge
4
Cannonsville
Reservoir Watershed
New York
(area = 1200 km
2
)
Example Application in Rainfall

Runoff Modelling
(monthly)
Input

output data
are scaled to [

1 +1]
8
Sigmoidal
Unit 1
0.62
Sigmoidal
Unit 2

0.04
Sigmoidal
Unit 3
0.81
Sigmoidal
Unit 4

8.65
Sigmoidal
Unit 5

0.77
25.92

1.42
8.70

6.60

17.55

0.05
0.25

0.02

2.65

0.68
Heights
Overall
Slopes
Locations
Output Bias
8.56

8.30

5.72

6.70

4.16

22.14
4.25
0.52
0.55
0.69
0.07

0.74

0.65
3.43
1.77
2.62
1.53
6.76

2.44
1.39
2.61

1.31

0.80
5.46
1.65
1.30
1.85
2.41
1
.93

14.95

5.07
Precipitation
Gauge
3
Walton
Runoff Gauge
Precipitation
Gauge 1
Precipitation
Gauge 2
Precipitation
Gauge 4
Cannonsville
Reservoir Watershed
New York
(area = 1200 km
2
)
PR1(t)
PR2(t)
PR3(t)
PR4(t)
PR
(
t

1)
TP(
t
)
Directional Slopes
1
6

1
6
1

1
0.8
9
Sigmoidal
Unit 1
0.62
Sigmoidal
Unit 2

0.04
Sigmoidal
Unit 3
0.81
Sigmoidal
Unit 4

8.65
Sigmoidal
Unit 5

0.77
25.92

1.42
8.70

6.60

17.55

0.05
0.25

0.02

2.65

0.68
Heights
Overall
Slopes
Locations
Output Bias
8.56

8.30

5.72

6.70

4.16

22.14
4.25
0.52
0.55
0.69
0.07

0.74

0.65
3.43
1.77
2.62
1.53
6.76

2.44
1.39
2.61

1.31

0.80
5.46
1.65
1.30
1.85
2.41
1
.93

14.95

5.07
Precipitation
Gauge 3
Walton
Runoff Gauge
Precipitation
Gauge 1
Precipitation
Gauge
2
Precipitation
Gauge
4
Cannonsville
Reservoir Watershed
New York
(area = 1200 km
2
)
PR1(t)
PR2(t)
PR3(t)
PR4(t)
PR
(
t

1)
TP(
t
)
Directional Slopes
8.65
8.56
Output Bias
2.65
10
1
6

1
6
1

1
reg
new
=
Performance function with regularization:
Conventional measure of regularization:
reg
conventional
=
Performance Function =
*
mse
+
*
reg
new
w
(
1

w
)
This measure directly quantifies the smoothness of the network response.
Among two networks with the same accuracy on the training data, the one with
smoother response (more regularized) is expected to have better
generalizability
11
This measure is applicable to both
ReNN
and traditional neural networks
Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward
neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588

1598, DOI:
1510.1109/TNN.2011.2163169.
Reformulated Neural Network (
ReNN
) is an equivalent reformulation of
multilayer
perceptron
(MLP) neural networks with the following benefits:
ReNN
is trained faster,
ReNN
has an interpretable Internal Geometry
–
e.g. useful for
Sensitivity Analysis, and
ReNN
has a direct measure of regularization (smoothness).
ReNN
can turn into traditional neural network.
For full information on
ReNN
formulation and derivations,
please refer to:
12
Razavi
, S., and
Tolson
, B. A. (2011). "A new formulation for
feedforward
neural
networks."
IEEE Transactions on Neural Networks
, 22(10), 1588

1598, DOI:
1510.1109/TNN.2011.2163169.
For full information on
ReNN
formulation and derivations,
please refer to:
Comments 0
Log in to post a comment