Weapon Engineering Assessment Technology C-6

triteritzyBiotechnology

Dec 14, 2012 (4 years and 4 months ago)

177 views

Generation of Pareto Optimal
Ensembles of Calibrated
Parameter Sets for Climate Models

Keith Dalbey, Ph.D.

Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification


Michael Levy, Ph.D.

Sandia National Labs, Dept 1442, Numerical Analysis and Applications

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed
Martin Company, for the United States Department of Energy’s National Nuclear
Security Administration under Contract DE
-
AC04
-
94AL85000.

December 12
-
17, 2010

Outline


Motivation


Approach: Pareto Ensemble


What Does “Pareto Optimal” Mean?


Finding a “Pareto Optimal” Ensemble


Results of Tuning Climate Model


Summary & Future Work


References

Jackson et al, “Error reduction and convergence in climate prediction,”
Journal of Climate
, 2008.

Eddy & Lewis, “Effective generation of pareto sets using genetic
programming,”
Proc. of ASME Design Engineering Technical
Conference
, 2001.

Dalbey & Karystinos, “Fast generation of space
-
filling latin hypercube
sample designs,”
Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis
and Optimization Conference
, 2010.

Motivation

Calibrating (tuning) climate models


choosing values of model parameters to predict well

Is difficult because


They have many inputs and outputs


Diverse parameters sets can match observations similarly well


Errors can compensate: “2 wrongs can make a right”
under
historical conditions


Climate change (new conditions) might expose a previously
hidden
mis
-
calibration, so…

History matching is necessary but not sufficient for
good predictions.


The future is uncertain, but we can quantify the uncertainty
(estimate statistics) for possible future climates.



Approach: Pareto Ensemble

How can we make good
statistical

predictions?


Use a
diverse ensemble
of “good” parameter sets to
determine the range/spread of possible future climates

QUESTION:

What’s the definition of a “good” parameter set?
There are multiple outputs and what’s good for one output can
be bad for another.

(AN) ANSWER:
It’s Pareto optimal. A point (parameter set)
is Pareto optimal if there is no other point that is as good or
better than it in
ALL
outputs.

What does the “Pareto” mean?


It’s just the name of the person who discovered it…


Vilfredo Federico Damaso Pareto was an Italian engineer,
sociologist, economist, and philosopher.


What Does “Pareto Optimal” Mean?

2D Pareto front schematics

What Does “Pareto Optimal” Mean?


Usually, the current approx. of the true Pareto front.


The Pareto front defines the “zero sum game” of all
optimal compromises you could make.


Unlike a weighted combination of objective
functions, it lets you choose a specific compromise/
combination AFTER the optimization is complete.


It does NOT say which compromise/combination is
best, just what all the “optimal” choices are.


It says “Don’t choose anything Pareto non
-
optimal
because there’s something better in all criteria.”

Finding a “Pareto Optimal” Ensemble


Used the Multi Objective Genetic Algorithm (MOGA)
in DAKOTA’s (
Design Analysis Kit for Optimization and
Terascale

Applications
) JEGA (
John Eddy’s Genetic Algorithm)

sub
-
package


GA’s typically need 1000’s of simulations, I could only
afford


1000…


Used test problem (find surface of radius=1 6D hyper
-
sphere in input space, 10 outputs) to tune MOGA
settings and initial population (
space
-
filling, specifically Binning
Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS
), for:


Large Pareto Ensemble


Mean radius close to 1


Uniform spread


Small radius variance

Finding a “Pareto Optimal” Ensemble

Use DAKOTA’s
MOGA on
a test problem with 6 inputs and 10
outputs;
true solution is a radius 1 hypersphere

Default Monte Carlo seed

PDF’s of the

Pareto Ensemble’s

1.
# of points

2.
Point spread

3.
Mean radius

4.
Standard
deviation of
radius

1

2

3

4

Finding a “Pareto Optimal” Ensemble

Use DAKOTA’s Multi Objective Genetic Algorithm
on a test problem with 6 inputs and 10 outputs
true solution is a radius 1 hypersphere

BOSLHS seed

Default Monte Carlo seed

Results of Tuning Climate Model

Summary & Future Work


Climate model parameters that match history well
might not predict well (climate change might expose
a previously hidden
mis
-
calibration of parameters).


Plan:
Use a diverse ensemble of “good” (Pareto
optimal) parameter sets to determine the
range/spread of possible future climates.


Used MOGA to find a (very large) Pareto optimal
ensemble of calibrated parameter sets.


Next steps:


down select Pareto optimal ensemble, and


simulate smaller ensemble out to 2100.

Some “Good” Parameter Sets

Inputs
% change in output mismatch relative to CCSM4 default
RHMINL
RHMINH
ALFA
TAU [hrs]
C0 [m^-1]
KE [(m^2s/kg)^0.5/s]
TREFHT
T
U
PS
RELHUM
LHFLX
LWCF
SWCF
PRECT
RADBAL
0.9348
0.7941
0.5527
4.426
3.445E-3
6.883E-6
10.5869
1.33
6.515
0.64
-9.13728
38.049
-33.32
-20.5
50.546
-45.917
0.9348
0.8789
0.3379
3.020
4.711E-3
7.867E-6
4.73702
0.42
9.581
0.75
23.3127
31.136
-26.88
-31.94
49.582
-65.122
0.9393
0.6727
0.2348
2.316
5.086E-3
4.148E-6
-0.309
3.88
11.42
1.44
-15.2271
6.6722
3.5036
14.204
13.99
-99.376
0.9496
0.7055
0.2365
6.652
5.836E-3
7.211E-6
14.4249
3.39
8.135
0.99
-29.4557
45.088
-31.76
-29.09
49.752
-73.457
0.9277
0.8039
0.3483
3.254
4.383E-3
3.383E-6
1.38947
0.47
1.366
1.22
-10.3006
30.064
-27.13
-16.6
46.626
-65.048
0.9316
0.7992
0.0715
2.151
5.133E-3
8.195E-6
-4.1146
1.59
14.62
1.12
44.5008
-2.331
0.5983
24.616
4.9049
-99.992
0.9348
0.6586
0.2090
2.316
5.273E-3
3.383E-6
-0.2539
3.38
7.114
0.62
-23.2472
5.3236
2.0664
20.214
11.527
-61.24
0.9348
0.8610
0.3379
3.020
4.711E-3
7.867E-6
1.34654
0.67
8.201
1.2
19.6143
24.863
-31.78
-26.22
40.619
-85.433
0.9393
0.6982
0.2348
2.316
5.086E-3
4.148E-6
3.17856
3.13
15.23
1.61
-11.5716
5.8576
-1.338
6.613
14.649
-90.469
0.9418
0.7371
0.2365
6.652
5.836E-3
7.211E-6
13.0888
2.74
8.034
0.39
-26.0691
51.814
-30.86
-25.73
49.191
-84.672
0.9316
0.7795
0.0715
2.151
5.133E-3
8.195E-6
-5.7248
2.6
19.24
0.81
35.4784
-1.642
4.6604
31.289
1.5978
-78.571
0.9324
0.7362
0.0973
0.910
4.570E-3
9.617E-6
-1.7177
2.77
27.21
0.37
27.8201
-15.75
30.85
45.384
-7.412
-99.809
0.9324
0.8320
0.1171
1.848
4.570E-3
8.523E-6
-1.021
2.58
27.97
0.45
58.6333
0.156
-6.863
13.883
5.2161
-92.776
CCSM4 default
0.91
0.8
0.1
1
3.500E-3
1.000E-6
0
0
0
0
0
0
0
0
0
0
Range
0.95
0.9
0.6
8
6.000E-3
1.000E-5
0.8
0.6
0.05
0.5
3.000E-3
3.000E-6