Science Plans: MME - Climate Prediction Center - NOAA

learnedmooseupvalleyΗλεκτρονική - Συσκευές

7 Νοε 2013 (πριν από 4 χρόνια και 1 μήνα)

95 εμφανίσεις

CTB Science Plan

For


Multi Model Ensembles (MME)



Suru Saha

Environmental Modeling Centre

NCEP/NWS/NOAA



Vision


A multi model ensemble forecast system that
leverages the best national and international models
for improved predictions on intraseasonal, interannual
and decadal time scales

Background


Studies have shown that the forecast skill of a multi
-
model system is higher than that of most of the
individual models


To maximize the amount of model information for
NOAA’s seasonal forecasts, a multi
-
model ensemble
strategy will be pursued by the CTB

Current Status of IMME


There has been progress on a possible International
MME (IMME) with the CFS and the EUROSIP
models (ECMWF, UKMO and MeteoFrance)


Preliminary discussions have taken place between
NCEP and ECMWF on protocols for the transfer of
the hindcast data, as well as real time forecasts.


Each center will make available new hindcasts, if and
when, they upgrade their seasonal forecast system.

Current Status of NMME


There has been little progress on a possible National
MME (NMME) with the CFS and the NCAR, NASA
and GFDL models


Results from a previous version of the GFDL model
and data assimilation system did not show significant
additional skill to the CFS, for the basic metrics of US
surface temperature and precipitation
.

Current Status of NMME (contd)


GFDL has made some new hindcasts.
Unfortunately, the CTB has no resources to
evaluate these hindcasts.


Regarding the NCAR model, Ben Kirtman has
been funded to make the hindcasts using the
same strategy as the CFSRR


Regarding the NASA model, there has been no
update on the completion of their hindcasts.

Gaps


There are three major gaps in executing a NMME
strategy


Firstly, skill levels for all models are barely above
climatological probabilities for US surface temps and
precipitation, at monthly and seasonal time scales.


Unless there is a significant improvement in these
metrics, the value of a NMME may not be sufficient
to justify the major cost of making hindcasts to
support an operational NMME.

Gaps (contd)


Secondly, the computational and human costs of
preparing hindcasts for evaluation of skill and
calibration is significant, and not presently funded by
NOAA for models other than the CFS.


Thirdly, the cost and logistics of running a NMME for
all models,
in operations
, are cosiderable and has not
yet been supported by NOAA.

Future Strategy of the NMME (Option 1)



With less than a 100% investment, including modest
computer resources and a staff of 1 full time and 2
-
3
part time personnel, the CTB could support the current
strategy of:


All participating centers make hindcasts of 2 initial
months (for the mid
-
May and mid
-
November release of
CPC’s official monthly and seasonal forecast) over the
period 1979
-
present (or more years, if possible).

Future Strategy of the NMME (Option 1)


A dataset of prescribed output variables from these hindcasts
(full fields) will be made available to the CTB consolidation
team. They will make an objective evaluation of these monthly
and seasonal hindcasts, to determine if skill is added to the CFS
for the basic metrics of US T & P.


Systematic error correction will be done for each tool separately,
and the regression coefficients will be determined under a three
-
years
-
out Cross
-
Validation approach.


If the weight assigned to a model is near zero, then the
consolidation will be redone without that model and the
originators of the model will be notified.

Future Strategy of the NMME (Option 1) contd


If there is additional skill to the CFS, the participating center
will work with EMC in porting the frozen model and data
assimilation systems to NCEP. Random hindcasts will be made
using this system to ensure that there is reproducibility of the
results on the NCEP operational/development computers.


This is important for the application of calibration and weights
for consolidation (which are derived from the retrospective
forecasts) to real time forecasts which will be made on the
NCEP computer.



This process will involve computer and human resources, which
will need to be funded.

Future Strategy of the NMME (Option 1) contd


If the results are reproducible, then the participating
center will complete the rest of the 10 calendar months
with the exact same frozen system.


They will make the same datasets available to the CTB
consolidation team, for further evaluation to determine
if skill is being added to the CFS for these months.


If there is additional skill to the CFS, then EMC will
work with NCO (computer operations) to implement
the NMME into operations at NCEP.

Future Strategy of the NMME (Option 2)


With 100% investment, including a dedicated NOAA
mainframe computer, with a huge number of nodes, disk space,
tapes, and a support staff on the order of 20 to 25 full time
personnel, CTB could support porting all participating systems
(including models and assimilation systems) to the CTB
computer.


CTB would then proceed with making the hindcasts as
described above. This could be done with individual models or
with combinations of model physics and numerics within the
Earth System Modeling Framework (ESMF) to form a true
NMME.


Future Strategy of the NMME (Option 2) contd

The seven year cycle


Start CFS Reanalysis at same time as porting all the other
models (GFDL, NCAR, NASA) to the NOAA NMME
computer.


Make sure Reanalysis (CFS) initial conditions can be used in
all these models (GFDL, NCAR, NASA), or make
Reanalysis for each model system.


This implies that improvements that have been made to these
models and data assimilation systems every 7 years will be
utilized in the NMME, in a continuous fashion, hopefully
leading to an increase in skill of the NMME every time.

Future Strategy of the NMME (Option 2) contd

The seven year cycle


Make hindcasts (2 months, mid
-
May and mid
-
November
release) for each model system separately or in an optimal
combination of different physics parameterizations and/or
different dynamic cores


Evaluate to see if NMME has additional skill to the CFS (CTB
Consolidation team)


If additonal skill exists, make hindcasts for remaining 10
calendar months


Redo evaluation for additional skill to the CFS for these
months


If additional skill exists, then transition to NCEP operations.

Why the Need for Long Hindcast Data

for S/I prediction ?

Huug van den Dool (CPC)

and
Suranjana Saha (EMC)

National Centers for Environmental Prediction,

NWS/NOAA/DOC

“Statistical correction of today’s
numerical forecasts using a long set
of reforecasts (hindcasts)
dramatically


improves its forecast skill”


(Hamill, Whitaker and Mullen 2006, BAMS)


(Single model, multi
-
membered ensemble, NWP)

Data Used for this study


DEMETER : 7 European Coupled GCMs


CFS


1981
-
2001 (21 years of hindcasts)


4 initial months


Monthly Data

Focus:

Skill in Tmp2m over US lead 3

February forecasts from November initial conditions

Explained Variance (%) for US monthly T2m

Feb 1981
-
2001; lead 3 (Nov starts)

Verification : Climate Division data)

SE

corr

CFS

ECM

PLA

MET
FRA

UKM

INGV

LOD

CER

0
years

2.1

1.2

0.0

0.0

0.0

0.4

0.2

0.0

8

years

4.3

7.1

1.4

1.4

7.5

1.4

0.4

2.2

21

years


11.2

(0.33
corr)

8.0

0.4

0.4

8.6

0.6

0.1

0.5

The 3 best models are operational models

Skill Estimates for 8
-
years are not stable

Best skill obtained for all 21 years

SE

corr

CFS

ECM

PLA

MET
FRA

UKM
O

INGV

LOD
YC

CER
F

MME
(EW)

0
years

2.1

1.2

0.0

0.0

0.0

0.4

0.2

0.0

0.2

8

years

4.3

7.1

1.4

1.4

7.5

1.4

0.4

2.2

3.8

21

years

(all)

11.2


8.0

0.4

0.4

8.6

0.6

0.1

0.5

2.0

Explained Variance (%) for US monthly T2m

Feb 1981
-
2001; lead 3 (Nov starts)

Verification : Climate Division data)

Skill of MME with equal weights for ALL models is
far less than the skill of the 3 best models

Explained Variance (%) for US monthly T2m

Feb 1981
-
2001; lead 3 (Nov starts)

Verification : Climate Division data)

SE

corr

CFS

ECM

PLA

MET
FRA

UKM

ING

LOD

CER

MME
(EW)

MME3

0
years

2.1
%

1.2

0.0

0.0

0.0

0.4

0.2

0.0

0.2

0.9

8

years

4.3

7.1

1.4

1.4

7.5

1.4

0.4

2.2

3.8

8.6

21

years

11.2


8.0

0.4

0.4

8.6

0.6

0.1

0.5

2.0

17.0

(corr=

51%)

Skill of MME with equal weights for 3 best models improves with 21 years


Need more years
to determine the
SE where/when
interannual stand
dev is large

8
-
year

21
-
year

8yr minus 21yr

CFS US TMP2M LEAD 3
FORECAST VERIFYING IN FEB

8
-
year

21
-
year

MME, ALL MODELS, EQUAL WEIGHT


US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8yr minus 21yr

Skill of MME is
very much less
than CFS when all
models are
included

MME, 3 BEST MODELS, EQUAL WEIGHT


US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8
-
year

21
-
year

8yr minus 21yr

8yr minus 21
-
year

Skill of MME for 3
best models is the
most for 21 years

Explained Variance (%) for US monthly Precip

Feb 1981
-
2001; lead 3 (Nov starts)

Verification : Climate Division data)

SE

corr

CFS

ECM

PLA

MET
FRA

UKM

INGV

LOD

CER

0
years

0.7

0.4

1.1

1.1

NA

1.1

0.1

0.9

8

years

5.1

0.6

1.9

1.6

NA

1.8

0.1

1.5

21

years

7.5

1.0

3.0

2.2

NA

3.3


0.1

1.9

Not much Skill in any of the models for US precipitation

Skill in SST Anomaly Prediction for Nino-3.4
[DJF 97/98 to AMJ 04]
5-member CFS reforecasts
50
60
70
80
90
100
1
2
3
4
5
6
Forecast Lead [Month]
Anomaly Correlation [%]
CFS
CMP
CCA
CA
MAR
CON
Skill in SST Anomaly Prediction for Nino-3.4
[DJF 97/98 to AMJ 04]
50
60
70
80
90
100
1
2
3
4
5
6
Forecast Lead [Month]
Anomaly Correlation [%]
CFS
CMP
CCA
CA
MRK
CON
15
-
member CFS reforecasts

Skill in SST Anomaly Prediction for Nino-3.4
[DJF 81/82 to AMJ 04]
5-member CFS reforecasts
50
60
70
80
90
100
1
2
3
4
5
6
Forecast Lead [Month]
Anomaly Correlation [%]
CFS
CMP
CCA
CA
MAR
Skill in SST Anomaly Prediction for Nino-3.4
[DJF 81/82 to AMJ 04]
50
60
70
80
90
100
1
2
3
4
5
6
Forecast Lead [Month]
Anomaly Correlation [%]
CFS
CMP
CCA
CA
MRK
15
-
member CFS reforecasts

“ONLY 1 BIG ENSO EVENT”

ALL ENSO EVENTS

CONCLUSIONS


Without SEC (systematic error correction) there is no skill
by any method (for presumably the best month: Feb)



With SEC (1
st

moment only), there is skill by only a few
models (5 out of 8 are still useless)



MME not good when quality of models varies too much



MME3 works well, when using just three good models

CONCLUSIONS (contd)



CFS improves the most from extensive hindcasts (21
years noticeably better than 8) and has the most skill.
Other models have far less skill with all years included.



Cross validation (CV) is problematic (leave 3 years out
when doing 8 year based SEC?)



Need more years to determine the SEC where/when the
inter annual standard deviation is large