Map comparison based model validation
How Well Do You Know Your
Model?
A Methodology for Map Comparison
-
Based
Model Validation
Map comparison based model validation
2
Why map comparison?
Good modelling practice
ƒ
Verification
Global behaviour analysis
Sensitivity analysis
Calibration
Validation
Map comparison based model validation
3
Criteria of model performance
Landscape structure
ƒ
Edge, Patch, Fractal, Diversity
Presence
Overlap / Near overlap
Multiple scale
Fuzzy Kappa
Patch size
Diversity
Euclidean
Edge
Map comparison based model validation
4
The meaning of a comparison
Typically two outputs
ƒ
Layer (map) of similarity / difference
Summary statistic
A single summary statistic is meaningless,
multiple comparisons are necessary to:
ƒ
Get understanding of distributions
Understand multi
-
facetted nature of GoF
Attach absolute meaning
-
> Judgement
Map comparison based model validation
A sensitivity analysis
Exploring parameter space
Map comparison based model validation
6
Model: Constrained Cellular Automata
Urban & Non
-
Urban
Š
Total land claim
exogenous
Š
Allocate urban to most
attractive location
ƒ
Neighbourhood
Random factor
Five parameters, 5
possible values
Brute force calibration:
3125 runs
0
0.5
1
1.5
2
2.5
3
1
2
3
4
5
6
7
8
Distance (km)
Impact
Impact Urban on Urban
Impact Non-urban on Urban
White, Engelen, Uljee 1992
Map comparison based model validation
7
Model: A sample of results for Portugal
Porto
Lisbon
Algarve
Map comparison based model validation
8
Sensitivity
Plotting parameters to GoF is not workable
ƒ
5 parameters + 1 GoF metric = 6 dimensions
Plotting GoF to GoF
Goodness-of-fit correspondance
-120
-100
-80
-60
-40
-20
0
20
40
60
80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa 2
Fuzzy Kappa 8
Goodness-of-fit correspondance
-50
-40
-30
-20
-10
0
10
20
30
40
-20
0
20
40
60
80
100
120
140
Kappa
Aggregation
Clusters appear!
Understanding clusters = understanding parameter space ?
Map comparison based model validation
9
The most pronounced clusters
Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
Map comparison based model validation
10
Separated clusters
Cluster
Parameter 1
Parameter 2
Map comparison based model validation
11
Sensitivity / Bifurcation
Same approach can be used to identify
bifurcations as a consequence of stochasticity
Monte Carlo simulation
0.745
0.75
0.755
0.76
0.765
0.77
0.775
0
1
2
3
4
5
6
7
8
9
Moving window, patch size
Percentage
agreement
Porto: 1 Large cluster
Porto: 2 Mid size clusters
Map comparison based model validation
Calibration
Using map comparison to select the best
parameters
Map comparison based model validation
13
Calibration requires a single metric
Can be a composite of metrics
Š
Metric drives convergence and solution!
Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
This parameter set?
Or this one?
Map comparison based model validation
14
Best fit: result overview
Low intra
-
metric sensitvity
Š
Strong inter
-
metric sensitivity
Aggregation
Kappa
Factor
Best
Best
2
LU2902
LU1553
4
LU2964
8
LU1096
16
LU1096
Fuzzy Kappa
Euclidean distance
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2902
2
10
LU2349
4
20
LU2902
4
20
LU1862
8
40
LU2902
infinite
10
LU1862
Patch size
Fractal dimension
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2497
2
10
LU1722
4
20
LU2497
4
20
LU1722
infinite
10
LU2497
infinite
10
LU1722
infinite
infinite
LU1573
infinite
infinite
LU2780
Map comparison based model validation
15
Best fit: Results per metric
DATA
2000
DATA
1990
Kappa
Overall
Fractal dimension
Moving window
Euclidean distance
Moving window
fractal dimension
Overall patch size
Moving window
patch size
Aggregated (> 8)
Fuzzy Kappa
Map comparison based model validation
16
Confronting expert judgment
High agreement
ƒ
Aggregation
Euclidean distance
Patch size (moving window)
Fractal dimension (moving window)
Low agreement
ƒ
Patch size (global)
Fractal dimension (global)
Kappa
Fuzzy Kappa
Contradicting earlier empirical
work !
Kuhnert et al 2005, Hagen 2002
Map comparison based model validation
17
Conclusions
Preferred method: Patch size moving window
ƒ
Best result
With least cost of information (aggregation)
Methods must discredit local overestimation
ƒ
Otherwise: edge growth at the cost of
emerging clusters
Methods must be spatial explicit
Otherwise: fringe solution
Map comparison based model validation
Validation
Judging and understanding
Map comparison based model validation
19
Make sense of diverse comparison
results
Step 1. Normalize to make results mutually
comparable
Step 2. Establish a reference level to be able to
pass absolute judgement
Map comparison based model validation
20
Land use change in La R
é
union
Recent urban expansion causes great concern
Š
What will the future bring
Exploration of alternative scenarios
Calibration of a Constraint Cellular Automata
1989
2002
60 km
N
Map comparison based model validation
21
Multi
-
scale, Multi
-
criteria
Comparison: Model 2002
–
Reality 2002
Š
Moving window based
structure
ƒ
Edge
Patch size
Euclidean distance
Patch size - RMSE
0
10
20
30
40
50
60
70
80
90
1
2
4
8
16
32
64
128
Scale
Error
Euclidean
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1
2
4
8
16
32
64
128
Scale
Error
Edge RMSE
0
0.1
0.2
0.3
0.4
0.5
0.6
1
2
4
8
16
32
64
128
Scale
Error
Map comparison based model validation
22
First step: Normalization
Normalize to level of historical change
Error = 75
Error = 50
Normalized Error =
75/50 = 1.5
MODEL
TRUTH
REFERENCE
Map comparison based model validation
23
Multi
-
scale, Multi
-
criteria
Multi-scale, multi-criteria
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
EDGE
EUCLID
PATCH
Map comparison based model validation
24
Accounting for constraints
Is the model performance caused by the intrinsics
of the model or by the constraints that are
exogeneously imposed on it?
Š
Neutral models
include
constraints and
exclude
a
process of interest
Š
Full model performs better than neutral model?
Increased confidence in the process
Source: McGarical 2001
Fractal
Map comparison based model validation
25
Neutral models of landscape
change
Growing clusters
-
Subject to area constraints
-
Minimal area of change
-
Location alongside existing clusters
Random constraint match
-
Subject to area constraints
-
Minimal area of change
-
Random locations
Map comparison based model validation
26
Random Constraint Match
Open:
-
32
City:
-
15
River: +18
Park: +29
Before
After
RCM
RCM /
After
RCM /
Before
% Correct
0.77
0.98
Kappa
0.35
0.94
Kappa Location
0.35
1.00
Kappa Histo
1.00
0.94
Map comparison based model validation
27
Patch size
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Presence (Euclidean)
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Expressing similarity relative to neutral
models
Edge
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Map comparison based model validation
28
Significance
Assess significance by means of Monte Carlo
simulation and application on multiple time
periods and regions
Similarity
= 0.8 +/
-
0.04
Similarity
= 0.7 +/
-
0.01
Model > Reference ?
98 %
MODEL
TRUTH
REFERENCE
Map comparison based model validation
29
Thanks for your attention
Map Comparison Kit
ƒ
Software, papers & documentation:
www.riks.nl/mck
/
Today’s slides and exercises
www.riks.nl/news
Contact
ƒ
Alex Hagen
-
Zanker
ahagen@riks.nl
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο