How Well Do You Know Your

Map comparison based model validation

How Well Do You Know Your
Model?

A Methodology for Map Comparison
-
Based
Model Validation

Map comparison based model validation

2

Why map comparison?


Good modelling practice

ƒ
Verification


Global behaviour analysis


Sensitivity analysis


Calibration


Validation

Map comparison based model validation

3

Criteria of model performance


Landscape structure

ƒ
Edge, Patch, Fractal, Diversity


Presence


Overlap / Near overlap


Multiple scale

Fuzzy Kappa

Patch size

Diversity

Euclidean

Edge

Map comparison based model validation

4

The meaning of a comparison


Typically two outputs

ƒ
Layer (map) of similarity / difference


Summary statistic


A single summary statistic is meaningless,
multiple comparisons are necessary to:

ƒ
Get understanding of distributions


Understand multi
-
facetted nature of GoF


Attach absolute meaning
-
> Judgement

Map comparison based model validation

A sensitivity analysis

Exploring parameter space

Map comparison based model validation

6

Model: Constrained Cellular Automata


Urban & Non
-
Urban

Š
Total land claim
exogenous

Š
Allocate urban to most
attractive location

ƒ
Neighbourhood


Random factor


Five parameters, 5
possible values


Brute force calibration:
3125 runs



0
0.5
1
1.5
2
2.5
3
1
2
3
4
5
6
7
8
Distance (km)
Impact
Impact Urban on Urban
Impact Non-urban on Urban
White, Engelen, Uljee 1992

Map comparison based model validation

7

Model: A sample of results for Portugal

Porto

Lisbon

Algarve

Map comparison based model validation

8

Sensitivity


Plotting parameters to GoF is not workable

ƒ
5 parameters + 1 GoF metric = 6 dimensions


Plotting GoF to GoF

Goodness-of-fit correspondance
-120
-100
-80
-60
-40
-20
0
20
40
60
80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa 2
Fuzzy Kappa 8
Goodness-of-fit correspondance
-50
-40
-30
-20
-10
0
10
20
30
40
-20
0
20
40
60
80
100
120
140
Kappa
Aggregation
Clusters appear!

Understanding clusters = understanding parameter space ?

Map comparison based model validation

9

The most pronounced clusters

Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
Map comparison based model validation

10

Separated clusters

Cluster

Parameter 1

Parameter 2

Map comparison based model validation

11

Sensitivity / Bifurcation


Same approach can be used to identify
bifurcations as a consequence of stochasticity


Monte Carlo simulation
0.745
0.75
0.755
0.76
0.765
0.77
0.775
0
1
2
3
4
5
6
7
8
9
Moving window, patch size
Percentage
agreement
Porto: 1 Large cluster

Porto: 2 Mid size clusters

Map comparison based model validation

Calibration

Using map comparison to select the best
parameters

Map comparison based model validation

13

Calibration requires a single metric


Can be a composite of metrics

Š
Metric drives convergence and solution!

Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
This parameter set?

Or this one?

Map comparison based model validation

14

Best fit: result overview


Low intra
-
metric sensitvity

Š
Strong inter
-
metric sensitivity


Aggregation
Kappa
Factor
Best
Best
2
LU2902
LU1553
4
LU2964
8
LU1096
16
LU1096
Fuzzy Kappa
Euclidean distance
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2902
2
10
LU2349
4
20
LU2902
4
20
LU1862
8
40
LU2902
infinite
10
LU1862
Patch size
Fractal dimension
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2497
2
10
LU1722
4
20
LU2497
4
20
LU1722
infinite
10
LU2497
infinite
10
LU1722
infinite
infinite
LU1573
infinite
infinite
LU2780
Map comparison based model validation

15

Best fit: Results per metric

DATA

2000

DATA

1990

Kappa

Overall

Fractal dimension

Moving window

Euclidean distance

Moving window

fractal dimension

Overall patch size

Moving window


patch size

Aggregated (> 8)

Fuzzy Kappa

Map comparison based model validation

16

Confronting expert judgment


High agreement

ƒ
Aggregation


Euclidean distance


Patch size (moving window)


Fractal dimension (moving window)



Low agreement

ƒ
Patch size (global)


Fractal dimension (global)


Kappa


Fuzzy Kappa

Contradicting earlier empirical
work !

Kuhnert et al 2005, Hagen 2002

Map comparison based model validation

17

Conclusions


Preferred method: Patch size moving window

ƒ
Best result


With least cost of information (aggregation)



Methods must discredit local overestimation

ƒ
Otherwise: edge growth at the cost of
emerging clusters



Methods must be spatial explicit


Otherwise: fringe solution


Map comparison based model validation

Validation

Judging and understanding

Map comparison based model validation

19

Make sense of diverse comparison
results


Step 1. Normalize to make results mutually
comparable


Step 2. Establish a reference level to be able to
pass absolute judgement

Map comparison based model validation

20

Land use change in La R
é
union


Recent urban expansion causes great concern

Š
What will the future bring


Exploration of alternative scenarios


Calibration of a Constraint Cellular Automata


1989

2002

60 km

N

Map comparison based model validation

21

Multi
-
scale, Multi
-
criteria


Comparison: Model 2002
–

Reality 2002

Š
Moving window based
structure

ƒ
Edge


Patch size


Euclidean distance



Patch size - RMSE
0
10
20
30
40
50
60
70
80
90
1
2
4
8
16
32
64
128
Scale
Error
Euclidean
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1
2
4
8
16
32
64
128
Scale
Error
Edge RMSE
0
0.1
0.2
0.3
0.4
0.5
0.6
1
2
4
8
16
32
64
128
Scale
Error
Map comparison based model validation

22

First step: Normalization


Normalize to level of historical change




Error = 75


Error = 50


Normalized Error =
75/50 = 1.5

MODEL

TRUTH

REFERENCE

Map comparison based model validation

23

Multi
-
scale, Multi
-
criteria

Multi-scale, multi-criteria
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
EDGE
EUCLID
PATCH
Map comparison based model validation

24

Accounting for constraints


Is the model performance caused by the intrinsics
of the model or by the constraints that are
exogeneously imposed on it?

Š
Neutral models
include

constraints and
exclude

a
process of interest

Š
Full model performs better than neutral model?


Increased confidence in the process


Source: McGarical 2001

Fractal

Map comparison based model validation

25

Neutral models of landscape
change

Growing clusters

-
Subject to area constraints

-
Minimal area of change

-
Location alongside existing clusters

Random constraint match

-
Subject to area constraints

-
Minimal area of change

-
Random locations

Map comparison based model validation

26

Random Constraint Match

Open:
-
32

City:
-
15

River: +18

Park: +29

Before

After

RCM

RCM /
After

RCM /
Before

% Correct

0.77

0.98

Kappa

0.35

0.94

Kappa Location

0.35

1.00

Kappa Histo

1.00

0.94

Map comparison based model validation

27

Patch size
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Presence (Euclidean)
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Expressing similarity relative to neutral
models

Edge
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Map comparison based model validation

28

Significance


Assess significance by means of Monte Carlo
simulation and application on multiple time
periods and regions

Similarity


= 0.8 +/
-

0.04

Similarity



= 0.7 +/
-

0.01


Model > Reference ?


98 %

MODEL

TRUTH

REFERENCE

Map comparison based model validation

29

Thanks for your attention


Map Comparison Kit

ƒ
Software, papers & documentation:
www.riks.nl/mck
/


Today’s slides and exercises
www.riks.nl/news



Contact

ƒ
Alex Hagen
-
Zanker


ahagen@riks.nl