How Well Do You Know Your

rucksackbulgeΤεχνίτη Νοημοσύνη και Ρομποτική

1 Δεκ 2013 (πριν από 3 χρόνια και 8 μήνες)

59 εμφανίσεις

Map comparison based model validation

How Well Do You Know Your
Model?

A Methodology for Map Comparison
-
Based
Model Validation

Map comparison based model validation

2

Why map comparison?


Good modelling practice

ƒ
Verification


Global behaviour analysis


Sensitivity analysis


Calibration


Validation

Map comparison based model validation

3

Criteria of model performance


Landscape structure

ƒ
Edge, Patch, Fractal, Diversity


Presence


Overlap / Near overlap


Multiple scale

Fuzzy Kappa

Patch size

Diversity

Euclidean

Edge

Map comparison based model validation

4

The meaning of a comparison


Typically two outputs

ƒ
Layer (map) of similarity / difference


Summary statistic


A single summary statistic is meaningless,
multiple comparisons are necessary to:

ƒ
Get understanding of distributions


Understand multi
-
facetted nature of GoF


Attach absolute meaning
-
> Judgement

Map comparison based model validation

A sensitivity analysis

Exploring parameter space

Map comparison based model validation

6

Model: Constrained Cellular Automata


Urban & Non
-
Urban

Š
Total land claim
exogenous

Š
Allocate urban to most
attractive location

ƒ
Neighbourhood


Random factor


Five parameters, 5
possible values


Brute force calibration:
3125 runs



0
0.5
1
1.5
2
2.5
3
1
2
3
4
5
6
7
8
Distance (km)
Impact
Impact Urban on Urban
Impact Non-urban on Urban
White, Engelen, Uljee 1992

Map comparison based model validation

7

Model: A sample of results for Portugal

Porto

Lisbon

Algarve

Map comparison based model validation

8

Sensitivity


Plotting parameters to GoF is not workable

ƒ
5 parameters + 1 GoF metric = 6 dimensions


Plotting GoF to GoF

Goodness-of-fit correspondance
-120
-100
-80
-60
-40
-20
0
20
40
60
80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa 2
Fuzzy Kappa 8
Goodness-of-fit correspondance
-50
-40
-30
-20
-10
0
10
20
30
40
-20
0
20
40
60
80
100
120
140
Kappa
Aggregation
Clusters appear!

Understanding clusters = understanding parameter space ?

Map comparison based model validation

9

The most pronounced clusters

Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
Map comparison based model validation

10

Separated clusters

Cluster

Parameter 1

Parameter 2

Map comparison based model validation

11

Sensitivity / Bifurcation


Same approach can be used to identify
bifurcations as a consequence of stochasticity


Monte Carlo simulation
0.745
0.75
0.755
0.76
0.765
0.77
0.775
0
1
2
3
4
5
6
7
8
9
Moving window, patch size
Percentage
agreement
Porto: 1 Large cluster

Porto: 2 Mid size clusters

Map comparison based model validation

Calibration

Using map comparison to select the best
parameters

Map comparison based model validation

13

Calibration requires a single metric


Can be a composite of metrics

Š
Metric drives convergence and solution!

Goodness-of-fit correspondance
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
-120
-100
-80
-60
-40
-20
0
20
40
60
80
Fuzzy Kappa
Fractal Dimension
This parameter set?

Or this one?

Map comparison based model validation

14

Best fit: result overview


Low intra
-
metric sensitvity

Š
Strong inter
-
metric sensitivity


Aggregation
Kappa
Factor
Best
Best
2
LU2902
LU1553
4
LU2964
8
LU1096
16
LU1096
Fuzzy Kappa
Euclidean distance
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2902
2
10
LU2349
4
20
LU2902
4
20
LU1862
8
40
LU2902
infinite
10
LU1862
Patch size
Fractal dimension
Halving Distance
Radius
Best
Halving Distance
Radius
Best
2
10
LU2497
2
10
LU1722
4
20
LU2497
4
20
LU1722
infinite
10
LU2497
infinite
10
LU1722
infinite
infinite
LU1573
infinite
infinite
LU2780
Map comparison based model validation

15

Best fit: Results per metric

DATA

2000

DATA

1990

Kappa

Overall

Fractal dimension

Moving window

Euclidean distance

Moving window

fractal dimension

Overall patch size

Moving window


patch size

Aggregated (> 8)

Fuzzy Kappa

Map comparison based model validation

16

Confronting expert judgment


High agreement

ƒ
Aggregation


Euclidean distance


Patch size (moving window)


Fractal dimension (moving window)



Low agreement

ƒ
Patch size (global)


Fractal dimension (global)


Kappa


Fuzzy Kappa

Contradicting earlier empirical
work !

Kuhnert et al 2005, Hagen 2002

Map comparison based model validation

17

Conclusions


Preferred method: Patch size moving window

ƒ
Best result


With least cost of information (aggregation)



Methods must discredit local overestimation

ƒ
Otherwise: edge growth at the cost of
emerging clusters



Methods must be spatial explicit


Otherwise: fringe solution


Map comparison based model validation

Validation

Judging and understanding

Map comparison based model validation

19

Make sense of diverse comparison
results


Step 1. Normalize to make results mutually
comparable


Step 2. Establish a reference level to be able to
pass absolute judgement

Map comparison based model validation

20

Land use change in La R
é
union


Recent urban expansion causes great concern

Š
What will the future bring


Exploration of alternative scenarios


Calibration of a Constraint Cellular Automata


1989

2002

60 km

N

Map comparison based model validation

21

Multi
-
scale, Multi
-
criteria


Comparison: Model 2002


Reality 2002

Š
Moving window based
structure

ƒ
Edge


Patch size


Euclidean distance



Patch size - RMSE
0
10
20
30
40
50
60
70
80
90
1
2
4
8
16
32
64
128
Scale
Error
Euclidean
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1
2
4
8
16
32
64
128
Scale
Error
Edge RMSE
0
0.1
0.2
0.3
0.4
0.5
0.6
1
2
4
8
16
32
64
128
Scale
Error
Map comparison based model validation

22

First step: Normalization


Normalize to level of historical change




Error = 75


Error = 50


Normalized Error =
75/50 = 1.5

MODEL

TRUTH

REFERENCE

Map comparison based model validation

23

Multi
-
scale, Multi
-
criteria

Multi-scale, multi-criteria
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
EDGE
EUCLID
PATCH
Map comparison based model validation

24

Accounting for constraints


Is the model performance caused by the intrinsics
of the model or by the constraints that are
exogeneously imposed on it?

Š
Neutral models
include

constraints and
exclude

a
process of interest

Š
Full model performs better than neutral model?


Increased confidence in the process


Source: McGarical 2001

Fractal

Map comparison based model validation

25

Neutral models of landscape
change

Growing clusters

-
Subject to area constraints

-
Minimal area of change

-
Location alongside existing clusters

Random constraint match

-
Subject to area constraints

-
Minimal area of change

-
Random locations

Map comparison based model validation

26

Random Constraint Match

Open:
-
32

City:
-
15

River: +18

Park: +29

Before

After

RCM

RCM /
After

RCM /
Before

% Correct

0.77

0.98

Kappa

0.35

0.94

Kappa Location

0.35

1.00

Kappa Histo

1.00

0.94

Map comparison based model validation

27

Patch size
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Presence (Euclidean)
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Expressing similarity relative to neutral
models

Edge
0
1
2
1
2
4
8
16
32
64
128
Scale
Error normalized to scale
CLUST
RCM
SIM
Map comparison based model validation

28

Significance


Assess significance by means of Monte Carlo
simulation and application on multiple time
periods and regions

Similarity


= 0.8 +/
-

0.04

Similarity



= 0.7 +/
-

0.01


Model > Reference ?


98 %

MODEL

TRUTH

REFERENCE

Map comparison based model validation

29

Thanks for your attention


Map Comparison Kit

ƒ
Software, papers & documentation:
www.riks.nl/mck
/


Today’s slides and exercises
www.riks.nl/news



Contact

ƒ
Alex Hagen
-
Zanker


ahagen@riks.nl