Cohen's Kappa

molassesitalianAI and Robotics

Nov 6, 2013 (3 years and 11 months ago)

73 views

Cohen’s Kappa

Christian
Börmel

Yuima
Irigaki

Mhairi

O’hara

Outline


Background


Cohen’s kappa


Application in Remote Sensing


Evaluation of Cohen’s Kappa




Background


Accuracy is important in land classification
data (remotely sensed data)


Failing to know the accuracy may lead to
incorrect data use


Attempt to attach the accuracy to the data


(
Congalton
, 1991)

Accuracy

A

B

C

Total

A

69

7

2

78

B

1

83

5

89

C

4

3

53

60

Total

74

93

60

227

Ground truth

classified

Possibility of being classified correctly
by chance
?

User’s accuracy =

Producer’s accuracy =

Overall accuracy =

N

the total number of observation

r

the number of rows in the matrix

x
ii

the marginal totals of row
i

and



column
i


(
Foody
, 2002)



i
ii
x
x
i
ii
x
x

100
1



N
x
r
i
ii
Cohen’s Kappa

A

B

C

Total

A

69

7

2

78

B

1

83

5

89

C

4

3

53

60

Total

74

93

60

227

Ground truth

classified

1 1
2
1
( * )
ˆ
( * )
r r
ii i i
i i
r
i i
i
N x x x
K
N x x
 
 
 




 

N

the total number of observation

r

the number of rows in the matrix

x
ii

the marginal totals of row
i

and



column
i


(
Foody
, 2002)


Kappa ranges from 1 (accurate) to 0 (inaccurate)

The kappa coefficient of the example is 0.853 (highly accurate)

Satellite Imagery Accuracy Assessment


Class Labels vs. Ground Data


Represented by Confusion or Error Matrix


Cases % Correctly Allocated (Overall Accuracy)


Statistical Accuracy Analysis expressed by
‘Cohen Kappa Coefficient of Agreement’


Evaluation of different Classification Algorithms


Gain understanding of Errors


Validate Use of Image for Research

Satellite Data Classification



Accuracy Assessment in Thematic Mapping looks
at

the
Degree of ‘Correctness’ of a Map or Classification



Satellite Imagery Errors


Sensor Properties


Preprocess



Image Classification


Confusion/Error Matrix

Land Forest Classification

Sungai
Tekai

Forest Reserve, Malaysia


Landsat TM data


Captured 8
th

May 2001


Supervised Classification
Method



Image Classes:

1.
Primary Forest

2.
Logged Over Forest

3.
Agricultural Crop

4.
Water Bodies

5.
Bare Land

Cohen Kappa’s Coefficient

Overall Kappa Statistic:
0.75

Excellent Agreement not due to Chance

Image Classification vs. Ground Data

Land Cover Error Matrix

K<0.4

Poor Agreement

0.4<K<0.7

Good Agreement

K>0.75

Excellent Agreement


Evaluation of Cohen’s kappa


Pros


Uses
not only the diagonal axis, but all
elements in the error matrix (
Foody
, 1992
)


Often
argued to be a standard measurement
of accuracy for remotely sensed data (
Foody
,
2002)


Cons


Over
-
estimated agreement by chance


→ Under
-
estimates actual classification accuracy

Evaluation of Cohen’s kappa


Alternatives


Tau: Similar calculation to Cohen’s kappa


Ratio between number of pixels grouped
correctly + number of pixels that were not
correctly grouped by random
assignment



(
Ma and Redmond, 1995)




Fleiss’ kappa:
used for more than two
raters

(
Fleiss
, 1971)


Campbell
, W. G., &
Mortenson
, D. C. (1989). Ensuring the quality
of geographic
information system data: a practical application of
quality control
. Photogrammetric Engineering and Remote Sensing
,
55, 1613


1618.

Canters, F. (1997). Evaluating the uncertainty of area estimates
derived from
fuzzy land
-
cover
classification.
Photogrammetric
Engineering and
Remote Sensing,
63, 403

414.

Cohen, J. (1960) A coefficient of agreement for nominal scales.
Educational and Psychological Measurement
20, 37
-
46.

Congalton
, R. G. (1991) A review of assessing the accuracy of classifications of remotely sensed data. Remote Sensing of
Environment 37:35
-
46.

Congalton
, R. G. (1994). Accuracy assessment of remotely sensed
data: future
needs and directions
.
Proceedings of
Pecora

12 land
information from
space
-
based
systems
, 383


388.

Congalton
, R. G.,
Balogh
, M., Bell, C., Green, K., Milliken, J. A., &
Ottman
, R
. (1998). Mapping and monitoring agricultural crops
and
other land
cover in the Lower Colorado river
basin.
Photogrammetric
Engineering and Remote Sensing
, 64, 1107


1113
.

Dicks, S. & Lo, T. (1990) Evaluation of thematic map accuracy in a land
-
use and land
-
cover mapping program.
Photogrammetric
Engineering and Remote Sensing,

56
,

1247
-
1252
.

Fleiss, J. L. (1971) Measuring nominal scale agreement among many raters.
Psychological bulletin,
76, 378.

Foody
, G. M. (1992) On the compensation for chance agreement in image classification accuracy assessment.
Photogrammetric
Engineering and Remote Sensing, 58, 1459
-
1460
.

Foody
, G.M. (2002) Status of land cover classification accuracy assessment.
Remote Sensing of Environment
, 80, 185


201
.

Hay, A. M. (1979). Sampling designs to test land
-
use map accuracy.
Photogrammetric Engineering
and Remote Sensin
g, 45, 529

533.

Ismail, M.H. and
Jusoff
, K. (2008) Satellite Data Classification Accuracy Assessment Based from Reference.
International Journal of
Computer and Information Engineering,
2(6).

Jensen
, J. R. (1996). Introductory digital image processing.
A remote sensing perspective
(2nd ed.). New Jersey: Prentice
-
Hall
.

Ma, Z. & Redmond, R. L. (1995) Tau coefficients for accuracy assessment of classification of remote sensing data.
Photogrammetric Engineering and Remote Sensing,

61
,

435
-
439.

Richards, J. A. (1996). Classifier performance and map accuracy.
Remote Sensing
of Environment
, 57, 161

166
.

Stehman
, S. V. &
Czaplewski
, R. L. (1998) Design and analysis for thematic map accuracy assessment: fundamental principles.
Remote Sensing of Environment,

64
,

331
-
344.

References