KIM Knowledge Driven Information Mining in Remote

kettlecatelbowcornerΤεχνίτη Νοημοσύνη και Ρομποτική

7 Νοε 2013 (πριν από 3 χρόνια και 10 μήνες)

126 εμφανίσεις








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
1

of
33







KIM

Knowledge Driven Information Mining in Remote
Sensing Image Archives


Project Executive Summary



Ref.: ESA
-
KIM
-
EXS
-
1.1

Issue: 1.0

12.11.02












Acknowledgment:

The KIM projects was accompanied with contributions of Mr. Hervé Touron,
Mr. Ju
an Luis Valero and Mr. Lucio Colaiacomo from EU

SC (Madrid). EU
-
SC provided the Landsat and Ikonos data and participated at the evaluation
procedure.



NAME
-

COMPANY

DATE

SIGNATURE

Prepared by:

Prof. Mihai Datcu DLR
-
IMF

Dr. Klaus Seidel ETH Zurich

12
.11.02


Verified by:

Oscar Guerra ACS

12.11.02


Approved by:

Prof. Manfred Schroeder DLR
-
IMF

12.11.02



Authorized by:

Prof. Mihai Datcu DLR
-
IMF

12.11.02









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
2

of
33


Document Status Sheet

Issue

Date

Details

0

20.10.02

Document Creation

1.0

07.11.02

Issue 1.
0

1.1

12.11.02

Upgrade with GUI images, references, acknowledgment

























KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
3

of
33


Table of Contents

1. INTRODUCTION ……………………………………………………………………………………………………….……...

4

1.1

Purpose of the document ……………………………………….……………………………………………….………

4

1.2

What is KIM? …
…………………………………………………………………………………………………………

4

1.3

Acronyms and abbreviation ………………………………….………………………………………………………….

5

2. BASIC KIM CONCEPTS ……………………………………………………………………………………………………...

6

2.1 Abstract ………………………………………………………………………………………………………….………

6

2.2

The EO
information burden and the technological challenges ………………………………………………….………

6

2.3

From data to information ……………………………………………………………………………………………….

7

2.4

A better grasp of user interaction ……………………………………………………………………………………….

9

2.5

KIM concepts ……………………………
…………………….………………………………………………………..

9

2.6

Logical Model ………………………………………………….……………………………………………………….

12

2.7

KIM system architecture ……………………………………….……………………………………………………….

13

2.8

The KIM system ………………………………………………………………………………………………………..

14

2.9

Examples of ap
plication scenarios ………………………………………………………………………………………

17

3. OVERVIEW OF DATA
INGESTED IN KIM …………
……………………………………………………
………………...

19

3.1 Mozambique
……………………………………………………………………………………………………………..

19

3.2

Switzerland ……………………………………………………………………………………………………………...

20

3.3

Nepal ……………………………………………………………………………………………………………………

21

3.4

Summary of data ingested in the KIM system and the type of function implemented …………………………………

22

3.5

Evaluation of the computation time at data ingestion ………….……………………………………………………….

23

3
.6

Evaluation of the Human Machine Interface ……………………………………………………………………………

24

4. EVALUATION PROCE
DURE ………………………………………
……………………………………………………
….

25

5. FUTURE ISSUES ……
……………………………………………………
………………………………………………….
..

27

5.1

How and where to use KIM …………………………………
…………………………………………………………

28

5.2

Further development …………………………………………………………………………………………………...

31

6. CONCLUSIONS …………
……………………………………………………
………………………………………………

33








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
4

of
33


1

INTRODUCTION


1.1

Purpose of the document

This document is intended to give a complete but s
imple presentation of the theoretical
concepts used to design the KIM system, to present the KIM system architecture, the
data which have been ingested and are now managed by the KIM system, to present and
analyse the results of the system evaluation, and
to summarise conclusions and
requirements for further work.

The document is also an easy guide to help understanding of KIM concepts and system,
and to facilitate the understanding of KIM documentation.


1.2 What is KIM?

KIM is a novel and unique the
oretical concept and frame of collaborative methods for:



Extraction and exploration of the content of large volumes of image or other
multidimensional signals



Establishing the link between the user needs and knowledge and the information
content of images



Communicating at high semantic abstraction between heterogeneous sources of
information and users with a very broad range of interest

KIM is the first prototype of a new generations of advanced tools and systems for:



Intelligently and effectively accessing

the information content in large EO data
repositories



Better exploration and understanding of Earth structures and processes



Decrease costs, and increase the accessibility and utility of EO data.














KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
5

of
33


1.3

Acronyms and abbreviation


ACS

Advanced Computer
System S.p.A.


DLR

German Aerospace Centre


DBMS

Data Base Management System


EO

Earth Observation


ESA

European Space Agency


ETH

Swiss Federal Institute of Technology, Zurich/Switzerland


GMRF

Gauss
-
Markov Random Fields


GRF

Gibbs Random Field


G
S

Ground Segment


HMI

Human Machine Interface


I2M

Image Information Mining


I
2
M

Image Information Mining


IMF

DLR Remote Sensing Technology Institute


MMI

Man Machine Interface


QL

Quick Look


RD

Reference Document


SAR

Synthetic Aperture Radar


SoW

Statement Of Work


SR

System Requirements


SRD

System Requirements Document


UR

User Requirements


EU
-
SC

European Union Satellite Center












KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
6

of
33


2. BASIC KIM CONCEPTS

2.1

Abstract

Information mining/knowledge discovery and the associated data mana
gement are changing the
paradigms of user/data interaction by providing simpler and wider access to Earth Observation (EO)
data archives. Today, EO data in general and images in particular are retrieved from archives based
on such attributes as geographica
l location, time of acquisition and type of sensor, which provide no
insight into the image’s actual information content. Experts then interpret the images to extract
information using their own personal knowledge, and the service providers and users combi
ne that
extracted information with information from other disciplines in order to make or support decisions.
In this scenario, the current offering, which is data sets or imagery , does not match the customer’s
real need, which is for information . The

information extraction process is too complex, too
expensive and too dependent on user conjecture to be applied systematically over an adequate
number of scenes. This hinders the access to already available or new data (data accumulation rate
is increasin
g), penalises large environmental
-
monitoring type projects, and might even leave critical
phenomena totally undetected. Emerging technologies could now provide a breakthrough,
permitting automatic or semi
-
automatic information mining supported by intellig
ent learning
systems.

2.2

The EO information burden and the technological challenges

Users in all domains require information or information
-
related services that are focused, concise,
reliable, low cost and timely and which are provided in forms and formats
compatible with the
user’s own activities. In the current Earth Observation (EO) scenario, the archiving centres
generally only offer data, images and other low level products. The user s needs are being only
partially satisfied by a number of, usually s
mall, value
-
adding companies applying time
-
consuming
(mostly manual) and expensive processes relying on the knowledge of experts to extract information
from those data or images.

In the future, these processes will become even more difficult to perform an
d to manage because of
the growing diversity of the user communities, the greater sophistication of user needs requiring, for
example, the fusion of multi
-
sensor or EO and non
-
EO data, and the exponential increase in the
volume and complexity of the data a
rchives, due to the rapid increases in:



number of missions (even constellations)



number of sensors



kinds of sensed data



sensor resolution



number of spectral bands



number of data formats



number, type and size of distributed archives.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
7

of
33


Today
’s Synthetic Aperture Radar (SAR) and optical sensors generate 10 to 100 Gbytes of data per
day, so that in a multi
-
sensor spacecraft scenario the volume of data to be archived annually easily
reaches 10 Tbytes. However, this figure can sometimes be at le
ast one order of magnitude larger:
The Shuttle Radar Topography Mission (SRTM) provided about 18Tbytes of SAR data in just 11
days, and ESA s Envisat spacecraft launched on 1 March 2002 is going to collect about 80 Tbytes of
multi
-
sensor data per year! Fut
ure European programmes like GMES (Global Monitoring for
Environment and Security) will be even more challenging, unless major progress is achieved soon.
Emerging technologies for the automatic extraction, classification and easy provision of
information,
from EO data alone or after fusion with data and information from other fields, could
provide this breakthrough.

After 30 years of remote sensing, for almost any site on Earth there are data takes piling up. They
contain valuable information that is not b
eing fully exploited because of the lack of automated tools.
New technologies are required to automatically analyse such data and data series to detect changes
and trends, for example, which could otherwise remain hidden forever or be detected only by
chan
ce.

Reference:

M. Datcu,


K. Seidel,


S. D Elia and P.G. Marchetti,


"
Knowledge Driven Information Mining in
remote sensing image archives
",


ESA Bulletin,

pp. 26
-
33,

2002


2.3

From data to information

In recent years, our ability to store large quantities of data has greatly surpassed our ability to access
and meaningfully extract information from it. The state
-
of
-
the
-
art of operational systems for remote
-
sensing

data access, particularly for images, allow queries by geographical location, time of
acquisition or type of sensor. This information is often less relevant than the content of the scene,
i.e. structures, objects or scattering properties. Meanwhile, many
new applications of remote
-
sensing data require knowledge of the complicated spatial and structural relationships between
objects within an image. This knowledge is hidden in the image’s structure and must be mined to
retrieve meaningful spectral or po
larimetric signatures or objects of higher
-
level abstraction, such as
cities, roads, rivers, forests, etc. The hidden information can relate to very localised phenomena,
such as subsidence or even to the structural stability of individual buildings, but ca
n also include
phenomena related to global change. Knowledge
-
driven information mining from EO archives
requires the exploitation of a family of methods for knowledge discovery, learning and automatic
information extraction from large amounts of data. It m
ay be performed with the identification of a
specific feature and application in mind, such as the high density of strong scatterers and structures
in SAR images to detect settlements, hot spots in ATSR products to detect fires, etc. Alternatively, it
may
be used to identify key features without having a specific application in mind at that very
moment. Companies and research centres around the World are devoting a large effort to the second
approach through the design and production of Content
-
Based Image
Retrieval (CBIR) systems.
Several attempts have already been made to apply the CBIR approach to EO archives, but
difficulties have been encountered in applying searching by global image similarity (the basic
concept of CBIR tools) because of the predominan
ce in the EO domain of grey
-
scale and false
-
colour imagery. The problem of how the system might remember particular features so that it gets
smarter with increasing use is also being addressed.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
8

of
33



Our approach, unlike traditional feature
-
extraction method
s that rely on analysing pixels and
looking for a predefined pattern, it is based on extracting and storing basic characteristics of image
pixels and areas, which are then selected (one or more and weighted) by users as representative of
the feature being
searched for. This approach has a number of advantages:



there is no need to re
-
scan the entire image archive to detect new features



the selected feature can be closer to the user s expectations and perceptions (the same feature
can have different meani
ngs for different users: e.g. a forest for an environmentalist, a forest
ranger, a geologist, or an urban planner)



the system can learn from experts knowledge.

This KIM project considered all of the above background into account, as well as the facts th
at:



The huge and exponentially growing volumes of existing and new EO data archives need to
be more fully exploited in terms of their true information potential.



Human
-
centred computing will play an increasing role in the design of EO data exploitatio
n,
i.e. intelligent man/machine interfaces, systems that infer and adapt to user needs, etc.



Fusion of sensor data with non
-
EO data and information will be used to better understand
the identities of the observed scenes and the Earth cover structures.



Information mining, knowledge discovery and other exploratory information
-
retrieval
methods should be used to try to fully understand highly complex data, phenomena or global
observations.



There is a need to enlarge and reinforce the reconnaissance/sur
veillance applications
spectrum.



It is necessary to migrate (adopt and promote) from data to information management and
dissemination.


Thus, KIM is a solution to satisfy the requirements of the various communities including:



End users (access to basic

information in a simple way).



The EO value
-
adding industry and service providers (access to data and information for
enhancing existing and providing new services).



The scientific community (access to large information sets, e.g. for the analysis of

global
change).



Civil protection agencies (access to specific information in support of their operational
activities, directly or through service providers).



Institutions involved in education (access to various data and information types to be used
as
examples or training cases).









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
9

of
33



2.4

A better grasp of user interaction


In today s EO ground segments, the retrieval of images is mainly based on sensor, location and time
criteria. A more user
-
friendly interaction model would also permit searches usin
g other attributes of
the image or its parts. Access to a desired image from the archive might thus involve a search
through:



mission attributes such as sensor, time and location



the presence of a particular combination of intensity, texture and shap
e



the presence of specific object types, e.g. forest, city, road, etc.



the presence of a particular type of event, e.g. burnt forest, flood.

This list of possible queries represents an increasing level of abstraction, complexity and answering
diffi
culty (requiring more and more reference to some body of external knowledge), corresponding
to the increasing complexity of the related attributes, which can be classified from low to high
complexity, as:

Basic attributes

Sensor, time, latitude, longitude
(or location name), directly related for example to raw data or
geocoded products

Primitive features

Intensity, texture or shape, for example: attributes that are both objective, and directly
derivable from the images themselves, without the need to refer

to any external knowledge base

Derived features

Sometimes known as logical features, these are objects of a given type (e.g. build up area ) or
specific objects (e.g. industrial site ). This level involves some degree of logical inference about the ide
ntity of
the objects and permits searches in user semantic terms (which can be assigned during system training to
weighted combinations of primitive features)

Abstract attributes

Involving a significant amount of high
-
level reasoning about the meaning and

purpose of the
objects or scenes depicted (e.g. illegal plantation). They are outside the scope of the current research activity.

A model based on primitive and derived features would also require semi
-
automated extraction of
primitive features, image ann
otation in terms of primitive and derived features, and the capability to
select, weight and combine primitive or derived features during the query process.


2.5


KIM concepts

The concept applied in KIM is aimed at building a system free from application

specificity, so as to
enable its open use in almost any scenario, and also to accommodate new scenarios required by the
development of new sensor technology or growing user expertise. These goals are reached by







KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
10

of
33


defining a hierarchy of information represen
tation, as presented in the Figure 1, such to enable the
communication between the image archive and the users













Figure 1
:
Hierarchical modelling of image content and user semantic

The image data (level 0) is processed for extracting the image

primitive features and meta features (levels 1 and
2). Information at level 1 is in form of parameter vector of a signal model. The specific signal model is the level 2
of information representation. Further we obtain by unsupervised information clusterin
g a characteristic
vocabulary of signal classes (level 3) for each signal model. User specific interests, that is, cover type labels, are
linked to combinations of these vocabularies using simple Bayesian networks. Levels 1 to 3 are obtained in a
completel
y unsupervised and application
-
free way during data ingestion in the system. The information at level 4
can be interactively defined by users using a learning paradigm. In an additional step of stochastic modeling, we
describe the stochastic link between s
ignal classes and user (subjective) labels using a vector of hyper
-
parameters.



Its first step is the extraction of primitive features and the reduction of the resulting data into
primitive feature clusters. The primitive features to be extracted need to

be carefully selected, since
they mainly determine the quality and capabilities of the resulting system. SAR and optical images,
for example, will have to be handled differently and texture primitive features will have to be
extracted at various resolutio
n levels, since different textures can dominate at different scales.

The steps necessary to properly extract primitive features in the EO context are


Image geo
-
coding

Permit co
-
registration of different images and absolute geographical reference of the
features

Sub
-
scenes

Large images are split into sub
-
scenes to reduce the probability that an image item contains all the
primitive features and therefore that it is always retrieved during a search

Multiresolution

Perform a pyramid of different resolutions

of each sub
-
scene to ensure that the various textures
are identified at the related scale

Texture analysis

Optical image Extract primitive features using the Gibbs Random Field approach

un
-

supervised
clustering

clusterin

clustering

un
-

supervised

clustering








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
11

of
33


SAR image

De
-
speckle the image and extract primitive features using

the Gauss
-
Markov Random Field
approach

Segmentation by geometry

The homogeneous areas that can be detected in geo
-
coded image are assimilated to
reference shapes and related to an absolute coordinate system

Primitive feature extraction generates a huge a
mount of data, which cannot be handled in practice
and therefore has to be compressed somehow. Each pixel of the image will be located in n
-
dimensional space in the position determined by the values of the contributing primitive features
(their units are n
on
-
commensurable, e.g. texture and spectral features). The features will tend to
group themselves into specific regions of this space. Through clustering the clouds of image
primitive features are replaced by parametric models of their groups, which can
be represented in
more compact forms. This reduces the precision of the system, similar to a quantisation process, but
permits its practical use thanks to the huge data reduction obtained. The primitive features are
grouped into clusters using the K
-
means
approach.

The clusters (condensed representation of primitive features) have no direct meaning, since they
group points in an n
-
dimensional space of non
-
commensurable variables. Still they represent
characteristics of the image seen as a multi
-
dimensional

signal. It is possible to associate meaning
with these clusters through training. A user can tell the system that a specific, weighted combination
of some clusters represents a derived feature of the image. By making this association, it is possible
to se
lect all images in the database that have that specific combination and may therefore contain the
feature that the user is searching for.

The second step in KIM is aimed at assigning
meaning

to the primitive features, i.e. at identifying
derived features
. This information
-
mining step involves a learning phase. The system presents
sample images in which the user marks areas with positive and negative traits, refining the
definition of the derived feature through an iterative process. Once this process/syst
em training has
been satisfactorily completed, the definition can be saved and used by other users, who then will
have only to request images containing the derived features corresponding to that definition.

The expert can train the system by pointing and

clicking via a graphical user interface on positive
and negative image
-
structure examples (and therefore the corresponding clusters), in two steps:



first of all to identify specific broad cover
-
type definitions, related to broad domains of
possibl
e user interest (e.g. geology, forestry, &)



thereafter to create from the aggregation of the above definitions, and the possible use of
additional training pixels, more precise definitions with semantic meaning (i.e. concepts ,
like wood, water, gras
s, urban area, etc.).

A simple Bayesian network links primitive feature clusters and definitions and these associations
can be stored and made available to users for subsequent interactive sessions.

With this approach, we are modelling and learning about

the users interests and actions, we
developed a system that adapts to the users particular interests and incorporates contextual
information to determine the user s intentions and degree of satisfaction with the results. It should
provide a breakthrough b
y establishing a new pattern for user
-
EO system (archive) interaction, and







KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
12

of
33


a quantum leap with respect to the more traditional feature
-
extraction systems. The aim with KIM is
to help users uncover the most relevant image information content, by providing a
n
eye
with which
to delve into multi
-
sensor and multi
-
temporal image data archives.


References:

M. Datcu, K.Seidel, M. Walessa, “Spatial Information Retrieval From Remote Sensing Images: Part I.
Information Theoretical Perspective”, IEEE Tr. on Geoscienc
e and Remote Sensing, Vol. 36, pp. 1431
-
1445,
1998.

M. Datcu, K.Seidel, G. Schwarz, “Elaboration of advanced tools for information retrieval and the design of a
new generation of remote sensing ground segment systems”, in I. Kanellopoulos, editor, Machine
Vision in
Remote Sensing, Springer, pp. 199
-
212, 1999.

M. Schröder,


H. Rehrauer,


K. Seidel and M. Datcu,


"
Interactive Learning and Probabilistic Retrieva
l in
Remote Sensing Image Archives
",


IEEE Trans. on Geoscience and Remote Sensing,

pp. 2288
--
2298,

2000


2.6

Logical Model

The logical structure of the system is presented in the following figure.




Figure 2

:
Overall KIM system logical model


At da
ta ingestion the geo
-
referenced or co
-
registered multi
-
mission images are tiled in
sub
-
images, and indexed. The images are processed for primitive feature extraction. The
primitive feature extracted are: texture parameters, spectral signature, geometrical
attributes. The extraction of texture and geometrical attributes is done for multiple
image scales. Each feature space is independency clustered (unsupervised). The tiled
images are stored in a repository. The DBMS catalogue system containing all types of







KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
13

of
33


meta
-
information. The query engine enables the image search based on the existing
meta
-
information. The browsing engine enables the visual search and evaluation of the
query results. By classification the information extracted from different image primitiv
e
features or different sensor data is aggregated (fused) such to explain the subjective
request of the user. The knowledge acquisition

modules stores the associations in
between the image content, the image primitive features and the user conjecture. Usin
g a
man
-
machine interface the system learns the user conjecture, and the user learns the
basic properties of the data in archive.





2.7

KIM system architecture

The principal components of KIM and their interaction is explained in the following
figure.




Figure
3

:
The main components of the Kim system









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
14

of
33


The dependency relationships in the component diagram depict how various modules use each
other. A colour convention has been adopted to distinguish modules to be considered as external.
Only yellow pa
ckages are KIM ones.

All the required components have been classified in separate packages and they are the following.

The Parameter Extraction and Ingestion modules are the components that extracts the primitive
information items from the input images and

store them in the database. The Database Server and
Archive is based on the Informix Dynamic Server and the Geodetic Datablade Module, and used to
perform geo
-
temporal queries. The Interactive Learning Applet is a Java applet used to interact with
the us
er in order to train a label. The Learning Servlets are located at the server side part assigned
to label training. The Data Presentation Servlets are used to present the data in a normal search
session (region of interest, time, satellite, sensor and
pre
-
defined label). The Data Presentation
Applets is used to present the data in a normal search session (region of interest, time, satellite,
sensor and pre
-
defined label). The Map Server is the VPF server side part needed to display a map
and navigate
it.

2.8

The KIM system

The KIM system is a server
-
client architecture designed for over
-
the
-
net operation.














Kim Server

The Web Client

The User Management Interface is based on HTML pages to provide the
possibility to register new user or, fo
r the registered one, to log in the system and to accede to KIM
services. The Interactive Learning module is a java applet used to define a new label or refine an
existing one. User can select an image’s area to zoom and apply lut on this area. The inte
rface for
Searching images with a fixed label enables a user to set some parameters to search in images data
base. In particular the user can perform search by a fixed previously defined label. The Data
presentation are HTML pages presenting the search
results.


To choose an image on which starting learning, either use an image identity number or decide to
visualize a set of random images. To select the image in the random set, click on the preview or on
“Learn with this image” button.


Web

client

Kim

server

Web browser

Java Plug
-
in

User Management Interface:

register new user, login user

Interactive Lear
ning: label definition

navigation, lut, search to refine

Search images with a fixed label.

Data presentation.

Informix dynamic server

Java VM

Web Server Tomcat

Map server VPF

User Management Servltes

Util Servlets

Search Servlets

Kim Data Base

VPF Data Base








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
15

of
33







Figure 4:
The gallery of images as example for starting a search session
.



The following image shows the learning phase interface. The graphics component have been labeled
to make easier the ex
planation of their functions.





Figure 5:

The “Interactive learning” GUI.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
16

of
33


The interactive learning panel is made by the following four main graphic componen
ts:



-
The image panel displays the image chosen for learning.


-
The a posteriori map
after each click is re
-
painted with different colours, representing

the probability of each point. Black to white denote posterior probabilities continuously
from 0 to

1. Above a certain threshold, this probability is depicted in red. Unclassified
pixels are coloured in cyan. Clicking on them does not have any effect on the learning
process.


-
The zoomed area shows a zoomed portion of image.


-
The distance bar panel sh
ows the contribution of each model to the learning phase.



To browse in the KIM catalogue user can download the presentation applet accessible from all
pages clicking on “Catalog browse”. As before the following image shows the presentation applet
where p
rincipal components have been labeled.





Figure 6:
The browsing GUI.


On the left panel of presentation applet user can define parameters for browsing in the catalogue,
while on
the right side are shown the search results. The central element consists of a map which
helps for the area definition and for the localization of results


The Kim Server

The User Management servlet, manage the register and the login of the users.
The Ut
il Servlets Provide to Interactive Learning the selected images, the classfiles and the cropped
and zoomed images. The Search Servlet manages the search in images data base by selected
parameters. The KIM Data Base stores and manages the all information us
ed by the system. The
VPF Data Base manages the data used for map visualization and navigation.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
17

of
33


2.9

Examples of application scenarios


The KIM system provides a wide variety of mining tools, including semantic querying by image
content and image exampl
e, and interactive classification and learning of image content. As an
example, we can take an archive of Landsat TM and ERS
-
1 images covering the whole of
Mozambique. The Landsat and ERS
-
1 scenes have been partitioned into sub
-
scenes of 2000 x 2000
pixels
, with all data geo
-
coded and co
-
registered in a pre
-
processing step. The user has available a
catalogue list of semantically valid land
-
cover structures, e.g. mountains, rivers, cities, roads, etc.

Figure 7

shows the result of a search for the cover stru
cture city and
Figure 8

demonstrates the
results of a cover
-
type search for riverbed.







Figure 7 :

Result of a search for cover structure city






Figure 8

:
Result of a search for cover
-
type riverbed



The interactive learning func
tion is a valuable mining tool for exploring the unknown content of
large image archives. A Graphical User Interface (GUI) enables the user to select, by clicking on the
image, those structures of greatest interest, which then appear in red on a grey
-
scale

visualisation of
the relief according to the Bayesian learning of the structure recognition.



The last examples, in
Figure 9
, are based on a panchromatic Ikonos image of outskirts of Maputo
(Mozambique) and show how the user can explore the image,

marking by pointing and clicking the
features of interest.









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
18

of
33







Figure 9

:
Image from top to down and left to right: Monochr. Ikonos image (1m) of an outskirt area of
Maputo (Mozambique). Selection of grassland, inhabited regions and industr
ial regions





In this case, the red areas associated with grassland, inhabited and industrial areas have been
generated by a previous user clicking on the image, the KIM interactive learning module
being able, in real time, to generate a set of supervised

image classification maps. The
interactively induced image classification is generalised over the entire image archive.










KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
19

of
33




3. OVERVIEW OF DATA
INGESTED IN KIM


This section presents an overview of the data ingested in the KIM system, the sites, the data
volume,
the extracted features, and an evaluation of the time for primitive feature extraction and ingestion in
the DBMS catalog.

3.1

Mozambique

In the KIM system are ingested multimission


optical and SAR


data of 2 sites in Mozambique.

In
Figure 10

i
s depicted the covarege of the site by the multimission data. The surface covered is
larger than 400km x 800km.



Figure 10 :
the covarege of the site by the multimission data












KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
20

of
33


Sensor 1: Landsat TM (multispectral)

No. Of frames = 14

Resolution = 30m

Processing level = georefernced

Remark: IR channel not used


The Landast images of Mozambique have very high complexity, both from the point of view of
image content and subjective understanding by an user. The images have huge diversity of spectral
signa
tures, and a very broad variety of structural information at all scales. Most of the structures are
natural, having intricated shapes and textures. From the point of view of visual understanding, the
scene, for many users, is an un
-
known scene, and due to
the different, geologic, climate, cultural and
technological environment, the observed structures are not easy to be understood. That why the
selection of this data set was a challenge, and a real typical task for Information Mining.


Sensor 2: ERS 1 (Sy
nthetic Aperture Radar)

No. Of frames = 43

Resolution = 30m

Processing level = GTC


Contrary to the Lansat TM optical images, the ERS 1 SAR images of Mozambique have rather low
information content. The low diversity of the image content is due to joint co
ntribution of several
factors, among them: the look angle ~23 dgr., the C band response for humid land cover or
vegetation, the type of materials used for the man
-
made structures. However, due to the SAR
sensitivity to the surface geometry, large scale str
uctures, like rivers, geomorphology, etc. are well
visible in the images.


Sensor 3: Ikonos (panchromatic)

No. Of frames = 2

Resolution = 1m

Processing level = georefernced

Remark: data corrected radiometrically


The images have a broad diversity of stru
ctures and objects, both natural and man
-
made.


3.2

Switzerland

The Swiss territory, a surface larger than 400km x 500km is covered by 5 Landsat TM scenes. The
Figure 11

presents the coverage with tiles of 2000x2000 pixels extracted from the original sce
nes.










KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
21

of
33




Figure 11

:
the coverage with tiles of 2000x2000 pixels extracted from the original scenes



Sensor 1: Landsat TM (multispectral)

No. of frames = 5

Resolution = 30m

Processing level = georefernced

Remark: IR channel no
t used


The images have rather high information content. The data set is typical for alpine regions.



3.3

Nepal

One Landsat 7 MS and one panchromatic scene in Nepal covering a surface of 100km x 100km are
used specifically for the evaluation of the cla
ssification accuracy of the KIM system.


Sensor 1: Landsat 7 (multispectral)

No. Of frames = 1

Resolution = 30m

Processing level = georefernced

Remark: IR channel not used










KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
22

of
33


Function

C

Mining

Archive

Scenes

Sensor

Site

CBIR

88

52MB

4.4 GB

5

Landsat TM

Switzerland

1.8 GB


254 MB

124 MB

800 MB

700 MB

12


1

4.5

4

14

216 MB

1

Landsat TM


Nepal

120 MB

1

Landsat
Pan.


Scene

Understanding

560 MB

2

Ikonos


20 GB

53

all

TOTAL

3.5 GB

30

ERS
-
1

Information

M
ining

10.5 GB

14


Mozanbique

Sensor 2: Landsat 7 (panchromatic)

No. Of frames = 1

Resolution = 15m

Processing
level = georefernced

Remark: used only for texture extraction at 30 and 60 m scales



3.4

Summary of data ingested in the KIM system and the type of
function implemented


The following table presents the summary of all data sets ingested in the KIM system
, organized by
site, sensor, showing the total volume of data archived (Archive), the volume of data necessary for
on
-
line operation of KIM (Mining), the compression factor C as the ratio of archive size vs. mining
information size. The table outlines the
nature of the function implemented in KIM, corresponding
also to the type of evaluation.


Table 1:
Summary of data ingested in the KIM system and the type of function implemented.




In the Content Based Image Retrieval, mode, the compression factor is v
ery high ~100, thus
enabling the search of very large volumes of data, definitely the accuracy of the search is limited,
proportional to the available information for on
-
line search.


The Information Mining mode is using a moderate compression factor, thus

enabling the access to
mote detailed information for on
-
line mining.









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
23

of
33


In the Scene Understanding mode, all image content information is used, thus enabling accurate
exploration and interpretation of the images and observed scenes




3.5

Evaluation of the

computation time at data ingestion


Further operational application of the KIM system, require the appropriate sizing of the data
computation power of the ingestion chain, i.e. the “real time” data ingestion, real time relative to the
application nature,
e.g. pipeline in a Ground Segment system, synchronization with periodic update
(refresh) of data in large robot archives, or operation in a surveillance task.


The Table 2 summarizes the computation time for primitive feature extraction, at different scale
s for
a typical number of estimation points 466 x 466 = 217 156 using a single CPU at moderate clock
speed. The following notations are used: EMBD


enhanced model based despeckling, GMRF
-

estimation of the texture parameters of a Gauss
-
Markov Random Field
, the two primitive features
are obtained in a single computation step.



Table 2 :
summary of computation time for data ingestion

Sensor

Primitive

feature

No of
estimation


points

Feature
Extraction

1 CPU

Clustering

1 CPU

time in min.

time in min.

Landsat

Spectral

466x466

0

LINUX, 800 MHZ

6

LINUX, 800 MHZ

Texture, s0

466x466

30

SUN, 800 MHZ

3

LINUX, 800 MHZ

Texture, s1

466x466

30

SUN, 800 MHZ

3

LINUX, 800 MHZ

Texture, s2

466x466

30

SUN, 800 MHZ

3

LINUX, 800 MHZ

Geometry, s1

466x466

30

LINUX
, 800 MHZ

1

LINUX, 800 MHZ

ERS
-
1

EMBD, s1

GMRF, s1

466x466

30

LINUX, 800 MHZ

1

LINUX, 800 MHZ

EMBD, s2

GMRF, s2

466x466

10

LINUX, 800 MHZ

1

LINUX, 800 MHZ




As example, the data ingestion of the Landsat images for the Mozambique site (438 tiles) requ
ired

1,5 days of computation using a 6 CPU SUN computer. The data ingestion of the ERS
-
1 images of
Mozambique (438 tiles) needed 0.25 day using a 8 CPU Linux based computer.


In condition of the today technology which makes available at low costs cluster
s of 100s of CPU of
more than 2GHz speed, the data ingestion is not a critical task. The algorithms for data ingestion do
not require parallelisation, data sets are distributed in MDSI strategy.










KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
24

of
33


3.6

Evaluation of the Human Machine Interface



The KIM sys
tem is based on Human Centered concepts, the implemented mining functions require
interactive operation, a man
-
machine dialog. Thus, the KIM response should be “real time” relative
to the reaction of the operator.


For evaluation we compare the applet loa
ding time and the duration of an interactive learning
session. The loading average time of the applet (
Figure 12
) is 7 seconds, relative to the average
duration of the interactive learning (
Figure 13
) 1 minute.


Applet loading time (test week)
0
10
20
30
40
50
60
0-3
4-5
6-10
11-15
16-30
31-60
61-120
> 120
Loading time (s)
% of sessions

Figure 12
:
The histogram of the applet loading time. (during
the evaluation period)



Interactive learning duration
0
5
10
15
20
25
30
35
40
0-10
11-15
16-30
31-60
61-120
121-
180
181-
240
> 240
Single step duration (s)
% of sessions

Figure 13
:
The histogram of the interactive learning time.
(during the evaluation period)


The results is satisfactory, the ratio is maintained als
o in over
-
the
-
net operation for normal network
speed.









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
25

of
33


4.

EVALUATION PROCEDURE


The evaluation of the KIM system required special organization and development of adequate tools.

The diagram in
Figure 14

presents the architecture used for the evaluation p
rocedure.


For the analysis and quantification of the objective performance of the system, was implemented a
tool to trace the user interaction, and a set of tools for statistical analysis of the results of the traced
parameters. The objective evaluation w
as based on measures of:



classification error for the quality of the interactive training



information transfer for the quality of the learning of the semantic labels



complexity for the man
-
machine dialog (HMI).


During the KIM system operation the report o
f objective evaluation measures was generated by of
-
line processing of the user tracing information.


At the subjective level, the user (evaluator) was asked to rank qualitatively the degree of satisfaction
after the operation of the system. The evaluatio
n was formalized as a questionary.













Figure 14
:
KIM system architecture for evaluation purpose


The evaluation was performed by image analysts from EU
-
SC, scientists from NERSC, and
scientific and technical staff of ESRIN.

The results of the ev
aluation have been analyzed by merging the two reports, the objective and the
subjective one. An example is shown in Table 3.




KIM

Server

User

Tracing

Statistical

Analysis

Analysis

Team

(DLR)



-

classification

accuracy

-

information

measure

-

complexity of HMI

Objective


Measurements




Obj. Eval. Subj.
Eval.

Subjective


Evaluation

Evaluation

Team

(ACS)

Eval.
Protocol




Measurements

KIM

Client








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
26

of
33


Table 3 :
Example of joint analysis of the objective and subjective evaluation reports.


During the evaluation period have be
en defined




31 labels for Landsat data



7 labels for ERS data



10 labels multi
-
mission (ERS and Landsat)


The analysis of the objective and subjective criteria resulted in the evaluation of the defined labels
as follows:




10 % very good



60 % good



20 %
acceptable



10 % not satisfactory


For the overall evaluation should be taken in consideration the very high complexity of the
Mozambique and Nepal images.


The 31 labels defined for the site Mozambique, have been assigned to ~75% of the images in the
archive. In
Figure 15

is presented the histogram of appearances of images in the result gallery.


too complex
label

learning failed


x





x


spectral

texture


farm



-


x

x

+



0


x


-

needs analysis:

strong primitive
features, thus

fast learning



spectral

texture


cloud

good label


x

spectral

texture

river

+

Remarks

Subjective

Evaluation

Objective
Measurements

Models

Label name








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
27

of
33



Figure 15
:
Histogram of archive coverage with the 31
semantic labels.


This is a measure quantifying the semantic coding of
the information content in the archive.



5. FUTURE ISSUES

The KIM prototype is designed to manage a huge volume of data containing high complexity
information. However, due to technological constrains, there are limitations, memory size, speed of
computat
ion, etc., which leaded us to propose a scalable architecture. The scalability refers the data
reduction rate vs. the increase of the extracted image detail. Three levels have been demonstrated as
presented in
Figure 16

( also in
Table 2
),






Figure 16
:
Diagram of the scalable system for information retrieval from very large image archives. Due to the
problem complexity the system scales
-
down the amount of data to be analyzed such that the detail of extracted
information can be increased.









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
28

of
33


Content Based Image Retrieval (CBIR)

The CBIR is based on utilization of semantic queries. Thus, CBIR enables an operator to see into a
large volume of high complexity images, base before actually retrieving data. The metho
d aims at
obtaining a better selectivity of the query process for different subjective user requests, in
conjunction with the use of queries based on “classical” meta
-
data.

Data/Information mining

Further refinement of the information content in a not so l
arge data set, can be obtained by data or
information mining. The goal is to explore the information content of the images and decide which
ones are relevant for the user's application. The tool we propose is mainly based on interactive
decision using data

visualization, e.g. probabilistic image retrieval integrated with interactive
learning and image classification.

Scene understanding

The last level of highest detail information extraction is scene understanding. Scene understanding
is the attempt to extr
act information about the observed scene, derive knowledge, interpret or
understand the structures and objects. The implemented algorithms allow accurate image
classification by fusion of information extracted from one data sets or multi
-
sensor data. The
i
nformation extracted represent pixel or structural/contextual characterization.


5.1

How and where to use KIM

The KIM prototype, depending on the scale of the system, can be integrated as:



Component of satellite ground segment systems



Tools to explore ve
ry large
historical

archives



Tools to explore distributed archives



Tools to discover and understand high complexity data



Tools to understand high complexity scenes



Integration as semantic WEB technology



. . . other fields: medical, biometrics, digital

pictures, etc.


Example 1: Feasibility of KIM integration as module of a satellite ground segment.

The data ingestion engine using a cluster of 100 CPU can process the primitive feature extraction
and the generation of the catalogue entries in pipeline
with a SAR processor.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
29

of
33


The required data storage, for a scenario using a
distilation

(compressinon factor) of 100, which
guaranties a good image content preservation, enables the on
-
line exploration of more than 10 TB of
data.

The access very large volum
es of data, the system can be operated in two steps 1. a classical query
based on meta
-
data and combined with semantic data grouping, and 2. information mining for the
results of the first query.




A similar system integration can be designed for ex
ploration of very large historical archives. A
simple approach is to explore the quick look images.

Candidate sensor data are:



ERS 1/2



Landsat 5/7



ENVISAT ASAR or MERIS









KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
30

of
33



Example 2: Feasibility of KIM integration as tool for exploration of meter resolutio
n images and
image archives.

Many of the new generation of imaging sensors, both optical and SAR, e.g. Ikonos, SPOT 5, or
TerraSAR, RADARSAT 2, provide or will provide images with resolution in the range of 1 m,
covering very large scenes, thus resulting i
n high complexity images of typical size 20 000 x 20 000
pixels.

The visual inspection of these images is a difficult task. KIM can be used to help an operator to find
similar structures, to
spot

on regions of interests. Thus KIM can become a tool for visu
al image
mining, an
extended retina
.


Example 3: KIM

in Image Information Mining for Medical Diagnosis Help (with contributions
of Prof. Roberto Murri and Ms. Clara Pasquali, University of Camerino)


The concepts and KIM system it self is not limited t
o use for exploitation of EO data. It can be used
on broader class of images and other multidimensional data. This paragraph presents preliminary
study of using the Image Information Mining concepts for the early diagnosis of melanoma, a tumor
which origin
ates from melanocytes, whose incidence has been increasing during the last years.


The diagnosis of this skin tumor in its early phase of development is indispensable to improve
prognosis and decrease mortality
. In fact, if melanoma is excised during the first phase of horizontal
growth, complete recovery can be achieved; on the contrary, as vertical thi
ckness increases, the
probability of presence of metastases heightens, and prognosis worsens. Moreover, a high level of
accuracy is necessary not only to excise precociously malignant lesions, but also to avoid
unnecessary excision of benign ones.


The im
ages were acquired by the SkinLab system in the laboratory of the Dermatological Unit of
INRCA, Ancona, (Italy) to ensures epiluminescence conditions, optimal illumination and removal
of all undesirable reflections. Images are acquired in RGB color space,
with a resolution of 1524 x
1012 pixels, 24 bits per pixel. The SkinLab is composed of 4 parts: lighting apparatus, head, digital
camera, and central processing unit. In this system light undergoes a double polarization: the first
one before illuminating
the skin and the second one before being caught by the CCD sensors.
In this
way all undesirable reflections are eliminated.

The analysis and interpretation of dermatological structures is a high complexity process, difficult to
be automatized. There are at

least two classes of problems: objective, i. e. signal processing and
subjective, i. e. regarding the interpreter conjecture. Image analysis of skin structures is hindered

by
many factors: illumination and pigmentation variability, large diversity of text
ure like patterns, and
the ambiguities of “texture” parameter matching with the visual recognition of a certain “pattern”.
Thus in the diagnosis the interpreter integrates important domain knowledge, which is difficult to be

automatically realized.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
31

of
33


A set o
f images was processed and analyses using interactive learning and probabilistic search
functions. An example is presented in
Figure 17
.




Figure 17
:
Probabilistic retrieval for the “pigment network” label


The preliminary results demonstrated th
e power of classification using fusion of color and texture
information, and the usefulness of generalization for understanding of the nature of the structures in
the image.





5.2

Further development

The development and evaluat
ion of the KIM system resulted in the identification of topics which
shall be subject of further optimization, and could result in a new generation of systems for
knowledge based exploration of EO data, and more, new concepts for synergetic understanding
of
scientific data.

The diagram in the
Figure 18

presents the topics of relevance for a first further up
-
grade of the KIM
system.








KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
32

of
33



Figure 18
:
The KIM schematic architecture with identification of modules or methods object of up
-
grade for
designing a new

version of knowledge based EO data exploration system.



In summary, several topics are of immediate interest for further R&D:



a better geometrical description



enhanced multiple model selection and search



R/D

analysis of the information content



automati
c geometric and radiometric registration



KDD formalism for joint exploration of structured and un
-
structured data




. . . more theory for processing and understanding high complexity infromation . .

.











KIM Ref.: ESA
-
KIM
-
EXS
-
1.1

Project Executive Summary

Release: 1.0

Date:

12.11.02







Page
33

of
33


6. CONCLUSIONS

The technologies for knowledge
-
driv
en image information mining are reaching a sufficient level of
maturity for their integration into commercial products, as has been demonstrated here for a variety
of remote
-
sensing applications. This opens new perspectives and offers huge potential for
c
orrelating the information extracted from remote
-
sensing images with the goals of specific
applications.

These technologies shift the focus from data to information, meeting user needs, promoting
scientific investigations, and supporting the growth of the
value
-
adding industry, service providers
and market, by permitting the provision of new services based on information and knowledge. They
will also profoundly affect developments in fields like space exploration, industrial processes,
exploitation of resou
rces, media, etc. The KIM prototype has demonstrated that:



the results of advanced and very highly complex algorithms for feature extraction can be
made available to a large and diverse user community



the users, who can access the image information co
ntent based on their specific background
knowledge, can interactively store the meta
-
information and knowledge


A new paradigm for the interaction with and exploitation of EO archives can be implemented,
paving the way for much easier access to and much w
ider use of EO data and services