Case Study Aircraft Service Monitoring

snufflevoicelessInternet and Web Development

Oct 22, 2013 (4 years and 22 days ago)

69 views

Case Study Aircraft Service Monitoring

The Problem

Recognise various service
activities of an aircraft turnaround
regarding

• completeness

• temporal constraint satisfaction


Examples
:

-

Arrival preparation


GPU
-
Positioned
-
in
-
GPU
-
Zone


Put
-
Down
-
Chocks

-

Aircraft
-
Arrival

-

Passenger
-
Stairs
-
Positioned

. . .


SCENIOR System Structure (1)

Annotated

Turnaround

Sequences

Hand
-
crafting

OWL + SWRL

Conceptual
Knowledge

Base

Probabilistic

Aggregate
Descriptions

Java

Constraint

Solver

Jess

Rule

Engine

BCH

Inference

Engine

Interpretations

Primitive Events

Jess

Conceptual

Knowledge Base

Hypotheses

Graphs

Temporal

Constraint Net

Templates

Probabilistic

Aggregate

Models

Converter

SCENIOR

using handcrafted models

SCENIOR System Structure (2)

Annotated

Turnaround

Sequences

OWL + SWRL

Conceptual
Knowledge

Base

Probabilistic

Aggregate
Descriptions

Java

Constraint

Solver

Jess

Rule

Engine

BCH

Inference

Engine

Interpretations

Primitive Events

Jess

Conceptual

Knowledge Base

Hypotheses

Graphs

Temporal

Constraint Net

Templates

Probabilistic

Aggregate

Models

Converter

SCENIOR

Learning

using learned models

Representing Activity Concepts in a

Formal Ontology

An ontology is an explicit representation of a conceptualisation.


First attempt in Scene Interpretation research to use the standardised
web ontology language OWL for activity concepts.


OWL variants:

OWL
-
Lite

OWL
-
DL


corresponds to a Description Logic

OWL
-
Full



As shown in the section "Scene Interpretation with Description
Logics", the biggest problem is to represent constraints between
parts of an aggregate.

Basic Structure for Service Activity Ontology

OWL
-
Thing

Conceptual Object

Primitive States

Composite Events

Physical Object

Mobile

Static

Vehicle

Person

Tanker

GPU

Vehicle
-

Inside
-
Zone

Vehicle
-

Outside
-
Zone

Tanker
-

Inside
-
Zone

Refueling

Baggage Loading

Baggage Unloading

Arrival Preparation

Upper model

Domain model


sub
-
class

relation


Compositional Hierarchy for

Turnaround Activities (1)

Departure

Arrival
-
Preparation

Airplane
-
Enters
-
ERA

Stop
-
Beacon

GPU
-
Enters
-
GPU
-
Zone

Drop
-
Chocks

Person
-
Goes
-
From
-
GPU
-
Door
-
To
-
GPU
-
Access
-
Area

Chocks
-
Appear

Turnaround

Arrival

Services

Compositional Hierarchy for

Turnaround Activities (2)

Turnaround

Arrival

Services

Departure

Refuelling

Waste
-
Removal

Air
-
Conditioning

Replace
-

Drinking
-
Water

Catering

Unloading

Loading

Unload
-
AFT

Unload
-
FWD

Unload
-
AFT
-
Belt

Trolley
-
Enters
-
AFT
-
Zone

Belt
-
Enters
-
AFT
-
Bay

Belt
-
Leaves
-
AFT
-
Bay

Unload
-
FWD
-
Belt

Belt
-
Enters
-
FWD
-
Bay

Belt
-
Leaves
-
FWD
-
Bay

Trolley
-
Enters
-
AFT
-
Zone

Compositional Hierarchy for

Turnaround Activities (3)

Turnaround

Arrival

Services

Departure

Refuelling

Waste
-
Removal

Air
-
Conditioning

Replace
-

Drinking
-
Water

Catering

Unloading

Loading

Load
-
AFT

Load
-
FWD

Load
-
AFT
-
Belt

Trolley
-
Enters
-
AFT
-
Zone

Belt
-
Enters
-
AFT
-
Bay

Belt
-
Leaves
-
AFT
-
Bay

Load
-
FWD
-
Belt

Belt
-
Enters
-
FWD
-
Bay

Belt
-
Leaves
-
FWD
-
Bay

Trolley
-
Enters
-
AFT
-
Zone

Compositional Hierarchy for

Turnaround Activities (4)

Tanker
-
Enters
-

Tanking
-
Zone

Tanker
-
Leaves
-

TankingZone

Pumping
-
Operation

Catering
-
Van
-
Raised

Catering
-
Van
-

Leaves
-

Catering
-
Zone

Turnaround

Arrival

Services

Departure

Refuelling

Replace
-

Drinking
-
Water

Catering

Unloading

Loading

Waste
-
Removal

Air
-
Conditioning

Catering
-
Van
-

Enters
-

Catering
-
Zone

Compositional Hierarchy for

Turnaround Activities (5)

Turnaround

Arrival

Services

Departure

Start
-
Beacon

Pushback

Structure of "Refuelling" Concept






Refuelling

Tanker
-
Enters
-

Fuel
-
Access
-
Area

Do
-
Refuel

Tanker
-
Leaves
-

Fuel
-
Access
-
Area

Tanker
-
Outside
-

Fuel
-
Access
-
Area

Tanker
-
Inside
-

Fuel
-
Access
-
Area

Tanker
-
Inside
-

Fuel
-
Access
-
Area

Tanker
-
Outside
-

Fuel
-
Access
-
Area

Conceptual constraints (modelled with SWRL rules)


equivalence constraints


temporal constraints


(e.g. TEFAA.end < DR.start < DR.end < TLFAA.start)


=

Protégé Representation of

OWL Concept for "Refuelling"



Typical SWRL Rule for Service Activity

domain:Refuelling(?v:refuelling)


捯n獴s慩nt猺h慳
-
獴慲t
-
ti浥⠿v:r敦eelling, ?v:獴慲琩t


捯n獴s慩nt猺h慳
-
晩ni獨
-
瑩浥m?v:r敦e敬ling, ?v:晩ni獨⤠


upp敲:h慳
-
p慲t
-
A⠿v:r敦e敬ling, ?v:vp⤠


upper:has
-
part
-
B(?v:refuelling, ?v:vpd)


constraints:has
-
start
-
time(?v:vpd, ?v:vpd
-
start)


捯n獴s慩nt猺h慳
-
晩ni獨
-
瑩浥m?v:vpd, ?v:vpd
-
晩ni獨⤠


upp敲:h慳
-
p慲t
-
C⠿v:r敦e敬ling, ?v:vlz⤠


upp敲:h慳
-
慧敮t⠿v:vp, ?v:愱⤠


upp敲:h慳
-
慧敮t⠿v:vpd, ?v:愲⤠


upp敲:h慳
-
慧敮t⠿v:vlz, ?v:愳⤠


upper:has
-
location(?v:vp, ?v:l1)


upper:has
-
location(?v:vpd, ?v:l2)


upp敲:h慳
-
lo捡瑩on(?v:vlz, ?v:l㌩3


→ swrlb:equal(?v:a1, ?v:a2)


swrlb:equal(?v:a2, ?v:a3)


獷rlb:敱u慬(?v:lㄬ ?v:l㈩2


獷rlb:敱u慬(?v:l㈬ ?v:l㌩3


constraints:same
-
start(?v:refuelling, ?v:vp)


constraints:same
-
finish(?v:refuelling, ?v:vlz)


慬l敮:b敦ere⠿v:vp, ?v:vpd⤠


慬l敮:b敦ere⠿v:vpd, ?v:vlz⤠


捯n獴s慩nt猺浩n
-
b敦or攨?v:晩ni獨, ?v:獴慲琬
-
㌷㈰〰3⤠


捯n獴s慩nt猺浩n
-
b敦or攨?v:vpd
-
晩nish, ?v:vpd
-
獴慲琬
-
㌶〰〰〩

"premise" of rule
serves for
establishing
variables

"consequence" of
rule specifies
equality and
temporal
constraints

Generating JESS Interpretation Rules

from an OWL Ontology

Three kinds of rules are used in SCENIOR:



Aggregate instantiation (moving up a compositional hierarchy)



Aggregate expansion (moving down a compositional hierarchy)



Instance specialisation (moving down a taxonomical hierarchy)

Rules are applied to
Hypotheses Structures

(HSs) generated initially from the
ontology:



A HS represents an expected aggregate hierarchy as an independent
interpretation goal



A partially instantiated HS provides expectations about missing
evidence



A recognised HS may provide input for other HSs as a "high
-
level
evidence"




Instance merging (Rule 4: unifying instances obtained separately) is
dispensable
in SCENIOR because of Hypotheses Structures.

Hypotheses Structure "Arrival Preparation"

properties interlinked by
identity constraints

Arrival
-
Preparation

GPU
-
Enters
-

GPU
-
Zone

GPU
-
Stopped
-

Inside
-
GPU
-
Zone

Drop
-
Chocks

GPU

GPU
-
Zone

Evidence Assignment Rule (1)

(defrule GPU
-
Enters
-
GPU
-
Zone_ea_rule


?e
-
id <
-

(GPU
-
Enters
-
GPU
-
Zone



(name ?gegz_17)



(status evidence))


?h
-
id <
-

(GPU
-
Enters
-
GPU
-
Zone



(name ?gegz_h)



(status ?status_1))


(test (or (eq ?status_1 hypothesised)


(eq ?status_1
hallucinated)))


;; check temporal constraints


=>


(modify ?e
-
id (status assigned))


(modify ?h
-
id (status instantiated))


;; update temporal constraint net

)

Evidence Assignment Rule (2)

Arrival
-
Preparation

GPU
-
Enters
-

GPU
-
Zone

GPU
-
Stopped
-

Inside
-
GPU
-
Zone

Drop
-
Chocks

gpu
-
enters
-

gpu
-
zone_14

Status changed
to "instantiated"
as consequence
of rule

Aggregate
-
Instatiantion
-
Rule (1)

(defrule Arrival
-
preparation_ai_rule







?h
-
id <
-

(Arrival
-
Preparation (name ?ap_h)





(status hypothesised)




(has
-
part
-
1 p1)




(has
-
part
-
2 p2)




(has
-
part
-
3 p3))


(GPU
-
Enters
-
GPU
-
Zone (name ?p1)




(status ?status_1))


(test (or (eq ?status_1 instantiated) (eq ?status_1
hallucinated)))


(GPU
-
Stopped
-
Inside
-
GPU
-
Zone (name ?p2)




(status ?status_2))


(test (or (eq ?status_2 instantiated) (eq ?status_2
hallucinated)))


(Drop
-
Chocks (name ?p3)




(status ?status_3))


(test (or (eq ?status_3 instantiated) (eq ?status_3
hallucinated)))


=>


(modify ?h
-
id (status instantiated)))

Aggregate
-
Instatiantion
-
Rule (2)

Arrival
-
Preparation

GPU
-
Enters
-

GPU
-
Zone

GPU
-
Stopped
-

Inside
-
GPU
-
Zone

Drop
-
Chocks

gpu
-
enters
-

gpu
-
zone_14

gpu
-
stopped
-

inside
-
gpu
-
zone_7

gpu
-
enters
-

gpu
-
zone_14

status changed to
"instantiated" as
consequence of rule

Specialisation Rule (1)

(defrule GPU
-
Enters
-
GPU
-
Zone_s_rule


?e
-
id <
-

(Vehicle
-
Enters
-
Zone


(name ?vez_14)


(status evidence)


(has
-
agent ?a1)


(has
-
location ?l1))


(GPU (name ?a1))


(GPU
-
Zone (name ?l1))


(not (GPU
-
Enters
-
GPU
-
Zone (name ?vez_14)))


=>


(retract ?e
-
id))



(assert (GPU
-
Enters
-
GPU
-
Zone


(name ?vez_14)


(status evidence)


(has
-
agent ?a1)


(has
-
location ?l1)))

Specialisation Rule (2)

Arrival
-
Preparation

GPU
-
Enters
-

GPU
-
Zone

GPU
-
Stopped
-

Inside
-
GPU
-
Zone

Drop
-
Chocks

vehicle
-
enters
--
zone_14

[GPU
-
Enters
-
GPU
-
Zone]

vehicle
-
enters
--
zone_14

[Vehicle
-
Enters
-
Zone]

Activity class is specialised as
consequence of rule

(e.g. if high
-
level context
provides additional evidence)


Note
:

This could have been achieved
by a DL reasoner, but we are in
a JESS rule system now ...

Interpretation Process



Initialisation phase:


JESS conceptual knowledge base is generated automatically


Separate interpretation thread is created for every Hypotheses
Structure with temporal constraint net (TCN) and independent
JESS engine



Working phase:


System receives primitive states and events as input and feeds

these as WMEs to every interpretation thread


Rules are applied


New aggregates are instantiated


High
-
level evidence is offered to other threads


Multiple Assignments

A

B

C

D

E

D

F

d

Clutter



Evidence may trigger several evidence assigment rules and cause
several new interpretation threads



Each evidence is a potential "false positive", hence it is also
assigned to "Clutter" (noise)

Beam Search

Threads



SCENIOR can compute up to 100 threads (different interpretation
possibilities) in parallel in real time



If the maximum number of threads is reached, less promising
threads are discarded based on a probabilistic preference measure
(ranking)

Using Probabilistic Models

for Ranking

General format of probabilistic scene model:

alternatives

m = 1 ... M

hidden

variables

observable

variables

Probability that model m has generated evidence
e
:

statistically
independent
noise

Bayesian Compositional Hierarchy

Turnaround

Arrival

Services

Departure

Aggregate JPD: P
Turnaround
(A B
1
C
2

B
2
C
3
B
3

)





=> P'
Turnaround
(B
1
C
2

B
2
C
3
B
3

| A)


T

Turnaround

Arrival

Services

Departure

A

B1

B2

B3

C2

C3

Scene JPD is product of aggregate JPDs (as in Bayesian Networks)

Requirements for a

Bayesian Compositional Hierarchy



Aggregate properties do not
depend on details below the
part properties.



Part properties depend only
on the properties of the
corresponding parent
aggregate.



Parts of different aggregates
are statistically independent
given their parent aggregates.

When is a compositional hierarchy a "Bayesian Compositional
Hierarchy" (BCH)?

Probability changes may be propagated along
tree
-
shaped hierarchy.

Change Propagation

A

Crisp evidence
e

for
B
i

is modelled as P(
B
i
=
e

) = 1 and P(
B
i

e

) = 0.

B
1
...
B
i

...
B
N

P(A
B
1

...
B
N
)



Propagating up
:


P(
B
i
) => P
´
(
B
i
)

P
´
(A
B
1

...
B
N
) = P(
A
1

...
A
N

B
) P
´
(
B
i
) / P(
B
i
)

followed by marginalizations


Propagating down
:

P(A) => P
´
(A)

P
´
(A
B
1

...
B
N
) = P(A
B
1

...
B
N
) P
´
(A) / P(A)

followed by marginalizations

Use of BCH for Estimation



Enter begin or end of events




Propagate change throughout BCH




Estimate non
-
instantiated


temporal variables

begin of

Pumping
-
Operation

at
T = 26

obtain dynamic priors
(context
-
dependent)

SCENIOR Interpretation Interface

hypothesised aggregate

instantiated
aggregate

evidence

Alternative Interpretation Threads

Number of interpretation threads in the course of an interpretation

(Sequence 6 of 300 recorded services in Project Co
-
Friend)

Conclusions



High
-
level interpretation of activities can be performed with
an interpretation system generated automatically from an
ontology



Beam search controlled by probabilistic ranking allows to
follow a large number of parallel interpretations



SCENIOR performed well, but the input provided by low
-
level
image analysis and tracking was very noisy


(missing and misclassified evidence)



Success rate was ca. 75%, too little for practical employment
of the system at Blagnac Airport