Applications of Neural Networks for Intelligent Data Interpretation in UAVs: Health Monitoring and Target Recognition

sciencediscussionΤεχνίτη Νοημοσύνη και Ρομποτική

20 Οκτ 2013 (πριν από 3 χρόνια και 8 μήνες)

116 εμφανίσεις



RTO
-
MP
-
SCI
-
202

27

-

1

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


Applications of Neural Networks for Intelligent Data
Interpretation in
UAVs: Health M
onitoring and Target Recognition


Dr. Peter Knappe, Richard Seitz, Christoph Stahl, Claudia Baisl, Tamer Koban

European Aeronautic Defence and Space Company (EADS)

85077 M
anching, Germany

peter.knappe@eads.com

NEURAL NETWORK APPLI
CATIONS IN UAVS

Every military company developing

uninhabited aerial vehicles (U
A
V
s) is confronted with the same
challenge: More and more pilot's capabilities shall be carried over into the UAV. T
herefore additional
functionalities, representing the human sensing and assessment capabilities, have to be implemented as
software solutions.

Artifical neural networks
(ANNs)
provide a learning based approach to classification
tasks building up a kind of
"experience" based on training data.
ANNs
of backpropagation type
are
applied with specific extensions on optimization and selection of training data sets. For typical avionics
applications the current status quo is demonstrated and discussed. Als
o the pot
ential for UAVs

is
assessed.

1.0
INTRODUCTION

A future trend in military aircraft will be the broad appearance of UAVs in reconnaissance and
surveillance roles, but also first UAVs in combat
missions

are on the hor
izon. It can be assumed that these

UAVs wi
ll not be operated remote
ly

controlled only but will possess autonomous capabilities.
Autonomy
in this context should be defined as the capability to sense the environment, make
decisions

and react
without human inter
vention
.

To gain high performance of th
is chain every link should
deliver its output in a quality as high as possible.
For the sensing link this means that the output should be delivered in a very condensed and "high
-
level"
way. What kind of information this could be can be seen by messages sen
t by a human pilot. He will not
declare that

there are some dark pixels in the
neighbourhood

of some lighter ones in a certain arrangement
but he will st
ate the message that there is a

tank in the image of a certain type in a certain position. So in a
cert
ain sense the pilot
'
s capabilities have to be transferred to the sensing system of the UAV. For example
according

to the specific problem area the results can be detected, classified or identified ground objects or
interpreted error messages.

In many cases

the delivery of highly condensed information requires a classification process. EADS
Military Air Systems (
MAS
)

has built up over 10 year
s

of
experience in the application of ANNs
for

classification tas
ks within avionics applications, mainly based on the
classical backpropagation networks.

Reasons to stay with the ANNs were the benefits of the learning based approach,
the capability of
generalisation


and of course the promising results. During this time the main effort was set on ANN
"engineering", e.g.
to set up efficient development environments, to develop strategies for optimal training
data sets or to impro
ve the optimization algorithms.

In this paper
exemplary

functions will be presented, all of them applying
ANNs

and also all of them with
the poten
tial
for

a future UAV avionics component.
These functions

demonstrate the state
-
of
-
the art of
ANN

applications in military aircraft avionics and are int
ended to give an impression of
possible
application areas of intellige
nt data interpretation in UAVs.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

2

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


2.
0
THEORETICAL BACKGROU
ND

2.1
Basics of Artificial Neural Networks

ANNs

belong to the field of research concerning artificial intelligence. They can be described as
mathematical models
based on
the information processing within a human brain. Their aim is n
ot to
recreate the complex human brain consisting of 10 to 100 billions of neurons
[1]

but to adopt some basic
functionalities and workflows.

EADS MAS does not concentrate on developing ANNs from scratch but e.g. on the compilation of
efficient training da
ta sets, the training of ANNs itself and the implementation of ANNs in major process
chains as described in chapter

2.2
. Because of that, detailed basics on ANNs are not discussed here but can
be found in

[2]
, for example. For all demonstrated
MAS applicat
ions, feedforward multilayer perceptrons
(MLP
s
) with the backpropagation learning algorithm are applied.

Working with ANNs one has to run through a

learning phase and an operation

phase. The learning process
of ANNs can be compared with the one of humans l
earning vocabulary, for example. Often a set of
vocabulary is
connected to
a certain situation like ordering a meal in a restaurant. Learning this specific
vocabulary requires the repetition of the meaning in the one language and the translation into the o
ther
language. Of course, the amount of repetitions heavily depends on the
difficulty

of the vocabulary and the
person's powers of comprehension. When the situation occurs for which the person prepared itself the
learned vocabulary can be recalled and appl
ied.

Analyzing this example of human learning one can extract several characteristics that can be also applied
to the learning of ANNs. These characteristics shall be described in the following with the help of
Figure

1
.


Figure

1
: Learning and operation
phase of artificial neural networks

In the operation phase the ANN takes input data (in the vocabulary example this would be e.g. an English
word) and gives back output data (e.g. the German translation). But before the operational application of
the netwo
rk it has to be adapted to the problem in a learning phase. The basis for the learning phase is a
database with so
-
called training data. These data consist of input data and corresponding output data (like
a dictionary in the vocabulary example). The train
ing data has to be representative and typical for the
problem and task respectively. During the learning phase the ANN processes through the training data
numerous times to learn the correlations between the input and the desired output data.
The advantage

of
this proceeding is that the problem needs not to be described explicitly before.

During the learning process
the internal parameters of the ANN are adapted by a learning algorithm to optimize the ANN’s
capabilities.
This kind of learning is called supe
rvised learning and is detailed in

Figure

2.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

3

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED



Figure

2
: Training of artificial neural networks

At the beginning of the training, the parameters of the ANN are usually initialized with random values.
The term parameters covers e.g. detailed information abo
ut the
structure

of the ANN or detailed properties
concerning the concrete training algorithm. More information about these parameters can be found in

[2]
.
In the first iteration, a set of training data is
fed

into the ANN. The ANN delivers an according ou
tput that
is denoted as actual output in

Figure

2
. The supervised learning approach compares the actual output of
the ANN with the
desired

output that exists for every example of the training set. Then the deviation
or
rather the
error between the actual o
utput and the
desired

output is determined. According to the error, the
parameters of the ANN are adjusted. This adjustment can be compared with the human approach of
memorizing and is because of that referred to as learning.

The paragraph above described
the process of one training epoch. Usually several
hundred

training epochs
are necessary with the aim to minimize the error between the actual output and the
desired

output.
Depending on the complexity of the task that the ANN has to fulfil

and the availab
le computing power
,
its

training can last from several hours up to several days.

After finishing the learning phase depicted in

Figure

1
,

the trained ANN
and the so
-
called weights and
design parameters respectively
are stored in a classifier database and c
an be embedded in frameworks to
form a concrete ANN application. In the operation phase, the ANN applications have the task to deliver an
output according to the actual input. In contrast to the long lasting learning phase this can happen in real
-
time as t
he ANN itself is not changed anymore.

2.2
Algorithm for
Automatic Target Recognition

The MAS
automatic target recognition (
ATR
)

approach is also based on ANNs as classifiers, but this
classification part is embedded in a
patented
image processing chain inc
luding pre
-

and post
-
processing
steps

as sketched in
Figure

3
.


Again this approach leads to two phases:
The learning phase
where

the classifiers
and parameters
are
adapted to detect the

target objects in the training
images and the operation or applicatio
n phase during
which the classifiers are changed no more but merely detect the target objects in the incoming images or
video streams.

The process chain
in Figure

3
consists of four
main steps that are applied during the learning and operation
phase.

A
t

th
e
first step the image is pre
-
processed. Here Gaussian and Laplace pyramids are
calculated
at
different resolutions to filter the image. Depending on the filter the focus of the subsequent processing is
on the lower or higher
spatial
frequencies of the ima
ge. In fact three filtered images with specific
resolutions are generated on the base of the original image. This multi
-
resolution approach makes the
Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

4

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


detection more robust against small changes in the size of the target objects. Furthermore a region of
int
erest (ROI) of the image is selected in which only regions with sufficient contrast remain. Thi
s method
speeds up the classification

as the target objects are not searched in all imag
e regions.


Figure
3
: The MAS ATR process chain

The second step is the

classification step

performed
on

the images
of the considered

pyramid levels
and
resulting in
a rating for each
p
ixel within the ROI

by the ANN
classifier
s
.
The input for the
ANN

classifiers is a vector formed by the pixels of a window with specific size
around the actually rated pixel
.
At the end of this step three

ratings for each
p
ixel in the ROI are available
.

In the third step, the so called
f
usion
, the three ratings are merged to one
single value for each
p
i
xel. Due
to

the usage of different pyramid

levels this fusion reduces the false alarms.

In the last step a post
-
processing of the image is conducted on the base of the merged ratings of the
f
usion
step. First of all
a threshold is applied to
the merged ratings so that only the ratings above a thre
shold
value remain. Until now all processing concerned the
p
ixel level of
the target objects. In the current
processing step the transition from
p
ixel level to object level is carried out. In the last step of the post
-
processing the connected clusters in a

specific relevant range

of size are searched.
Each remaining cluster

is interpreted as a
detection of the target object.

For the learning phase a training database is necessary which contains a representative compilation of
example ima
gery with target obj
ects and the so called ground truth
like type, position and shape. For the
success of the training the compilation of an efficient training data set is essential. Therefore MAS
administrates an image
database

with additional context data. For each scenario

and target object type a
classifier
database

is generated which contains the adapted classifiers. The already trained classifiers can
be taken as an initial point for further adaptations of the classifiers, if the scenario for a mission changes.
This appr
oach speeds up the training phase for new scenarios and missions.

One main advanta
g
e

of the described approach is seen in the fact that
no explicit rule has to be found by
the user to classify the target objects. The classifiers are adapted by an algorith
m automatically on the base
of the training data

set during the learning phase.

Also important for ATR is a mature assessment procedure and a test and validation concept. Further more
integration aspects of the operation phase in the application
are

a chal
lenge as many applications require
real
-
time implementations and commercial of the shelf hardware and software. Therefore MAS optimizes
the core software to ensure the usage of multi
-
core processing by parallelizing the code
.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

5

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED


3.0
EADS MAS

APPLICATIONS

3.1
Health Monitoring

Fulfilling extensive mission tasks an UAV is fe
atured with lots of sensors, on
board computing devices and
communication equipment. The complexity of UAVs additionally increases with the fact that the
subsystems of an UAV are often impleme
nted by different suppliers. Beside the effort to integrate all the
subsystems to a functional UAV, there is also the need to monitor this complex network of subsystems.
Because of that,

one important aspect is the health monitoring of the subsystems
.

In e
very mission, incidents can occur after which a subsystem can no longer operate in its normal
modality. In this cases status messages or so
-
called arisings are generated by the subsystem containing
information about the incident and the status of the subsy
stem. The immediate evaluation of the arisings is
very important, as incidents can of course be
of certain criticality
.

Arisings that are sent by the subsystems are collected at a central location. An automated classification
functionality for the subseque
nt evaluation of the arisings is in development and is described in the
following paragraphs.
This concept related to
a modern
fighter

aircraft
on whose development EADS
MAS is participating,
is one suggestion

for handling arisings in UAVs.


Figure
1
: Concept of automated classification

The main task of the automated classification functionality is to distinguish between arisings pointing to
real failures and arisings announcing so called bogus failures.
Figure

4

visualizes this ta
sk as unclassified
arisings are inputted to the automated classification functionality that provides the arisings with a
proposed classification.

In general, arisings include information resulting from built in tests of the subsystem. Built in tests are
ex
ecuted in the power up phase of
an UAV

but also during
the normal operation phase. U
sually several
subsy
stems are connected to work on

a
certain functionality
. Because of the complexity of those
connected systems
,
situations can occur that
are misleadingly

interpreted as failure

conditions

by
subsystems
.
Therefore an arising is generated by the concerned subsystems although
the situation resulted
from

an inappropriate behaviour

of subsystems
.

For example,
each subsystem checks the availability of
necessary
partner subsystems during the power up phase. Of course, it is not possible to boot all
subsystems at the same time. Because of that, an already booted subsystem complains about missing
partner subsystem
s

sending an arising. From its point of view, the sub
system complains rightly but from
the overall view, these arisings do not point to real failures and have to be rejected or classified as bogus
Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

6

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


failures respectively. If in contrast the same subsystem diagnoses a sudden missing partner subsystem
during ope
ration phase it sends an arising that has to be interpreted as a real failure
since

the partner
subsystem probably dropped out. Hence, the same arising has to be interpreted in a different kind of way
depending on the mission phase of the UAV in which the
arising was generated. This was only one of
many examples of incidents that can occur in an UAV. Summarized one can say that real failures can be
mission critical and require sudden decisions to guarantee the successful completion of the mission. In
contra
st, bogus failures are often false alarms or subsequent failures of real failures and can be evaluated
after the UAV mission during mission debriefing or maintenance.

EADS MAS relies on
ANNs

as classifier for the assignment of arisings as real or as bogus
failures. As
already described in basic chapter
2.1

ANNs

have to be trained to fulfil a certain classification task. A
broad spectrum of example data reflecting the classification problem is necessary for training. For this
reason, a huge database with man
y thousands of arisings derived from the
modern fighter

aircraft

mentioned above

and classified by humans was established. Engineers together with the maintenance crew
have analyzed documents concerning the subsystems, they questioned pilots, and they have

incorporated
their own experience for classifying the arisings. This process is not finished yet but the database is daily
in pr
ogress and in extension. As shown

in

Figure

5

this arising database is the basis for training the
ANN
.


Figure

5
: System conce
pt of automatic failure classification

The
ANN
that is used for the automated classification functionality is a feedforward multilayer perceptron
(MLP) consisting of about 50 neuron
s. An input vector with about a dozen

elements represents each
arsing. The
training of the
ANNs

is an iterative process. Lots of training iterations and epochs respectively
are necessary to enable the ANN to distinguish between real and bogus failures. Usually the training
converges after
10
4

epochs. Passing through this learning

phase the trained ANN has to be embedded into
a software framework developed by EADS MAS and can afterwards come
into operation. This operation

phase proceeds on ground for the
fighter

aircraft now. Concerning UAVs, it is considered to transfer the
automa
ted classification functionality on board as depicted in
Figure

5
.


Figure

6
: History concerning the training of ANNs for failure detection

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

7

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED


At the moment the automated classification functionality using
ANN
s achieve a classification
performance of 95% con
cerning the real failures. The classification performance for bogus failures is also
about 95%. The underlying data is taken from a database containing arisings of flights of
a modern fighter

aircraft developed by EADS MAS. The most important aspect of the

automated classification task is to
find real failures with a very high confidence in the whole amount of arisings. Of course, the confidence
for classifying bogus fail
ures as such is important, too. B
ut
classifying a bogus failure as real does not
introd
uce any criticality, only effort for repairs.
Because of that fact, the training of the ANN concentrated
primarily on the classification of real failures. A history curve depicting some major milestones in the
training of ANNs can be seen in

Figure

6
.

The
first point in the history curve at 0% marks the receipt of the first training data set containing arisings
from the
fighter aircraft
. The training of
ANN
s of the type MLP resulted in the first milestone with a
classification performance of 81% for real fa
ilures. Other types of
ANN
s like the Jordan/Elman networks
were not as successful as the MLP networks. Because of this fact, the further development work
concentrated only on the latter type of ANNs.

To reach the 95% classification performance under realis
tic conditions several more improvement steps
have been applied. As already mentioned above the arising database is updated permanently as more and
more information is available that helps the engineers and maintenance crew to classify arisings correctly.
After obtaining a few updates concerning the arisings used for training
,

the performance of the MLP could
be increased.

The creation of an equal proportion of real failures and bogus failures
for the

training arisings also
improved the classification perfo
rmance. As there exists a multiple of bogus failures than real failures this
fact was also reflected in the built up of the training data set. Hence, the MLP got more information about
bogus failures and was more sensible in classifying this type of arisin
g than classifying real failures.

A very important aspect concerning enhancement of the classification performance was definitely the
long
-
term adjustment of configuration parameters of the MLP. This concerns e.g. the built
-
up of the MLP
itself, which mean
s how many neurons e.g. form the MLP. In addition, the parameters of the learning
algorithm called backpropagation are concerned.

The last but most important improvement aspect that shall be mentioned here can be ascribed to the
competence of the MLPs itse
lf. The MLPs were not able to learn the complete amount of arisings that
were used for the training. That means after several thousands of training iterations the MLP still assigned
several arisings to the contrary class than the engineers and the maintena
nce crew did. These arisings were
delivered to the engineers with the suspicion that the human classification could be wrong. This suspicion
was confirmed after the engineers returned the extracted arisings having reclassified over 50%. After
another train
ing phase using the improved training arisings
,

the
training
success

could be
increased.

The automated classification functionality based on
ANN
s offers the ability to assign automatically
arisings to the class of real failures or the class of bogus failu
res. The possible classification performance
was demonstrated bas
ed on arisings derived from a modern fighter

aircraft. At the moment this automated
classificati
on
is not in
service but certain pilot cases were worked through to show its performance.

In se
rvice applications
based on this automated classification will offer the following benefits: First time
saving due to the fast classification performance.
Online and o
ffline
ten thousands

of parameter sets coul
d

be analyzed within seconds. Second an improv
ed maintenance cycle due to constant performance 24 hours
daily on seven days the week.

Disruptions or distractions of humans that possibly lead to some
misclassified training data
sets

will not influence an automatic system. Also the confidence assignment

given by ANNs can
priories

the focus of a maintenance crew.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

8

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


Looking forward into the future onboard decision
-
making is one of the most important aspects concerning
the autonomy of UAVs. The environment in which the UAV is operating and especially the heal
th of the
UAV itself must be well known for decision
-
making. Without an automated functionality for interpreting,
each arising must be sent via a data link to the operator on ground for interpretation



causing time delays
at least.

Further steps can be to

provide failure procedures for real failures in addition. Failure procedures are lists
that describe counteractive measures for trouble shooting. EADS MAS has developed a concept for the
generation of such action lists for another fighter aircraft several

years ago, cf.

[3]
. Therefore, each action
list is assigned to a certain real failure. Of course, the detection of real failures is the first step of the
concept according to that specific actions have to be applied. These detections were also delivered b
y
ANNs as shown in the sample application above. The concept was already
implemented

and tested within
an experimental pod on a fighter aircraft developed by EADS MAS amongst others.

3.2
Automatic Target Recognition Applications

3.2.1
Requirements of
Autom
atic Target Recognition

Before the
MAS automatic target recognition (ATR) applications are presented
it seems useful to mention
some general requirements resulting from the envisaged use in military aircraft avionics systems.

The main objective of ATR is t
o detect target objects on images auto
matically, with sensor, objects and
backgrounds covering a broad range of mission scenarios.

Figure

7
shows some sample objects and
scenarios recorded with different sensor types.

The most used sensors in military airc
raft are infrared (IR;
Figure

7
a,

b,

c), electro optical (EO;
Figure

7
d) and
s
ynthetic aperture radar (SAR;
Figure

7
e). The objects
range from tanks in
Figure

7
a and 7d
("T72", "Leopard", "BTR1" and 2 "Gepard" air defence tanks), over
aircrafts like
the 2
"Transalls" in
Figure

7
b
up to patrol boats in
Figure

7
e,
multiple launch rocket system
s

(MLRS)

(red circle in
Figure

7
e)
,

vehicles and

SCUD launcher
s (
blue

circle in
Figure

7
e)
.


(a)


(b)


(c)


(d)


(e)

Figure
7
: Sample imagery for ATR. Images fr
om a to c show IR imagery, image d shows an EO
image and image e shows a SAR image.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

9

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED


A flexible ATR algorithm should scope with all these kinds of sensors, objects and backgrounds, ideally
with as few
adaptation

work as possible.

Due to various hardware
pla
tforms

already introduced in
different military air systems there is
also the wish on hardware
independence
, i.e. the algorithms should
not be restricted to dedicated hardware.

Central requirement is the real
-
time capability of ATR algorithms.
To support r
econnaissance, weapon delivery or to aid the pilot the ATR results have

to be calculated

at
best at the sensors recording speed

Additionally
the time needed to adapt the ATR algorithms to new scenarios (sensors, objects
,
backgrounds) should be short.

3.2.2

ATR for G
round

Based Objects

on a Force Protection Scenari
o

A good example to show the set up and capability of
MAS ATR application
s

was the
demonstration
during the
Force Protection exhibition at WTD91, the German Technical Centre for Weapons and
Ammunit
ion, located in Meppen in northern Germany. The topic of the exhibition was the protection of
friendly military forces from attacks by hostile persons in the context of asymmetric warfare. Therefore
automatic systems should be demonstrated which support a
security operator during the surveillance of the
surrounding environment of a military camp. Thereby the detection rate of the persons has to be at least
comparable to a human operator and the false alarm rate has to be very low.

The objective towards
MAS
was the demonstration of the ability to detect a person on IR imagery and to discriminate the IR signature
of a person from other objects like vehicles or background structures of any kind to keep the false alarm
rate very low.

Before the demonstration a r
ecording campaign was carried out to obtain the image
database

required for
the training of the classifiers. The image
database

was recorded during one day with the Zeiss
OPHELLIOS IR camera provided by WTD91. The resolution of this camera is 640 by 480
p
i
xel
s
.
Altogether 3 hours of IR video material with varying scenarios have been recorded. In
Figure

8

some
example imagery can be seen which illustrate the performance of the camera and the challenge of the
scenario for the ATR application. On image (
8
a) a
person crosses the street with 4 moving vehicles in the
background.
In this scene challenges result from the cars due to their
changing
p
ixel signature

and from
the
lampposts

due to

their long and narrow shape which is “similar” to persons
.

In image (
8
b) 2

persons can be seen behind a car on a wide street with a surveillance tower in the
background. The size of the persons in image (
8
b) is noticeable smaller than the size of the persons in
image (
8
a). As the size of the persons is small in image (
8
b) the di
scrimination between persons and other
objects becomes more diff
icult.

On image (
8
c) two small persons, a container and a truck in front of a varying background are shown.
Red
arrows indicate the position of the persons.
The varying background behind the p
ersons is also a challenge
for the ATR application, because the
ANN
s have to learn to be non
-
sensitive to background
p
ixels near
the
p
ixels belonging to a person. Finally image (
8
d) shows a van together with a small running person on a
street. The
p
ixel si
gnature of a running person changes much compared to the signature of a walking
person
.

A selection from this image
database

was used to train the classifiers. For the
supervised
training the
grey
values in the filter window together with the

“ground truth

(persons marked by an operator) were used as
training sets.


Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

10

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED



(a)


(b)


(c)


(d)

Figure
8
: Sample imagery from the recording campaign for the ATR demonstration at the Force
Protection exhibition of WTD91

This database was used by the MA
S automati
c training scheme

which optimizes all relevant parameters

to
deliver the ready to use ATR application.

After testing, evaluation and retraining of the classifiers a final
version of
the classifiers has been chosen, t
he whole training phase was finished aft
er two weeks on a
commercial of the shelf PC with an Intel Q6600 processor.

In the operation

phase

t
hese optimized classifiers
are passed to the ATR f
ramework application

with the
aim to
apply the classifier to image frames of an input video stream. The ou
tput of the classifier, the
detected target objects, is visualized by overlays on the image

which
indicate the object class.


Figure
9
: Test setup of the ATR demonstration at the Force Protection exhibition of WTD91. The
used IR cameras are placed on the
military truck.

For the force protection demonstration the general setup can be seen in
Figure

9
.
The sensors and the
demonstration area together with the evaluation equipment have been set up besides a long strait street.
During the demonstration persons
crossed the street in different distances to the sensors. The
re were two

IR sensors
in use, both mounted on a military truck and shown in more detail in
Figure

10
a. O
n the left
hand side the smaller Zeiss OPHELLIOS IR camera provided by WTD91 can be seen.
For this camera the
classifiers have been trained. On the right hand side
it's successor,
the ATTICA IR camera system from
Zeiss
,

can be seen. This camera system

has

a very narrow field of view and is able to scan a field of regard
mechanically by remote c
ontrol. The test setup is completed by the presentation tent (
Figure

10
b) hosting
the MAS ATR
commercial of the shelf Notebook with an Intel Core2 Duo processor

and the

presentation
screen
s
.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

11

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED




(a)



(b)

Figure
10
: Test setup of the IR cameras and the vi
sualization board for the ATR demonstration at
the Force Protection exhibition of WTD91.

In the first phase of the
demonstration the video output stream
of
the OPHELLIOS IR camera

was used as
input for the A
TR Framework application
.

S
ome persons in differ
ent distances to the camera crossed the
street. The classifiers detected all persons successfully with no false alarms during the demonstration.
Typical outputs are shown
in
Figure

11
.


Figure
11
: Result of the ATR demonstration at the Force Protection ex
hibition of WTD91. On the
left hand side image the running application on a Laptop can be seen. On the right hand side the
recognized person is marked by the application by a red square.

In the second phase
the video stream of the ATTICA IR camera was conn
ected with the MAS ATR
Framework application. As the ATTICA camera has a very narrow field of view the imaged scene was not
on the street but on the test side of the WTD91 in a distance

between 0.5

and 2 km. After minor
adaptations of the sys
tem the detect
ion worked
well with only one false alarm during the presen
tation.

Altogether the demonstration has been very successfully as
there was a 100% hit rate for the persons
and
only 1 false alarm occurred during 3 hours of demonstration.
Frame rates
of 18 Hz un
der operational
conditions

on a commercial of the shelf hardware indicate the real
-
time capability.

Due to the switch of
the sensors during the demonstration the ATR algorithm has been
proven

to be quite robust with respect to
changes of image sources. Las
t but not least with the
high level of automation
a short turn around time
from training campaign to field application of about 3 weeks could be achieved.


Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

12

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


3.2.3
Other applications

The
MAS
ATR application has been applied in the described manner to differe
nt images covering sensors
like SAR

[4]

(
Figure

12
a), EO (
Figure

12
b) or IR
[5]
,
[6]
both
forward

looking and line scanner
(
Figure

12
c
,

d
). The typical task is the coarse classification of ground targets like ships, tanks, trucks or
aircraft. In typical sce
narios the ATR application has to decide between about 10 object classes and to
mark their positions in the image.

Some sample results are shown in
Figure

12
. In the example of
Figure

12
a the shown classes shou
l
d be classif
i
ed, in
Figure

12
b it was th
e tas
k to pick out the tank
"Leopard".

A quite similar situation was given in
Figure

12
b, this time in IR imagery and the requested
object type was the
"
BMP
"
.

It should be noted, that the ATR application works on high definition imagery and videos the same way
than on standard definition sources.

Also non ANN based classification applications have been developed
like the model based ATR applications [7].





(a)


(b)


(c)


(d)

Figure
12
:
ATR results for ground objects

Of course ATR applications are
not restricted to air to ground scenarios.

UAVs which
have
to operate
autonomous
due to mission reasons or due to loss of data link need an airborne collision avoidance
system. For cooperative traffic transponder based collision avoidance systems exist but

for non
-
cooperative vehicles (examples are shown in
Figure

13
) an alarm must be raised sufficient early to initiate
any collision avoid
ance actions.

An image processing based method has to cover a quite large angular area with
-
30 to +25 degrees for the
e
levation angle and
-
110 to +110 degree for the azimuth field of view. To achieve performance like a
human pilot detection rates better than 95% are required.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

13

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED



(a
)


(b
)


(c
)

Figure
13
:
Non
-
cooperative objects

To check whether the lea
rning based ATR app
lication could

reach this level of confidence
,

a trial with
imagery generated by a MAS
recording
campaign was performed at Manching site.
Figure

14

shows some
imagery from this campaign with different objects like helicopters (14a), transport aircraft (14b
) and
fighter aircraft (14c).

During the recording campaign videos of 1 hour with different aerial objects and
backgrounds have been generated. For the training of the classifiers a selection of about 30 image
s were
used.
T
he
evaluations of the
ATR
show,

t
hat the required detection rate is achieved as
the d
etection
performs well with
an average detection rate

of 95%
.


(a)


(b)


(d)


Figure
14
: Sample imagery for the ATR Air to Air application.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

14

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED


4.0
CONCLUSION AND OUTLO
OK

Taking the human pilot as an a
rchetype a key capacity of sensing systems is the concentration of the
output information to a high level


especially for autonomous operating air vehicles.
In situations where
the output depends on a large number of input values, typical examples are hea
lth condition monitoring or
automatic target recognition, this task is equivalent to classification.

From the experience built up so far ANNs seem to have the potential to deliver good results in complex
tasks with
large

data input streams.
ANNs offer the
benefit
s

of a training based approach
, so the input
-
output relations have not to be formulated
explicitly
, a high flexibility to different applications and a quite
easy
real
-
time

implementation taking additional advantage from parallel processing capabilit
ies, e.g. in
multicore
environments
.

To develop

ANNs
of high quality
high emphasis has to be placed on the construction of the training and
test data sets as well as on the definition of hit and error rates. Automatisms to find optimal parameter sets
and t
o control the training cycles
spe
ed up ANN development and gua
rantees high turn around rates.

The examples presented show that ANN applications can deliver results with sufficient quality to be of
operational use. In health monitoring field trials to suppo
rt maintenance personal are envisaged, the
automatic target recognition applications are already available commercially. The application
"AUTOPOL" is a software component to aid the FLIR
-
operators of German Police Helicopter Squadrons
in the exploitation o
f FLIR images, e.g.

in the search of missed people. The current implementations cover
a wide range on hardware platforms ranging from PC type systems to avionics mission computers on
Power PC
b
asis, in all cases
real
-
time

behaviour could be achieved


also

in the challenging case of
automatic target recognition.

After the successful demonstrations and first applications in spin off type projects the way is paved for
applications in the

core products of Military Air S
ystems. On this way operational scenarios

and
integration aspects have to be
addressed
. In image processing aspects the move to high definition sensors
and the corresponding generation of very high data rates
will perhaps the next challenge to
real
-
time

performance.


5.0
REFERENCES

[1]

Nikolaus Petry
,
"
Fuzzy Logik und neuronale Netze
"
, JurPC Web
-
Dok.
187/1999, Abs. 1
-
54
http://www.jurpc.de/aufsatz/19990187.htm#fn00
.

[2]

J. Hertz, R. Palmer, A. Krogh, "Introduction to the theory of neural compu
tation",
Perseus Books
,
1991
.

[3]

R. Seitz, G. Kunerth, W. Mansel, P. Hurst, "
Diagnostic system with neural networks for aircraft",
DaimlyerCrysler Aerospace
.
In:
Proceedings of the 20th Symposium
Aircraft Integrated Monitoring
Systems (AIMS)
, Garmisch
-
Partenk
irchen, Germany
, May 2000
.

[4]

Heiko Seidel, Christoph Stahl, Peter Knappe, and Peter Hurst, "Dynamic generation of artificial
HRSAR imagery for ATR development and cockpit simulation", Proc. SPIE 5426, 264 (2004)
.

[5]

Christoph Stahl and Paul Schoppmann, "Advance
d automatic target recognition for police helicopter
missions", Proc. SPIE 4050, 61 (2000).

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition

RTO
-
MP
-
SCI
-
202

27

-

15

NATO
/PFP UNCLASSIFIED

NATO
/PFP UNCLASSIFIED


[6]

Christoph Stahl, Stefan Haisch, and Peter Wolf, "Automatic target recognition flight prototype for
police helicopters", Proc. SPIE 4726, 141 (2002).

[7]

Heiko Seidel, C
hristoph Stahl, Wolfgang Ensinger, Frode Bjerkeli, Paal Skaaren
-
Fystro, Kirsten
Rosseland, Per Inge Jensen, "Assessment of model
-
based automatic target recognition on recorded
and simulated infrared imagery", Proc. SPIE 6967, 2008
.

Applications of Neural Networks for Intelligent Data

Interpretation in UAVs: Health Monitoring and Target Recognition






27

-

16

RTO
-
MP
-
SCI
-
202

NATO
/PFP UNCLASSIFIED


NATO
/PFP UNCLASSIFIED