Proposal for AIMS TDP Advanced Imaging for Surveillance: Target Detection

lynxherringΤεχνίτη Νοημοσύνη και Ρομποτική

18 Οκτ 2013 (πριν από 3 χρόνια και 7 μήνες)

74 εμφανίσεις

DRAFT

Proposal for AIMS TDP

Advanced Imaging for Surveillance: Target Detection



July29, 2004


James Elder, PhD

Associate Professor

Centre for Vision Research

York University

4700 Keele Street

Toronto, Canada

M3J 1P3


jel
der@yorku.ca

www.yorku.ca/jelder

416
-
736
-
2100 ext. 66475



Jocelyn Keillor, PhD

Defence Scientist

DRDC Toronto

1133 Sheppard Avenue West

P.O. Box 2000

Toronto, Ontario

M3M 3B9


Jocelyn.Keillor@drdc
-
rddc.gc.ca

416
-
635
-
2187

DRAFT

Background

Corporate background, areas of expertise, and relevance of technology to the project.

The Human
-
Computer Interaction Group at DRDC Toronto was responsible for
overseeing the dev
elopment of the Human
-
Machine Interface for ELVISS (Enhanced
Low
-
light Visible and Infrared Surveillance System), the predecessor to AIMS developed
at DRDC Valcartier. Jocelyn Keillor, the lead scientist responsible for advanced
imaging concepts related

to AIMS at DRDC Toronto, has over the past five years worked
on National Search and Rescue Secretariat (NSS) new SAR Initiatives
-
funded research
designed to establish the utility of potential enhancements to multi
-
sensor imaging
systems for search and res
cue. Dr. Keillor has served as project manager for an New
SAR Initiatives
-
funded evaluation of map and symbology enhancements related to multi
-
sensor SAR. Ongoing projects include the development of a software tool for defining
optimal sensor sweeps as a

function of terrain, and an investigations into the
improvement of operators’ size and distance judgment and situational awareness for
multi
-
sensor systems employing different fields of view and levels of zoom.

DRDC Toronto houses a search and rescue simu
lator that is used to evaluate the efficacy
of design concepts for multi
-
sensor systems, able to simulate complete missions over real
Canadian terrain. The Human
-
Computer Interaction group at DRDC Toronto conducts
research related to visualization of terr
ain, attentional management and situational
awareness, symbology, and human interaction with multi
-
sensor systems.

Elder Lab: Knowledge, Expertise and Human Resources

The Elder laboratory currently consists of 10 individuals: James Elder, the principal
in
vestigator, two postdoctoral fellows, one senior research associate, four graduate
students and two undergraduate research assistants.

Prof. Elder is an expert in computer and human vision. He obtained his PhD in Electrical
Engineering from McGill Unive
rsity in 1995, and was a Senior Research Associate at the
NEC Research Institute in Princeton, NJ before joining the faculty at York University in
1996. Dr. Elder received the Young Investigator award from the Canadian Image
Processing and Pattern Recogni
tion Society in 2001, and the Premier’s Research
Excellence Award from the Ontario Ministry of Enterprise, Opportunity and Innovation
DRAFT

in 2003. Dr. Elder is a member of the Centre for Vision Research and the graduate
programmes in Computer Science, Mathema
tics and Statistics and Psychology at York
University.

Collectively, the Elder laboratory has substantial knowledge and expertise in the areas of:
visual detection and localization, attention, image segmentation and perceptual
organization, multi
-
sensor s
urveillance and three
-
dimensional vision. The laboratory is
highly interdisciplinary, bringing together engineers, computer scientists and
psychologists to work on problems in human and machine perception. The team has
considerable experience in bringing

ideas through the algorithm, system integration,
physical prototyping and evaluation stages, and have developed substantial C++ and
MATLAB software libraries to support this process. The lab is well
-
integrated with the
international research community, a
s well as with the collaborative
(academic/industrial/government) research and development community within Canada.

The Elder laboratory participated in the ESVS 2000 TDP, led by Lt. Col. Rick
Thompson, contributing research on the development of texture m
aps for synthetic
terrains to optimize pilot performance in helicopter search
-
and
-
rescue missions. These
texture maps were used in the Bell 205 system demonstration held at the NRC Flight
Research Lab in 2001. Our main collaborators were CAE, UTIAS and t
he NRC FRL.
Since that time, we have been working with CCRS (NRCan) under GEOIDE and
CRESTech grants to develop novel algorithms for computing higher
-
accuracy GIS terrain
databases using high
-
resolution satellite imagery. We intend to apply for additiona
l
support for the proposed project from CRESTech.


Physical Resources

The Elder laboratory occupies 900 square feet over five rooms within the Centre for
Vision Research component of the Computer Science and Engineering Building of York
University. The la
boratory is very well
-
resourced, with roughly 30 Pentium, SGI and
Power Mac workstations, eye
-

and head
-
tracking facilities, stereo digital cameras and
stereo projection devices. Surveillance research is based upon two proprietary sensor
platforms: an at
tentive panoramic sensor and an attentive wide
-
field sensor.

DRAFT

As part of the Centre for Vision Research the laboratory has access to a host of other
resources, including: high
-
end immersive reality environments, a very complete and
well
-
staffed electronics

and metalworking workshop, and computing and secretarial
support.

The Elder laboratory currently attracts roughly $300,000 in research funding annually, all
from Canadian sources. This support comes from the Natural Science and Engineering
Research Counc
il (NSERC), the Institute for Robotics and Intelligent Systems (IRIS), the
Pre
-
Competitive Advanced Research Network (PRECARN), Geomatics for Informed
Decisions (GEOIDE), the Centre for Research in Earth and Space Technologies
(CRESTech), and the Ontario M
inistry of Enterprise, Opportunity and Innovation. The
Centre for Vision Research, of which the Elder laboratory is a part, attracts roughly $6M
in funding annually.

Prototype Technologies and Intellectual Property

The Elder laboratory has developed sever
al technologies for attentive visual surveillance.
These include:



Proprietary Attentive Panoramic Surveillance Sensor. A visual sensor capable of
maintaining an instantaneous panoramic (360 degree) surveillance over an open
area, while obtaining high
-
res
olution zoomed footage at points of interest.
Patents were filed in the U.S. and Canada in May 2002 on key aspects of this
technology.



Proprietary Wide
-
field Attentive Sensor. Similar to above, but designed to survey
an entire room from a corner mount.



Proprietary algorithm for reliable frontal and non
-
frontal face detection at low
-
resolution, and capturing of high
-
resolution facial snapshots.



Proprietary face tracking technology.

For more detailed information on the Elder laboratory, visit
www.elderlab.yorku.ca
.

DRAFT

Contribution

Identify the overall contribution of the proposal, highlighting the uniqueness of proposed
contribution.

Human target detection in search
-
and
-
rescue and other air
-
to
-
ground surveillance
applications is an imperfect process. Error rates depend upon the spatial acuity and
colour sensitivity of individual observers, and fatigue is known to play a major role
(Donderi, 1994). Improved sensing technologies (e.g., range
-
gated near
-
IR, thermal
IR,
laser range data) have the potential to improve performance. Ultimately, however,
performance will be determined by how accurately the vast quantities of data generated
by these technologies are interpreted, either by computer or by the human visual
system.
The goal of the proposed project is to research and develop advanced imaging techniques
that will ensure that the tremendous power of these diverse sensing technologies is
harnessed effectively to yield faster and more reliable detection rates.

Th
ere is limited published work on the application of image processing techniques to
boost detection rates for search
-
and
-
rescue applications. Sumimoto et al. (1994, 2000,
2001) have conducted preliminary investigations into the use of colour and motion cue
s
for automatic detection in a hypothesize
-
and
-
test framework. Frame averaging and
neural network techniques have been used to boost signal
-
to
-
noise ratios for radar
detection of small watercraft in choppy waters (Leung et al., 2002; Dugan et al., 2003).

Our goal is to greatly extend this prior work to consider a broad range of sensor
modalities, target models, computer vision and statistical estimation techniques. We will
also emphasize thorough evaluation on realistic datasets.

A guiding principle of t
he proposed work is that a highly
-
trained human operator will
always play the key role in the detection process. Thus our project is directed specifically
toward understanding how advanced imaging algorithms can be designed so as to reliably
boost human d
etection performance, and to under no circumstances interfere with or limit
human observation. The SAR simulator at DRDC Toronto will serve as a test
-
bed for
evaluating the efficacy of all image enhancements.

Despite the complexity of human
-
in
-
the
-
loop pr
oblems in general, as a first
approximation, the detection problem can be decomposed into two cascaded problems:

DRAFT

1.

Image processing stage.

Mapping of incoming visual data and mission models to
a spatial probability map .

2.

Visualization stage.

Mapping of spa
tial probabilities, in conjunction with
incoming visual data, to visualization modules for human interpretation.


The research team, with substantial expertise in both machine vision and human
perception, is uniquely qualified to address both stages of the

problem. While prior
studies have focused either on automatic target detection and visualization, few have
addressed these problems from a systems perspective, with thorough human
-
in
-
the
-
loop
evaluation. This is our goal.

Major challenges in the develop
ment of effective air
-
to
-
ground target detection
algorithms for applications such as search
-
and
-
rescue include:

1.

The small size of the target relative to the resolution of the sensors. In many
cases the target may subtend only a small number of pixels.

2.

Var
iations in data quality due to atmospheric effects, tracking imperfections,
partial target occlusion etc.

3.

Imperfect target knowledge.

The nature of these challenges suggests a statistical approach in which multiple weak but
complementary modalities and cue
s are combined to produce a relatively reliable
inference. Research in the Elder lab at York University has demonstrated the
effectiveness of this approach in far
-
field and wide
-
field detection and tracking of human
activity (Prince et al., in press) and
in the segmentation of terrain features in IKONOS
panchromatic satellite data (Elder et al., 2003). Research conducted by the Human
-
Computer Interaction Group at DRDC Toronto using the DRDC Toronto SAR simulator
has demonstrated the impact of interface c
onfigurations on detection rates in simulated
search and rescue using multi
-
sensor systems (Neal, 1999; Keillor et al., 2002; Crebolder
et al., 2003). Jointly, we intend to build on our experience from past projects in
addressing the target detection prob
lem in a search and rescue context.

In order to be relevant to search
-
and
-
rescue operations, algorithms developed will be
real
-
time or near real
-
time. To achieve this, we will take advantage of real
-
time
DRAFT

techniques we have recently developed for human sur
veillance applications (Prince et al,
2005) and integral image and attentive cascading techniques for fast object detection
from the recent computer vision literature (Viola & Jones, 2001).

Applications

Provide a description of the potential technology app
lications in a military context,
noting an Air
-
to
-
Surface environment with Search & Rescue and Surveillance
implementations.

In the proposed work we will consider a number of different complementary data
sources, including:


1.

Range
-
gated near
-
IR

2.

Thermal IR

3.

Laser range data

4.

Conventional video

5.

Range data from stereo motion algorithm


Methods for fusing these data at both image processing and visualization stages will be
developed and evaluated. Our goal is to develop general detection techniques that will
app
ly to a range of targets and terrains, including ocean surfaces. A generalized target
model framework that can incorporate varying levels of target appearance knowledge
will be developed.


DRAFT

Proposed Work

Provide a detailed description of proposed work eff
ort. An outline WBS should be
provided.

1.

Acquire test datasets. These will consist of airborne imagery from several types
of sensors, including range
-
gated near
-
IR, thermal IR, laser range data, and
conventional video. In the first phase of the project,

existing datasets will be
employed. These will include airborne IR data from flight trials over both water
and ground, available from DRDC Valcartier. One trial involves a man
swimming the St. Lawrence river, another involves a vehicle, and a man walkin
g
in woods near CFB Petawa. Additional data with associated ground truthing will
be acquired as the AIMS project develops, and these data will be incorporated
into the proposed project as it becomes availble.

2.


Develop and evaluate representations for targ
et and background, given modest
target knowledge, that will improve effective signal
-
to
-
noise in search and rescue
missions.

3.

Develop and evaluate improved algorithms for fusing multiple sources of sensing
data for the purpose of small object detection in s
earch
-
and
-
rescue missions.

4.

Develop and test methods for visual communication of target location
probabilities to human search
-
and
-
rescue personnel.

Data Sources

We hope to obtain dataset for development and evaluation purposes from several sources:

1.

DRDC Va
lcartier

2.

Obzerv?

3.

Neptec?

4.

NRCAN?

5.

NIEC

6.

US Coast Guard?

DRAFT


Risk Assessment

Identify technological, cost, schedule or other relevant risk areas. Identify the degree of
the risk (low, medium, high) and the probability of occurrence (low, medium, high)

Risk 1. A
cquisition of Datasets

In order to achieve the stated objectives we must obtain appropriate datasets from DRDC
or other sources. This is a critical requirement, so the impact of complete failure here
would be high. However, we are confident that at the v
ery minimum we will have access
to existing airborne IR data from flight trials over both water and ground, available from
DRDC Valcartier. Moreover, there are explicit plans within the AIMS project to
continue to acquire new data as the project develops.

Thus we believe that the risk here
is very low.

Risk 2. Access to Classified Military Research

Ongoing research by the Canadian Forces on targeting for military operations is generally
classified and it is highly probable that details regarding specific

algorithms used in
processing and post
-
processing for targetting will not accessible to the proposed project.
However the present project is directed primarily at target detection for fixed
-
wing
search
-
and
-
rescue operations, and the specific parameters o
f this particular problem will
to a great extent determine what algorithms and visualization strategies are most
effective. Also, the range of sensing modalities and sensor properties will be somewhat
particular to the problem of search
-
and
-
rescue. We ar
e therefore confident that the
proposed project will make a substantial contribution to knowledge and technology
design strategies for search
-
and
-
rescue operations beyond what is presently available
from research on military targetting. Our results may al
so well lead to wider application
within the Canadian Forces.


Risk 3. Personnel

Achievement of the stated objectives will depend critically upon personnel in the Elder
laboratory. Most important will be a postdoctoral fellow or fellows who will assume
DRAFT

m
ajor responsibility for processing datasets, coding algorithms and performing
experiments.

Simon Prince, a current postdoctoral fellow in the Elder laboratory, is expected to
continue in the laboratory until at least February 2006, and probably beyond th
at date.
He is a highly capable individual with strong training in both computer vision and human
psychophysics. However, he will only be able to devote at most 50% of his time to the
proposed project.

A second postdoctoral fellow, Antonio Robles
-
Kelly,
will be arriving in January 2005,
and is expected to remain in the laboratory until January 2007. Antonio has a very strong
background in computer vision and will be able to devote up to 50% of his time to the
project.

Should the project be funded, we wil
l likely advertise for a third postdoctoral fellow to
join the laboratory. In our most recent recruiting experience, we received over 65 strong
applications, and so we are confident that a talented individual will be found. This
individual will provide c
ontinuity if and when Simon Prince leaves the lab.

The exact percentage contribution of each of these individuals to the project is at this
point undetermined, and will probably evolve over the course of the project. At this point
we estimate the stated w
ork requires the equivalent of one full
-
time postdoctoral fellow,
as budgeted (see below).

We will also require a graduate student trained in psychophysics and perception to help
evaluate the visualization stage of the project. This could conceivably invo
lve one of the
students currently in the laboratory, but is more likely to be a graduate student starting in
September, 2005. Progress on this stage of the project will depend on hiring this
individual, and on their training progress. This risk is mitiga
ted somewhat by the
availability of postdoctoral fellows to assist in training and experimentation if required.
Thus the impact of a failure here is moderate, but the probability is low.

Also important to the research team is our Research Associate, Bob H
ou, who does all
systems administration, purchasing and commissioning of equipment, and systems
programming in the laboratory. Bob is of great assistance in maintaining continuity in
DRAFT

the lab and helping to train new recruits. We have budgeted 30% of his
time for the
project. Bob is a permanent employee: there is very little risk of his leaving.


Risk 4. System Performance

Our goal is to develop advanced imaging algorithms that will improve human detection
performance in search
-
and
-
rescue and other surv
eillance tasks. Given the sophistication
of the human visual system and the excellence of trained personnel, it is possible that
algorithms developed will fail to improve upon unaided human performance. On the
other hand, the fact that prior studies (Don
deri, 1994) have revealed systematic
differences in human performance related to acuity, colour sensitivity and fatigue
suggests that there are cases where image processing techniques could benefit. In the
worst case, a finding that advanced imaging algor
ithms have little or no effect on human
performance will still represent an important contribution to knowledge that may impact
future design and procurement of air
-
to
-
ground surveillance technologies by the
Canadian Forces. Thus both the probability and
impact of failure here is relatively low.


Cost Estimate (Rough Order of Magnitude)


Year 1

Year 2

Year 3

Total

Postdoctoral Fellow (100%)

$52,000

$55,500

$60,000

$167,500

Graduate Student (100%)

$21,000

$22,000

$23,000

$66,000

Research Associate (30%)

$20,000

$21,000

$22,000

$63,000

Co
-
op Students (100%)

$24,000

$24,000

$24,000

$72,000

Travel

$15,000

$14,000

$14,000

$43,000

Hardware, Software Licenses

$40,000

$25,000

$25,000

$90,000

Sensor Equipment

$10,000

$5,000

$5,000

$20,000

Subject Fees

$2,000

$2,000

$2,000

$6,000

Total

$184,000

$168,500

$175,000

$527,500

DRAFT

Deliverables

Identify Hardware, Software, and Reports that would be delivered

Deliverable

Description

Date

1

Prototype software for mapping air
-
to
-
ground
sensing data to target probability
maps.

September, 2007

2

Prototype software for mapping target
probability maps and air
-
to
-
ground sensing data
to visual displays for human observation

March, 2008

3

Monthly and quarterly progress reports

Ongoing

4

Conference papers

Ongoing in FY 06/07
a
nd 07/08

5

Archived journal publications

FY 07/08

6

Final report prepared for DRDC documenting
delivered software and new knowledge
acquired.

Apr 1, 2008



Progress Reporting

Written progress reports


monthly

Review meetings
-

quarterly


DRAFT

Schedule

Id
entify major activity schedule and milestones.

Milestone

Description

Date

1

Acquire initial datasets

Sept 1, 2005

2

Report on representations and target detection results

Apr 1, 2006

3

Report on data fusion results

Apr 1, 2007

4

Report on attentive tar
get detection results

Apr 1, 2008

5

Report on human
-
in
-
loop results

Apr 1, 2008

6

Final Report Published as DRDC Toronto Technical
Report

December,
2008


References


1.

J.M. Crebolder, T.D.M. Unruh, and S.M. McFadden.
Search performance using
imaging displ
ays with restricted field of view
, TR2003
-
007, Defence Research
and Development Canada

Toronto: Toronto ON, 2003.

2.

J.P. Dugan, C. C. Piotrowski and D.C. Campion, Detection of small targets in
ocean wave clutter using panchromatic time series imagery
.

Proc
. OCEANS
2003, vol. 5 ,

22
-
26, pp. 2560


2565, 2003.

3.

J. H. Elder, A. Krupnik and L.A. Johnston, Contour grouping with prior models,
IEEE Transactions on Pattern Analysis and Machine Intelligence
, vol. 25, no. 25,
661
-
674, 2003.

4.

J. Keillor, K. Hodges, M. P
erlin, N. Ivanovic, N., and J.G. Hollands, Imaging
Systems in Search and Rescue: Implications for Geographic Orientation,
Proceedings of the Human Factors and Ergonomics Society
-
46
th

Annual Meeting.
Human Factors and Ergonomics Society: Santa Monica, CA.
pl70
-
174.

5.

H. Leung, N. Dubash and N. Xie, Detection of small objects in clutter using a
GA
-
RBF neural network
.
Aerospace and Electronic Systems, IEEE Transactions
on
,

vol. 38, no. 1,

pp. 98


118, 2002.

6.

B. Neal,
Development of ELVISS human
-
machine interf
ace concepts. ELVISS
human engineering design approach document operator
, DCIEM
-
99
-
CR
-
078,
Canadian Marconi Company Canada: Kanata ON, 1999.

7.

S.J.D. Prince, J.H. Elder, Y. Hou and M. Sizintsev, Pre
-
Attentive Face Detection
for Foveated Wide
-
Field Surveillan
ce,
Proc. 2005 IEEE Workshop on
Applications in Computer Vision
(in press).

DRAFT

8.

T. Sumimoto, K. Kuramoto, S. Okada, H. Miyauchi, M. Imade, H. Yamamoto
and Y. Arvelyna,
Image processing technique for detection of a particular object
from motion images
.
Proc.
IEEE International Symposium on Industrial
Electronics
,

vol. 3,

pp. 1662


1666, 2001.

9.

T. Sumimoto, K. Kuramoto, S. Okada, H. Miyauchi, M. Imade, H. Yamamoto
and Y. Arvelyna
,
Detection of a particular object from environmental images
under various conditio
ns.

Proc. IEEE International Symposium on Industrial
Electronics
,

vol. 2,

pp. 590


595, 2000.

10.

T. Sumimoto, K. Kuramoto, S. Okada, H. Miyauchi, M. Imade, H. Yamamoto
and Y. Arvelyna
,
Detection of a particular object from motion images under bad
condition
.
Proceedings of the 18th IEEE Conference on Instrumentation and
Measurement Technology
, vol. 1,

pp. 318


322, 2001.

11.

T. Sumimoto, K. Kuramoto, S. Okada, H. Miyauchi, M. Imade, H. Yamamoto
and T. Kunishi,
Machine vision for detection of the rescue target
in the marine
casualty.
Proc.

20th International Conference on Industrial Electronics, Control
and Instrumentation
, vol. 2 ,

pp. 723


726, 1994.

12.

P. Viola and M.J. Jones, Rapid object detection using a boosted cascade of simple
features. Proc. IEEE Conf.

on Computer Vision and Pattern Recognition, vol. 1,
pp. 511
-
518, 2001.