Anytime Algorithms for Real-time Image Processing

aroocarmineAI and Robotics

Oct 29, 2013 (3 years and 9 months ago)

93 views

Anytime
Algorithms
for Real
-
time
Image
Processing
Angelo
Genovese
Wednesday, March 20, 2013
Outline

Introduction
-
Anytime algorithms
-
Main characteristics
-
Main fields of application

Anytime algorithms for real
-
time
image processing
-
Object detection/recognition
-
Path/motion planning
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
2
Anytime algorithms:
Introduction

A class of algorithms with specific features:
-
Are able to provide a valid solution at anytime
-
Can be interrupted before completion
-
The accuracy of the solution must increase with time

Different kinds of algorithms can have anytime
features:
-
Iterative
-
Randomized
-
Genetic
-

Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
3

Special structure of the algorithms
-
Iterative
-
Modular

Fast computation of initial solution
-
Subsequent refinement
Anytime algorithms:
Main characteristics
(1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
4

Features:
-
Measurable quality

Quality can be measured
-
Recognizable quality

Quality
can be
measured
easily at run
-
time
-
Monotonicity

Quality is a non
-
decreasing
function of time and input
quality
-
Consistency

Quality is correlated with time and
input quality
Anytime algorithms:
Main characteristics
(2/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
5
-
Diminishing returns

I
mprovement
is larger at earlier
stage, and
diminishes
with time
-
Interruptibility

Can
be interrupted and provide a
valid result
-
Preemptability

Can
suspended and resumed with
minimal overhead

Real
-
time applications
-
Available time is limited
-
Complexity of input data might change

Limited computational power
-
Constraints on accuracy

Unpredictable situations
-
Available time can vary
-
Computational power might change
Anytime algorithms:
Main fields of application (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
6

Object detection/recognition

Path/motion planning

Signal processing

Classification

Computational intelligence

Operational research

Automatic controls
Anytime algorithms:
Main fields of application
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
7

Object detection/recognition:
-
Obstacle tracking
-
Image alignment
-
Novelty detection

Path/motion planning:
-
3
-
D reconstruction
Anytime algorithms
for real
-
time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
8

Object detection/recognition:
-
Obstacle tracking
-
Image alignment
-
Novelty detection

Path/motion planning:
-
3
-
D reconstruction
Anytime algorithms
for real
-
time image
p
rocessing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
9
Object
detection/recognition:
Obstacle tracking (1/2)

Real
-
time obstacle tracking:
-
Automotive applications
-
Based on range images
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
10
Object detection/recognition:
Obstacle tracking
(2/2)

Problem features:
-
Hard
real
-
time
requirements

Safety
-
Problem variables might change:

Complexity of the scene

Computational power

Resolution of imaging device

Time
available
-
High
specificity not
required

False positives can be tolerated

False negatives can
not
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
11

Object tracking problem:
-

to

matching


components at time
(
𝑡

1
)


components at time
𝑡
Object detection/recognition:
Object tracking method (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
12

Matching not
1
to
1
-
Occlusions
-
Splitting
-
Segmentation errors
-
Change of scene
Object detection/recognition:
Object
tracking
method (2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
13

Object matching algorithm:
-
Object detection at time
𝑡

1
and
𝑡
-
Each range level considered separately
-
Possible groups of objects are computed
at
𝑡

1
and
𝑡

Adjacency graphs
-
For each group, all possible subgroups are computed
-
All possible
subgroups
at
𝑡

1
are matched with all
possible subgroups at
𝑡

Matches chosen
based
on similarity
of shapes and positions
Object detection/recognition:
Object
tracking
method (3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
14

Anytime implementation:
-
Solution of a reduced problem

1
to

matching
-
Subsequent solution of optimal problem


to

matching

Each iteration solves a different range level

If (and until) time is available
Object detection/recognition:
Anytime implementation (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
15

Reduced
1
to

matching
-
Works on different range levels separately
-
Components at time
(
𝑡

1
)
merged in groups

Subgroups are not computed
-
Each group is matched to

components at time
𝑡
Object detection/recognition:
Anytime implementation
(2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
16

Optimal

to

matching
-
Current reduced
matchings
are ranked
-
Ranking based on similarity of shapes and positions
-
Iteration on
matchings
-
Selection of worst matching
-
Refinement using optimal matching
Object detection/recognition:
Anytime implementation
(3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
17
Object detection/recognition:
Results of obstacle tracking using anytime algorithms
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
18
(a)
t
-
1
t
(b) Restricted matching
(c) Restricted and optimal matching

Object detection/recognition
-
Obstacle tracking
-
Image alignment
-
Novelty detection

Path/motion planning
-
3
-
D reconstruction
Anytime algorithms
for real
-
time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
19
Object
detection/recognition:
Image alignment (1/2)

Real
-
time image alignment:
-
Basic operation in computer vision applications

Problem features:
-
Time
-
consuming
-
Real
-
time
requirements
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
20

Classical direct image alignment:
-
Alignment described by a set of transformation
parameters
φ
-
Similarity defined as function of the transformation
parameters:
𝐷
=
𝑓
φ
-
Find the optimal transformation parameters that
maximize the similarity
Object detection/recognition:
Image alignment
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
21

Image alignment algorithm:
-
Optimization based on steepest descent algorithm
-
Improvement guided by
the gradient of the similarity
measure:

Mean Squared Difference

Mutual
Information
Object detection/recognition:
Image alignment
method
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
22

Anytime implementation:
-
Measure of similarity as a function of the number of
pixels
𝑝
used:
𝐷
=
𝑓
(
φ
,
𝑝
)
-
Function used
to compute the number of pixels to
use

Implemented
as look
-
up
table, trained
offline

Based on desired accuracy and the current magnitude of the
gradient
-
More pixels are incrementally considered during the
computation
Object
detection/recognition:
Anytime implementation
(1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
23

Anytime algorithm:
-
The number of pixels to use in the first step is computed

Based on known desired accuracy

Based on first gradient computation
-
The optimal transformation parameters are computed

Steepest descent optimization
-
The gradient of the similarity measure is computed
-
The number of pixels to use is increased
-
The transformation parameter are further optimized
Object
detection/recognition:
Anytime implementation
(2/3
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
24

Number of pixels can
be tuned
-
Required accuracy

More
pixels can be used
-
If time is available
-
If greater accuracy is
required
Object
detection/recognition:
Anytime implementation
(3/3
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
25

Object detection/recognition
-
Obstacle tracking
-
Image alignment
-
Novelty detection

Path/motion planning
-
3
-
D reconstruction
Anytime algorithms
for real
-
time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
26

Real
-
time novelty detection
-
Automotive applications

Problem features:
-
No high specificity required
-
Hard real
-
time constraints
Object
detection/recognition:
Novelty detection (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
27

Novelty detection problem:
-
Two
-
class classification problem
-
Segmenting from the image everything not recognized

Datasets of examples
Object detection/recognition:
Novelty detection
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
28

Outline of the method:
-
Extraction of features at each time step
𝑡
-
Feature reduction

Multiple Discriminant Analysis
-
Computation of similarity of current
scene
to each
example up to time
(
𝑡

1
)

SVM
-
based classifier
-
Computation of aggregated similarity

Linear combination
Object
detection/recognition:
Novelty detection method
(1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
29

If
result exceeds threshold
-
Scene
is not
novel
-
Example discarded

If result does not exceed threshold
-
Scene is novel
-
Example list is updated
-
Classifier
re
-
trained
online with new
example
Object
detection/recognition:
Novelty detection method
(2/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
30

Anytime implementation:
-
Computation of similarity limited

Interrupted when threshold is exceeded
-
Most significant examples processed first

Example that exceeds threshold moved at beginning of time
series
-
Premature stopping generates false alarms

Does not cause missed detections
Object
detection/recognition:
Anytime implementation
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
31
Object
detection/recognition:
Results of anytime novelty detection
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
32

Object detection/recognition
-
Obstacle tracking
-
Image alignment
-
Novelty detection

Path/motion planning
-
3
-
D reconstruction
Anytime algorithms
for real
-
time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
33

3
-
D reconstruction
-
Based on stereoscopic images
-
Preliminary step for motion
planning
-
Automotive applications

Problem features:
-
Real
-
time constraints
Path/Motion planning:
3
-
D reconstruction for motion planning
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
34

Point
-
based 3
-
D stereoscopic reconstruction:
-
Point extraction using Speeded Up Robust Features
(SURF):

Designed for speed and robustness

Extraction of points from left image

Extraction of points from right image
-
Matching of extracted points

Based on distance between features vectors
-
Triangulation

Based on camera calibration data
Path/Motion
planning:
3
-
D
reconstruction method
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
35

Anytime implementation:
-
Novel task handling method

Allows
interruptibility

Leaves computation in a consistent state
-
Anytime SURF algorithm for point extraction and
matching

Divided in different scale spaces

Iterative
Path/Motion
planning:
Anytime implementation (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
36

Each task divided in Main Part (MP) and Exception
Part (EP)
-
MP contains code

E.g. point extraction
-
EP handles exception and consistency
Path/Motion
planning:
Anytime implementation
(2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
37

Two
-
level scheduler
-
EPs are scheduled
to start as late as possible
-
MPs
are scheduled to end as early as possible

Anytime SURF
algorithm:
-
Subdivision in different scale spaces

The coarsest scale space is processed first
-
For each scale space the tasks are performed:

Extraction of points from left image

Extraction of points from right image

Point matching

Triangulation
-
Task are divided in MPs and EPs
Path/Motion
planning:
Anytime implementation
(3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
38

Results of 3
-
D triangulation:
Path/Motion
planning:
Results of anytime 3
-
D reconstruction
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
39

Anytime algorithms for image processing
applications were reviewed:
-
Object detection

Obstacle tracking

Image alignment

Object classification (e.g. novelty detection)
-
Motion planning

3
-
D reconstruction of the scene
Conclusions (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
40

Important field of application in automotive
scenarios
-
High accuracy often not necessary
-
Hard
-
real time constraints

Anytime algorithms for real
-
time image processing
are still not very explored
Conclusions (2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
41
References
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
42
1.
S
.
Zilberstein
,
“Using
Anytime
Algorithms
in
Intelligent
Systems”,
AI
Magazine,
Vol
.
17
,
No
.
3
,
1996
,
pp
.
73
-
83
.
2.
T
.
Dean
,
M
.
S
.
Boddy
,
“An
Analysis
of
Time
-
Dependent
Planning”,
In
Proceedings
of
the
Seventh
National
Conference
on
Artificial
Intelligence,
1988
,
Tom
M
.
Mitchell,
Reid
G
.
Smith
(Eds
.
)
pp
.
49
-
54
.
3.
M
.
S
.
Boddy
,
T
.
Dean,
“Solving
Time
-
Dependent
Planning
Problems”
.
Technical
Report
.
Brown
University,
Providence,
RI,
USA,
1989
.
4.
J
.
Grass
,
S
.
Zilberstein
,
"Programming
with
Anytime
Algorithms",
In
Proceedings
of
the
IJCAI
-
95
Workshop
on
Anytime
Algorithms
and
Deliberation
Scheduling
,
1995
,
pp
.
22
-
27
.
5.
K
.
Sobottka
,
H
.
Bunke
,
"Any
-
time
behavior
for
obstacle
tracking
,“
In
Proceedings
of
the
1999
IEEE/IEEJ/JSAI
International
Conference
on
Intelligent
Transportation
Systems,
1999
,
pp
.
368
-
373
.
6.
R
.
Brooks,
T
.
Arbel
,
D
.
Precup
,
“Fast
image
alignment
using
anytime
algorithms”,
In
Proceedings
of
the
20
th
international
joint
conference
on
Artifical
intelligence
(IJCAI'
07
),
2007
,
Rajeev
Sangal
,
Harish
Mehta,
and
R
.
K
.
Bagga
(Eds
.
)
.
Morgan
Kaufmann
Publishers
Inc
.
,
San
Francisco,
CA,
USA,
pp
.
2078
-
2083
.
7.
B
.
Sofman
,
J
.
A
.
Bagnell
,
A
.
Stentz
,
"Anytime
online
novelty
detection
for
vehicle
safeguarding,"
2010
IEEE
International
Conference
on
Robotics
and
Automation
(ICRA
),
2010
,
pp
.
1247
-
1254
.
8.
K
.
Wetzelsberger
,
T
.
Pfannschmidt
,
T
.
Ihme
,
"
Real
-
time
image
processing
for
motion
planning
based
on
realistic
sensor
data,"
2010
IEEE
International
Workshop
on
Robotic
and
Sensors
Environments
(ROSE
)
,
2010
,
pp
.
1
-
6
.
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
43
Thank you for your attention!