Anytime
Algorithms
for Real

time
Image
Processing
Angelo
Genovese
Wednesday, March 20, 2013
Outline
•
Introduction

Anytime algorithms

Main characteristics

Main fields of application
•
Anytime algorithms for real

time
image processing

Object detection/recognition

Path/motion planning
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
2
Anytime algorithms:
Introduction
•
A class of algorithms with specific features:

Are able to provide a valid solution at anytime

Can be interrupted before completion

The accuracy of the solution must increase with time
•
Different kinds of algorithms can have anytime
features:

Iterative

Randomized

Genetic

…
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
3
•
Special structure of the algorithms

Iterative

Modular
•
Fast computation of initial solution

Subsequent refinement
Anytime algorithms:
Main characteristics
(1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
4
•
Features:

Measurable quality
Quality can be measured

Recognizable quality
Quality
can be
measured
easily at run

time

Monotonicity
Quality is a non

decreasing
function of time and input
quality

Consistency
Quality is correlated with time and
input quality
Anytime algorithms:
Main characteristics
(2/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
5

Diminishing returns
I
mprovement
is larger at earlier
stage, and
diminishes
with time

Interruptibility
Can
be interrupted and provide a
valid result

Preemptability
Can
suspended and resumed with
minimal overhead
•
Real

time applications

Available time is limited

Complexity of input data might change
•
Limited computational power

Constraints on accuracy
•
Unpredictable situations

Available time can vary

Computational power might change
Anytime algorithms:
Main fields of application (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
6
•
Object detection/recognition
•
Path/motion planning
•
Signal processing
•
Classification
•
Computational intelligence
•
Operational research
•
Automatic controls
Anytime algorithms:
Main fields of application
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
7
•
Object detection/recognition:

Obstacle tracking

Image alignment

Novelty detection
•
Path/motion planning:

3

D reconstruction
Anytime algorithms
for real

time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
8
•
Object detection/recognition:

Obstacle tracking

Image alignment

Novelty detection
•
Path/motion planning:

3

D reconstruction
Anytime algorithms
for real

time image
p
rocessing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
9
Object
detection/recognition:
Obstacle tracking (1/2)
•
Real

time obstacle tracking:

Automotive applications

Based on range images
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
10
Object detection/recognition:
Obstacle tracking
(2/2)
•
Problem features:

Hard
real

time
requirements
Safety

Problem variables might change:
Complexity of the scene
Computational power
Resolution of imaging device
Time
available

High
specificity not
required
False positives can be tolerated
False negatives can
not
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
11
•
Object tracking problem:

to
matching
components at time
(
𝑡
−
1
)
components at time
𝑡
Object detection/recognition:
Object tracking method (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
12
•
Matching not
1
to
1

Occlusions

Splitting

Segmentation errors

Change of scene
Object detection/recognition:
Object
tracking
method (2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
13
•
Object matching algorithm:

Object detection at time
𝑡
−
1
and
𝑡

Each range level considered separately

Possible groups of objects are computed
at
𝑡
−
1
and
𝑡
Adjacency graphs

For each group, all possible subgroups are computed

All possible
subgroups
at
𝑡
−
1
are matched with all
possible subgroups at
𝑡
Matches chosen
based
on similarity
of shapes and positions
Object detection/recognition:
Object
tracking
method (3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
14
•
Anytime implementation:

Solution of a reduced problem
1
to
matching

Subsequent solution of optimal problem
to
matching
Each iteration solves a different range level
If (and until) time is available
Object detection/recognition:
Anytime implementation (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
15
•
Reduced
1
to
matching

Works on different range levels separately

Components at time
(
𝑡
−
1
)
merged in groups
Subgroups are not computed

Each group is matched to
components at time
𝑡
Object detection/recognition:
Anytime implementation
(2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
16
•
Optimal
to
matching

Current reduced
matchings
are ranked

Ranking based on similarity of shapes and positions

Iteration on
matchings

Selection of worst matching

Refinement using optimal matching
Object detection/recognition:
Anytime implementation
(3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
17
Object detection/recognition:
Results of obstacle tracking using anytime algorithms
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
18
(a)
t

1
t
(b) Restricted matching
(c) Restricted and optimal matching
•
Object detection/recognition

Obstacle tracking

Image alignment

Novelty detection
•
Path/motion planning

3

D reconstruction
Anytime algorithms
for real

time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
19
Object
detection/recognition:
Image alignment (1/2)
•
Real

time image alignment:

Basic operation in computer vision applications
•
Problem features:

Time

consuming

Real

time
requirements
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
20
•
Classical direct image alignment:

Alignment described by a set of transformation
parameters
φ

Similarity defined as function of the transformation
parameters:
𝐷
=
𝑓
φ

Find the optimal transformation parameters that
maximize the similarity
Object detection/recognition:
Image alignment
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
21
•
Image alignment algorithm:

Optimization based on steepest descent algorithm

Improvement guided by
the gradient of the similarity
measure:
Mean Squared Difference
Mutual
Information
Object detection/recognition:
Image alignment
method
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
22
•
Anytime implementation:

Measure of similarity as a function of the number of
pixels
𝑝
used:
𝐷
=
𝑓
(
φ
,
𝑝
)

Function used
to compute the number of pixels to
use
Implemented
as look

up
table, trained
offline
Based on desired accuracy and the current magnitude of the
gradient

More pixels are incrementally considered during the
computation
Object
detection/recognition:
Anytime implementation
(1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
23
•
Anytime algorithm:

The number of pixels to use in the first step is computed
Based on known desired accuracy
Based on first gradient computation

The optimal transformation parameters are computed
Steepest descent optimization

The gradient of the similarity measure is computed

The number of pixels to use is increased

The transformation parameter are further optimized
Object
detection/recognition:
Anytime implementation
(2/3
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
24
•
Number of pixels can
be tuned

Required accuracy
•
More
pixels can be used

If time is available

If greater accuracy is
required
Object
detection/recognition:
Anytime implementation
(3/3
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
25
•
Object detection/recognition

Obstacle tracking

Image alignment

Novelty detection
•
Path/motion planning

3

D reconstruction
Anytime algorithms
for real

time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
26
•
Real

time novelty detection

Automotive applications
•
Problem features:

No high specificity required

Hard real

time constraints
Object
detection/recognition:
Novelty detection (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
27
•
Novelty detection problem:

Two

class classification problem

Segmenting from the image everything not recognized
Datasets of examples
Object detection/recognition:
Novelty detection
(2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
28
•
Outline of the method:

Extraction of features at each time step
𝑡

Feature reduction
Multiple Discriminant Analysis

Computation of similarity of current
scene
to each
example up to time
(
𝑡
−
1
)
SVM

based classifier

Computation of aggregated similarity
Linear combination
Object
detection/recognition:
Novelty detection method
(1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
29
•
If
result exceeds threshold

Scene
is not
novel

Example discarded
•
If result does not exceed threshold

Scene is novel

Example list is updated

Classifier
re

trained
online with new
example
Object
detection/recognition:
Novelty detection method
(2/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
30
•
Anytime implementation:

Computation of similarity limited
Interrupted when threshold is exceeded

Most significant examples processed first
Example that exceeds threshold moved at beginning of time
series

Premature stopping generates false alarms
Does not cause missed detections
Object
detection/recognition:
Anytime implementation
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
31
Object
detection/recognition:
Results of anytime novelty detection
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
32
•
Object detection/recognition

Obstacle tracking

Image alignment

Novelty detection
•
Path/motion planning

3

D reconstruction
Anytime algorithms
for real

time image processing
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
33
•
3

D reconstruction

Based on stereoscopic images

Preliminary step for motion
planning

Automotive applications
•
Problem features:

Real

time constraints
Path/Motion planning:
3

D reconstruction for motion planning
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
34
•
Point

based 3

D stereoscopic reconstruction:

Point extraction using Speeded Up Robust Features
(SURF):
Designed for speed and robustness
Extraction of points from left image
Extraction of points from right image

Matching of extracted points
Based on distance between features vectors

Triangulation
Based on camera calibration data
Path/Motion
planning:
3

D
reconstruction method
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
35
•
Anytime implementation:

Novel task handling method
Allows
interruptibility
Leaves computation in a consistent state

Anytime SURF algorithm for point extraction and
matching
Divided in different scale spaces
Iterative
Path/Motion
planning:
Anytime implementation (1/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
36
•
Each task divided in Main Part (MP) and Exception
Part (EP)

MP contains code
E.g. point extraction

EP handles exception and consistency
Path/Motion
planning:
Anytime implementation
(2/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
37
•
Two

level scheduler

EPs are scheduled
to start as late as possible

MPs
are scheduled to end as early as possible
•
Anytime SURF
algorithm:

Subdivision in different scale spaces
The coarsest scale space is processed first

For each scale space the tasks are performed:
Extraction of points from left image
Extraction of points from right image
Point matching
Triangulation

Task are divided in MPs and EPs
Path/Motion
planning:
Anytime implementation
(3/3)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
38
•
Results of 3

D triangulation:
Path/Motion
planning:
Results of anytime 3

D reconstruction
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
39
•
Anytime algorithms for image processing
applications were reviewed:

Object detection
Obstacle tracking
Image alignment
Object classification (e.g. novelty detection)

Motion planning
3

D reconstruction of the scene
Conclusions (1/2)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
40
•
Important field of application in automotive
scenarios

High accuracy often not necessary

Hard

real time constraints
•
Anytime algorithms for real

time image processing
are still not very explored
Conclusions (2/2
)
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
41
References
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
42
1.
S
.
Zilberstein
,
“Using
Anytime
Algorithms
in
Intelligent
Systems”,
AI
Magazine,
Vol
.
17
,
No
.
3
,
1996
,
pp
.
73

83
.
2.
T
.
Dean
,
M
.
S
.
Boddy
,
“An
Analysis
of
Time

Dependent
Planning”,
In
Proceedings
of
the
Seventh
National
Conference
on
Artificial
Intelligence,
1988
,
Tom
M
.
Mitchell,
Reid
G
.
Smith
(Eds
.
)
pp
.
49

54
.
3.
M
.
S
.
Boddy
,
T
.
Dean,
“Solving
Time

Dependent
Planning
Problems”
.
Technical
Report
.
Brown
University,
Providence,
RI,
USA,
1989
.
4.
J
.
Grass
,
S
.
Zilberstein
,
"Programming
with
Anytime
Algorithms",
In
Proceedings
of
the
IJCAI

95
Workshop
on
Anytime
Algorithms
and
Deliberation
Scheduling
,
1995
,
pp
.
22

27
.
5.
K
.
Sobottka
,
H
.
Bunke
,
"Any

time
behavior
for
obstacle
tracking
,“
In
Proceedings
of
the
1999
IEEE/IEEJ/JSAI
International
Conference
on
Intelligent
Transportation
Systems,
1999
,
pp
.
368

373
.
6.
R
.
Brooks,
T
.
Arbel
,
D
.
Precup
,
“Fast
image
alignment
using
anytime
algorithms”,
In
Proceedings
of
the
20
th
international
joint
conference
on
Artifical
intelligence
(IJCAI'
07
),
2007
,
Rajeev
Sangal
,
Harish
Mehta,
and
R
.
K
.
Bagga
(Eds
.
)
.
Morgan
Kaufmann
Publishers
Inc
.
,
San
Francisco,
CA,
USA,
pp
.
2078

2083
.
7.
B
.
Sofman
,
J
.
A
.
Bagnell
,
A
.
Stentz
,
"Anytime
online
novelty
detection
for
vehicle
safeguarding,"
2010
IEEE
International
Conference
on
Robotics
and
Automation
(ICRA
),
2010
,
pp
.
1247

1254
.
8.
K
.
Wetzelsberger
,
T
.
Pfannschmidt
,
T
.
Ihme
,
"
Real

time
image
processing
for
motion
planning
based
on
realistic
sensor
data,"
2010
IEEE
International
Workshop
on
Robotic
and
Sensors
Environments
(ROSE
)
,
2010
,
pp
.
1

6
.
Wednesday, March 20, 2013
Anytime Algorithms for Real Time Image Processing
43
Thank you for your attention!
Comments 0
Log in to post a comment