Applied Machine Vision Session 9 Session 9

coatiarfΤεχνίτη Νοημοσύνη και Ρομποτική

17 Οκτ 2013 (πριν από 3 χρόνια και 7 μήνες)

77 εμφανίσεις

Applied Machine Vision
Session 9
Session 9
presented by
Xie Ming
Associate Professor
SchoolofMechanical&AerospaceEngineering
School

of

Mechanical

&

Aerospace

Engineering
1
http://www.ntu.edu.sg/home/mmxie
Machine Vision Applications:
Special Case Studies
P1
ISbilii

P
art
1
:
I
mage
S
ta
bili
zat
i
on

Part2
:TargetTrackinginImage
Part

2
:

Target

Tracking

in

Image
•Part 3
: Target Tracking in Space
2
http://www.ntu.edu.sg/home/mmxie
Part 1:
Part 1:
Image Stabilization
Image Stabilization
3
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Si

S
cenar
i
o

ProblemStatement
Problem

Statement
•Image Center Tracking
•Image Center Control
•Q & A
4
http://www.ntu.edu.sg/home/mmxie
Scenario
5
http://www.ntu.edu.sg/home/mmxie
What is the process of acquiring a
digital image?
Y

Y
our answer:
?
?
?
DiitlI
6
http://www.ntu.edu.sg/home/mmxie
Di
g
it
a
l

I
mage
What is the process of acquiring a
digital image?
Thfiiiti

Th
e process o
f

i
mage acqu
i
s
iti
on
involves:
a) capture of lights
b)sensingoflights
Captureof
b)

sensing

of

lights
c) digitization
Capture

of
Lights
Sensingof
Sensing

of
Lights
Digitization
DiitlI
7
http://www.ntu.edu.sg/home/mmxie
Di
g
it
a
l

I
mage
What is the necessary condition
for an image to remain still?
Y

Y
our answer:
a)
a)
8
http://www.ntu.edu.sg/home/mmxie
What is the necessary and sufficient
condition for an image to remain still?
Ndffiidii

N
ecessary an
d
su
ffi
c
i
ent con
di
t
i
on:
Therelativemotionbetweenacameraandasceneiszero
The

relative

motion

between

a

camera

and

a

scene

is

zero
.
R, T
9
http://www.ntu.edu.sg/home/mmxie
What is another sufficient condition for
an image to remain still?
Y

Y
our answer:
a)
a)
10
http://www.ntu.edu.sg/home/mmxie
What are the sufficient conditions
for an image to remain still?
Sffiidii

S
u
ffi
c
i
ent con
di
t
i
on:
Bothacameraandasceneremainstatic
Both

a

camera

and

a

scene

remain

static
.
R, T
11
http://www.ntu.edu.sg/home/mmxie
Problem Statement
12
http://www.ntu.edu.sg/home/mmxie
What is a sequence of stable
images?
Y

Y
our answer:
13
http://www.ntu.edu.sg/home/mmxie
What is a sequence of stable
images?
Afbliiihihhi

A
sequence o
f
sta
bl
e
i
mages
i
s a sequence,
i
n w
hi
c
h
t
h
e
i
mage
center remains unchanged.
14
http://www.ntu.edu.sg/home/mmxie
What are the conditions which
cause unstable images?
Y

Y
our answer:
a)
a)
b)
R, T
15
http://www.ntu.edu.sg/home/mmxie
What are the conditions which
cause unstable images?
Ilhi

I
n genera
l
, t
h
ere two ma
i
n cause:
a)randommotionofacamera
a)

random

motion

of

a

camera
b) random motion of a scene
R, T
16
http://www.ntu.edu.sg/home/mmxie
What are the possible causes
behind a scene’s random motion?
Y

Y
our answer:
R, T
17
http://www.ntu.edu.sg/home/mmxie
What are the possible causes
behind a scene’s random motion?
Thiblldb

Th
e poss
ibl
e causes cou
ld

b
e:
a)wind
a)

wind
d) wave of water surface
c) vibration of floor
R, T
18
http://www.ntu.edu.sg/home/mmxie
What are the possible causes
behind a camera’s random motion?
Y

Y
our answer:
R, T
19
http://www.ntu.edu.sg/home/mmxie
What are the possible causes
behind a camera’s random motion?
Thiblild

Th
e poss
ibl
e causes
i
nc
l
u
d
e:
a
)
hand’s tremblin
g

)g
b) random motions of fixed
mountingplatforms
mounting

platforms
c) random motions of mobile
R, T
mounting platforms
20
http://www.ntu.edu.sg/home/mmxie
What will be the problem of acquiring a
sequence of stable images?
Y

Y
our answer:
Capture of
Lights
Sensing of
Lights
Digitization
?
21
http://www.ntu.edu.sg/home/mmxie
What will be the problem of acquiring a
sequence of stable images?
Thblithttk

Th
e pro
bl
em
i
s
t
o
h
ow
t
o
t
rac
k
, or
control, the shifts of the image
center, and output the images
centeredattheimagecenter
centered

at

the

image

center
.
Capture of
Lights
Sf
S
ensing o
f
Lights
Digitization
Image Center
Tracking
Image Center
Control
22
http://www.ntu.edu.sg/home/mmxie
Stable ImagesStable Images
Image Center Tracking
23
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Tddfllhhfhi

T
o
d
etect an
d

f
o
ll
ow t
h
e c
h
ange o
f
t
h
e
i
mage center:
……
Sub-image Extraction
……
24
http://www.ntu.edu.sg/home/mmxie
Implementation
Implementation
Camera
Update Sub-image
Sub-image Scaling
Image Scaling
Sub-image Matching
Output
25
http://www.ntu.edu.sg/home/mmxie
Image Center Control
26
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Tlhiblihdif

T
o contro
l
t
h
e
i
mage center
b
y cance
li
ng out t
h
e ran
d
om mot
i
on o
f
a
camera, or a scene:
RT
R
,
T
27
http://www.ntu.edu.sg/home/mmxie
Implementation
Implementation
UdtRf
U
p
d
a
t
e
R
e
f
erence
Image
+
Camera
+
-
Motion
Execution
Motion
Control
28
http://www.ntu.edu.sg/home/mmxie
Q & A
29
http://www.ntu.edu.sg/home/mmxie
Q & A
Q & A
Plibilifi

P
ropose your so
l
ut
i
on to sta
bili
ze a sequence o
f

i
mages.

Describeanddiscussyoursolutionasdetailaspossible
Describe

and

discuss

your

solution

as

detail

as

possible
.
30
http://www.ntu.edu.sg/home/mmxie
Part 2:
Part 2:
Target Tracking in Image
Target Tracking in Image
31
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Si

S
cenar
i
o

ProblemStatement
Problem

Statement
•Measurement of Target Motion
•Prediction of Target Motion
•Q & A
32
http://www.ntu.edu.sg/home/mmxie
Scenario
Video
33
http://www.ntu.edu.sg/home/mmxie
What is the necessary condition
for a target to move in image?
Y

Y
our answer:
34
http://www.ntu.edu.sg/home/mmxie
What is the necessary and sufficient
condition for a target to move in image?
Ndffiidii

N
ecessary an
d
su
ffi
c
i
ent con
di
t
i
on:
Therelativemotionbetweenacameraandatargetisnotzero
The

relative

motion

between

a

camera

and

a

target

is

not

zero
.
35
http://www.ntu.edu.sg/home/mmxie
What is another sufficient condition for
a target to move in image?
Y

Y
our answer:
36
http://www.ntu.edu.sg/home/mmxie
What is another sufficient condition for
a target to move in image?
Sffiidii

S
u
ffi
c
i
ent con
di
t
i
on:
Acameraoratargetismoving
A

camera
,
or

a

target
,
is

moving
.
37
http://www.ntu.edu.sg/home/mmxie
Problem Statement
38
http://www.ntu.edu.sg/home/mmxie
Will a target’s image remain still if its
motion with respect to a camera is not
motion with respect to a camera is not
zero?
Y

Y
our answer:
R, T
39
http://www.ntu.edu.sg/home/mmxie
What are the properties of a
target’s motion in an image plane?
Y

Y
our answer:
a)
a)
b)
R, T
c)
40
http://www.ntu.edu.sg/home/mmxie
What are the properties of a
target’s motion in an image plane?
Ilthtiftt

I
n genera
l
,
th
e mo
ti
on o
f
a
t
arge
t

in an image plane will include:
a) translation (i.e. linear motion)
b)rotation(ieangularmotion)
R, T
b)

rotation

(i
.
e
.
angular

motion)
c) scaling
41
http://www.ntu.edu.sg/home/mmxie
How to describe a target’s linear
motion over a sequence of images?
Y

Y
our answer:
a)
u
a)
b)
u
v
42
http://www.ntu.edu.sg/home/mmxie
How to describe a target’s linear
motion over a sequence of images?
Iilttti

I
n an
i
mage p
l
ane, a
t
arge
t
mo
ti
on
can be describe by:
u
a) path (i.e. positions)
b)trajectory(ievelocitiesand
u
v
b)

trajectory

(i
.
e
.
velocities

and

accelerations)
43
http://www.ntu.edu.sg/home/mmxie
How to describe a target’s linear
motion over a sequence of images?
Bfi
u

B
ecause a sequence o
f

i
mages
corresponds to a set of discrete
times, a target’s linear motion in
animageplaneisdescribedb
v
an

image

plane

is

described

b
y:
a) a set of discrete positions
b) a set of average velocities
c) a set of average accelerations
Time Positions Velocities Accelerations
1
),(
11
vu

),(
11
vu
&&

),(
11
vu
&&&&

2
),(
22
vu

),(
22
vu
&&

),(
22
vu
&&&&

… … … …
k
),(
kk
vu

),(
kk
vu
&&

),(
kk
vu
&&&&


44
http://www.ntu.edu.sg/home/mmxie
How to describe a target’s angular
motion over a sequence of images?
Y

Y
our answer:
a)
u
a)
b)
u
v
c)
45
http://www.ntu.edu.sg/home/mmxie
How to describe a target’s angular
motion over a sequence of images?
Ofi
u

O
ver a sequence o
f

i
mages, a
target’s angular motion can be
described by:
v
θ
愩⁡⁳整映摩獣牥瑥⁡湧畬慲a
p
潳楴楯湳
p
戩⁡⁳整映摩獣牥瑥⁡湧畬慲
癥汯捩瑩敳
Time Positions Velocities Accelerations
1
1
θ

1
θ
&

1
θ
&&

velocities
c) a set of discrete angular
1
1
2
2
θ

2
θ
&

2
θ
&&

… … … …
k
k
θ
k
θ
&
k
θ

accelerations
k

k
θ
=
k
θ
=
k
θ
=
=
46
http://www.ntu.edu.sg/home/mmxie
What will be the motion model of a
target in an image plane?
Y

Y
our answer:
u
u
v
θ

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
What will be the motion model of a
target in an image plane?
Ofi’idlb

O
ver a sequence o
f

i
mages, a target

s mot
i
on mo
d
e
l
can
b
e
described by:
Time Positions Velocities Accelerations
1
),,(
111
θ
vu

),,(
111
θ
&
&&
vu

),,(
111
θ
&&
&&&&
vu

2
),,(
222
θ
vu

),,(
222
θ
&
&&
vu

),,(
222
θ
&&
&&&&
vu

… … … …
k
),,(
kkk
vu
θ

),,(
kkk
vu
θ
&
&&

),,(
kkk
vu
θ
&&
&&&&


u
v
θ

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
What is the problem of target
tracking in image?
Y

Y
our answer:
C
UdtTt
C
amera
U
p
d
a
t
e
T
arge
t
u
Target Matching
v
θ
?
?
啰摡瑥U
䵯瑩潮⁍潤敬

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
What is the problem of target
tracking in image?
Thblftttkii

Th
e pro
bl
em o
f

t
arge
t

t
rac
ki
ng
i
n
image is to measure, or predict, a
target’s motion model over a
seqenceofimages
Camera
Update Target
seq
u
ence

of

images
.
Target Matching
u
v
Motion
Measrement
Motion
Pditi
v
θ
䵥慳
×
牥浥湴
P


c


啰摡瑥
啰摡瑥
=
䵯瑩潮⁍潤敬

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
Measurement of Target Motion
51
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Camera
Update Target
u
Target Matching
Ifmatchingissuccessful
u
v
θ
䵯瑩潮M
䵥慳畲敭敮M

=
浡瑣桩湧
=

=
獵捣敳獦畬
θ
啰摡瑥U
䵴iM摬
M
o

潮o
M
o
d
e
l

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
Equations
Equations







=
=
kk
kk
vv
uu
ˆ
ˆ




−=
−=


)(
)(
1
1
kkk
kkk
vvv
uuu
&
&




+−=
+
−=
−−
−−
)2(
)2(
21
21
kkkk
kkkk
vvvv
uuuu
&&
&&


=
kk
θθ
ˆ


−=

)(
1kkk
θθθ
&


+−=
−−
)2(
21kkkk
θθθθ
&&
Time Positions Velocities Accelerations
1
),,(
111
θ
vu

),,(
111
θ
&
&&
vu

),,(
111
θ
&&
&&&&
vu

2
),,(
222
θ
vu

),,(
222
θ
&
&&
vu

),,(
222
θ
&&
&&&&
vu

… … … …
k
),,(
kkk
vu
θ

),,(
kkk
vu
θ
&
&&

),,(
kkk
vu
θ
&&
&&&&


53
http://www.ntu.edu.sg/home/mmxie
Prediction of Target Motion
54
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Camera
Update Target
Target Matching
Ifmatchingfails
u
v
Motion
Pditi
If

matching

fails
v
θ
P


c


啰摡瑥
啰摡瑥
=
䵯瑩潮⁍潤敬

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
Equations
Equations


+
=

k
i
k
k
u
u
u
1
1
&


=

−1
1
k
i
k
u
u
&&&&




+=
+



=

k
i
k
k
i
i
k
k
v
k
vv
u
k
u
u
1
1
1
1
&



+=
+=


k
k
k
kkk
vvv
uuu
&&&&
&&&&
1
1




=




=
1
2
1
2
k
i
k
i
i
k
vv
u
k
u
&&&&





+=



=
k
ikk
i
i
k
k
k
k
1
1
1
1
θθθ
&



+=
−kkk
k
k
k
θθθ
&&&&
1
1





=




=
1
2
2
1
2
k
i
k
i
i
k
k
k
θ
θ
&&&&
Time Positions Velocities Accelerations


=i
k
1



=2
2
i
i
k
k
1
),,(
111
θ
vu

),,(
111
θ
&
&&
vu

),,(
111
θ
&&
&&&&
vu

2
),,(
222
θ
vu

),,(
222
θ
&
&&
vu

),,(
222
θ
&&
&&&&
vu

… … … …
k
),,(
kkk
vu
θ

),,(
kkk
vu
θ
&
&&

),,(
kkk
vu
θ
&&
&&&&

56
http://www.ntu.edu.sg/home/mmxie

Q & A
57
http://www.ntu.edu.sg/home/mmxie
Q & A
Q & A
Plifkiifi

P
ropose your so
l
ut
i
on o
f
trac
ki
ng a target
i
n a sequence o
f

i
mages.

Describeandexplainyoursolutionindetail
Describe

and

explain

your

solution

in

detail
.
58
http://www.ntu.edu.sg/home/mmxie
Part 3:
Part 3:
Target Tracking in Space
Target Tracking in Space
59
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Si

S
cenar
i
o

Camera

sMotionModel
Cameras

Motion

Model
•Update of Angular Acceleration
•Update of Angular Velocity
•Update of Angular Position
•Q & A
60
http://www.ntu.edu.sg/home/mmxie
Scenario
Video
61
http://www.ntu.edu.sg/home/mmxie
What is the necessary and sufficient
condition for a target to be seen by a
condition for a target to be seen by a
camera?
Y

Y
our answer:
62
http://www.ntu.edu.sg/home/mmxie
What is the necessary and sufficient
condition for a target to be seen by a
condition for a target to be seen by a
camera?
Ndffiidii

N
ecessary an
d
su
ffi
c
i
ent con
di
t
i
on:
Atargetiswithinthefieldofviewofacamera
A

target

is

within

the

field

of

view

of

a

camera
.
63
http://www.ntu.edu.sg/home/mmxie
What is another sufficient condition for
a target to be seen by a camera?
Y

Y
our answer:
64
http://www.ntu.edu.sg/home/mmxie
What is another sufficient condition for
a target to be seen by a camera?
Sffiidii

S
u
ffi
c
i
ent con
di
t
i
on:
Acameraismountedonapan
-
tiltplatformwithfullrange
A

camera

is

mounted

on

a

pan
tilt

platform

with

full

range
.
65
http://www.ntu.edu.sg/home/mmxie
Problem Statement
66
http://www.ntu.edu.sg/home/mmxie
How to describe the viewing
direction of a target in space?
Y

Y
our answer:
a)
a)
b)
67
http://www.ntu.edu.sg/home/mmxie
How to describe the viewing
direction of a target in space?
Thiiditiftti
ImagePlane

Th
e v
i
ew
i
ng
di
rec
ti
on o
f
a
t
arge
t

i
n
a camera’s coordinate system can
be described by two angles:
Z
α
䥭慧I
=
偬慮P
愩⁶敲瑩捡氠慮杬攠潦⁶楥a
Y
戩⁨潲楺潮瑡氠慮杬攠潦⁶楥b
Z
Z
β
䥭慧攠偬慮I
Z
X
Z
X

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
X
Y
How to describe the viewing
direction of a target in space?
Eifillfi

E
quat
i
on o
f
vert
i
ca
l
ang
l
e o
f
v
i
ew:
Z
Image Plane
D
)
(
Z
Y
α
f
D
vv
y
×−
=
)
(
)tan(
0
α
)()tan(
0
vvk−×=
α
α
Z
69
http://www.ntu.edu.sg/home/mmxie
X
Y
How to describe the viewing
direction of a target in space?
Eifhillfi

E
quat
i
on o
f

h
or
i
zonta
l
ang
l
e o
f
v
i
ew:
β
䥭慧攠偬慮I
Z
β
f
D
uu
x
×−
=
)(
)tan(
0
β
X
)()tan(
0
uuk−×=
β
β
Z
70
http://www.ntu.edu.sg/home/mmxie
X
Y
How to keep a target along the
optical axis of a camera?
Y
RT

Y
our answer:
R
,
T
71
http://www.ntu.edu.sg/home/mmxie
How to keep a target along the
optical axis of a camera?
Idtkttlth
ImagePlane

I
n or
d
er
t
o
k
eep a
t
arge
t
a
l
ong
th
e
optical axis of a camera, we must
control the pan-tilt platform so that:
Z
α
䥭慧I
=
偬慮P
愩⁶敲瑩捡氠慮杬攠潦⁶楥眠楳⁺敲a
Y
戩⁨潲楺潮瑡氠慮杬攠潦⁶楥眠楳⁺敲a
Z
Z
β
䥭慧攠偬慮I
Z
X
Z
X

桴瑰㨯⽷睷⹮瑵⹥摵⹳术桯浥⽭浸楥
X
Y
What is the motion model of a
camera, which tracks a target?
Y

Y
our answer:
Camera
Pan
Pan
Tilt
73
http://www.ntu.edu.sg/home/mmxie
What is the motion model of a
camera, which tracks a target?
Ihi’idlif

I
n t
hi
s case, a camera

s mot
i
on mo
d
e
l
cons
i
sts o
f
:
a) a set of discrete positions of the angles of view
b)asetofdiscretevelocitiesoftheanglesofview
Camera
P
b)

a

set

of

discrete

velocities

of

the

angles

of

view
c) a set of discrete accelerations of the angles of view
P
an
Tilt
Time Positions Velocities Accelerations
1
),(
11
β
α
=
),(
11
βα
&
&

),(
11
βα
&&
&&

2
),(
22
β
α
=
),(
22
βα
&
&

),(
22
βα
&&
&&

… … … …
k
),(
kk
β
α

),(
kk
βα
&
&

),(
kk
βα
&&
&&


74
http://www.ntu.edu.sg/home/mmxie
What is the problem of tracking a
target in space?
Y

Y
our answer:
Camera
Update Target
Pan
Tilt
Tar
g
et Matchin
g
gg
?
?
?
?
Update
?
75
http://www.ntu.edu.sg/home/mmxie
Update
Motion Model
?
What is the problem of tracking a
target in space?
Thblftkitti

Th
e pro
bl
em o
f

t
rac
ki
ng a
t
arge
t

i
n
space is to measure, or predict,
the next motion that a camera has
toperforminordertokeepa
to

perform

in

order

to

keep

a

target along the optical axis.
Camera
Pan
Update Target
Tilt
Target Matching
Motion
Measurement
Motion
Prediction
Update
Motion Model
Motion Control
76
http://www.ntu.edu.sg/home/mmxie
Motion Measurement
77
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Camera
Pan
Update Target
Tilt
Target Matching
Ifmatchingissuccessful
Motion
Measurement
If

matching

is

successful
Update
Motion Model
Motion Control
78
http://www.ntu.edu.sg/home/mmxie
Equations
Equations



−×=
−×=
))(arctan(
))(arctan(
0
0
uuk
vvk
kk
kk
β
α
β
α



−=
−=


1
1
k
k
k
kkk
βββ
ααα
&
&



+−=
+−=


−−
21
21
2
2
k
k
k
k
kkkk
ββββ
αααα
&&
&&


Time Positions Velocities Accelerations
1
),(
11
β
α
=
),(
11
βα
&
&

),(
11
βα
&&
&&

2
),(
22
β
α
=
),(
22
βα
&
&

),(
22
βα
&&
&&

… … … …
k
),(
kk
β
α

),(
kk
β
α
&
&

),(
kk
β
α
&&
&&


79
http://www.ntu.edu.sg/home/mmxie
Motion Prediction
80
http://www.ntu.edu.sg/home/mmxie
Principle
Principle
Camera
P
Update Target
P
an
Tilt
Target Matching
Motion
Prediction
If matching fails
Prediction
Update
MtiMdl
Motion Control
M
o
ti
on
M
o
d
e
l
81
http://www.ntu.edu.sg/home/mmxie
Equations
Equations

−1
1
k

k
1






=



=
1
2
1
2
1
k
i
ik
k
α
α
&&&&
&&&&



+
=
+=

k
k
k
kkk
β
β
β
ααα
&&&&
&&&&
1
1





+=


=

k
i
ikk
k
1
1
1
1
α
α
α
&
&




=

=2
2
1
i
ik
k
β
β

+

k
k
k
E
E
E
1
°
°
¯
?
¦

?
i
ikk
k
1
1
1
β
β
β
θi浥⁐潳楴楯湳mVe汯捩瑩敳l䅣捥汥牡瑩潮猠
1
)
,
(
1
1
β
α
)
(
1
1
β
α
&
&
)
(
1
1
β
α


1

)
,
(
1
1
β
α
=
)
,
(
1
1
β
α
=
)
,
(
1
1
β
α
=

),(
22
β
α
=
),(
22
βα
&
&

),(
22
βα
&&
&&













k
),(
kk
β
α

),(
kk
βα
&
&

),(
kk
βα
&&
&&


82
http://www.ntu.edu.sg/home/mmxie
Q & A
83
http://www.ntu.edu.sg/home/mmxie
Q & A
Q & A
Plifkii

P
ropose your so
l
ut
i
on o
f
trac
ki
ng a target
i
n space.

Describeandexplainyoursolutionindetail
Describe

and

explain

your

solution

in

detail
.
84
http://www.ntu.edu.sg/home/mmxie
End
85
http://www.ntu.edu.sg/home/mmxie