Applied Machine Vision Session 8 Session 8

coatiarfΤεχνίτη Νοημοσύνη και Ρομποτική

17 Οκτ 2013 (πριν από 3 χρόνια και 7 μήνες)

82 εμφανίσεις

Applied Machine Vision
Session 8
Session 8
presented by
Xie Ming
Associate Professor
SchoolofMechanical&AerospaceEngineering
School

of

Mechanical

&

Aerospace

Engineering
1
http://www.ntu.edu.sg/home/mmxie
Machine Vision Applications:
General Case Studies
P1
Vii
iddIi

P
art
1
:
Vi
s
i
on-gu
id
e
d

I
nspect
i
on

Part2
:Vision
-
guidedManipulation
Part

2
:

Vision
guided

Manipulation
•Part 3
: Vision-guided Navigation
2
http://www.ntu.edu.sg/home/mmxie
Part 1:
Part 1:
Vision
-
guided Inspection
Vision
guided Inspection
3
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Ii

I
nspect
i
on

InspectionforQualityControl
Inspection

for

Quality

Control
•Inspect for Security Control
•Q & A
4
http://www.ntu.edu.sg/home/mmxie
Inspection
5
http://www.ntu.edu.sg/home/mmxie
What can an image tell?
What can an image tell?
Y

Y
our answer:
a)
a)
b)
c)
6
http://www.ntu.edu.sg/home/mmxie
What can an image tell?
What can an image tell?
Pl“iihhhdd”I

P
eop
l
e says,

an
i
mage
i
s wort
h
more t
h
an t
h
ousan
d
wor
d
s

.
I
n
general, an image can give us information about:
a) Geometry
b) Appearance
c)Configuration
c)

Configuration
7
http://www.ntu.edu.sg/home/mmxie
What constitute a geometry?
What constitute a geometry?
Y

Y
our answer:
a)
a)
b)
c)
d)
8
http://www.ntu.edu.sg/home/mmxie
What constitute a geometry?
What constitute a geometry?
Ilif

I
n genera
l
, a geometry cons
i
sts o
f
:
a)location
a)

location
b) orientation
c) shape
d) dimension
9
http://www.ntu.edu.sg/home/mmxie
What constitute an appearance?
What constitute an appearance?
Y

Y
our answer:
a)
a)
b)
10
http://www.ntu.edu.sg/home/mmxie
What constitute an appearance?
What constitute an appearance?
Ilif

I
n genera
l
, an appearance cons
i
sts o
f
:
a)distributionofcolors
a)

distribution

of

colors
b) distribution of intensities
11
http://www.ntu.edu.sg/home/mmxie
What constitute a configuration?
What constitute a configuration?
Y

Y
our answer:
a)
a)
b)
12
http://www.ntu.edu.sg/home/mmxie
What constitute a configuration?
What constitute a configuration?
Ilfiiif

I
n genera
l
, a con
fi
gurat
i
on cons
i
sts o
f
:
a)alignment
a)

alignment
b) topology
13
http://www.ntu.edu.sg/home/mmxie
What is inspection?
What is inspection?
Y

Y
our answer:
a)
a

)
14
http://www.ntu.edu.sg/home/mmxie
What is inspection?
What is inspection?
Bdfiiiiihihifihhh

B
y
d
e
fi
n
i
t
i
on,
i
t
i
s a process, w
hi
c
h
ver
ifi
es w
h
et
h
er or not t
h
e
properties and/or behaviors of an entity are conform to standards.
15
http://www.ntu.edu.sg/home/mmxie
Inspection for Quality Control
16
http://www.ntu.edu.sg/home/mmxie
What is quality
What is quality
Y

Y
our answer:
a
)

)
17
http://www.ntu.edu.sg/home/mmxie
What is quality?
What is quality?
•By definition, quality
refers to the degree of
conformityto
conformity

to

specification. It is
inversel
y

p
ro
p
ortional
Quality =
1
1+ Variation
ypp
to variation (i.e.
deviation from
ifii)
spec
ifi
cat
i
on
)
.
18
http://www.ntu.edu.sg/home/mmxie
How to control quality?
How to control quality?
Y

Y
our answer:
a)
a)
b)
Quality =
1
1+ Variation
19
http://www.ntu.edu.sg/home/mmxie
How to control quality?
How to control quality?
Illilil

I
n genera
l
, qua
li
ty contro
l

i
nvo
l
ves two steps:
a)Measureproperties
a)

Measure

properties
b) Compare properties
Measure
Properties
Target
Under
Inspection
Inspection
Compare
Specifications
20
http://www.ntu.edu.sg/home/mmxie
How to control quality?
How to control quality?
El1
Alignment

E
xamp
l
e
1
:
Configuration:

Alignment
•Topology
Properties
Target
Under
Measure
Properties
Inspection
Compare
Specifications
21
http://www.ntu.edu.sg/home/mmxie
How to control quality?
How to control quality?
El2
•Configuration

E
xamp
l
e
2
:
•Geometry
Target
Under
Measure
Properties
Under
Inspection
Compare
Specifications
22
http://www.ntu.edu.sg/home/mmxie
Coplanarity
Presence/Absence
(
Missin
g
Balls
)
(g)
BllPii
(XYdRdilE)
B
a
ll

P
os
i
t
i
on
(X
,
Y
an
d

R
a
di
a
l

E
rror
)
Ball Pitch (Center-to-center distance to
nearest ball
)

)
23
http://www.ntu.edu.sg/home/mmxie
Ball Diameter
Ball Quality or Shape Factor
Grid Position (X and Y)
24
http://www.ntu.edu.sg/home/mmxie
How to control Quality?
How to control Quality?
El3
Color

E
xamp
l
e
3
:
Appearance:

Color
•Intensity
Target
Under
Measure
Properties
Under
Inspection
Compare
Specifications
25
http://www.ntu.edu.sg/home/mmxie
26
http://www.ntu.edu.sg/home/mmxie
How to control quality?
How to control quality?
El4
•Location

E
xamp
l
e
4
:
Geometry:
•Orientation
•Shape
•Dimension
Target
Under
Measure
Properties
Under
Inspection
Compare
Specifications
27
http://www.ntu.edu.sg/home/mmxie
28
http://www.ntu.edu.sg/home/mmxie
Inspection for Security Control
29
http://www.ntu.edu.sg/home/mmxie
What is security?
What is security?
Y

Y
our answer:
a)
a)
30
http://www.ntu.edu.sg/home/mmxie
What is security?
What is security?
•By definition, security
refers to the degree of
conformity to rule. It is
inversely proportional
Security =
1
1+ Committed Threats
to committed threats
(i.e. violation of rule).
31
http://www.ntu.edu.sg/home/mmxie
What are the threats?
What are the threats?
Y

Y
our answer:
a)
a)
b)
Security =
1
1+ Committed Threats
c)
d)
32
http://www.ntu.edu.sg/home/mmxie
What are the threats?
What are the threats?
Vandalism
People
Asset
Terrorism
Criminal
Lieu
Asset
Activism
33
http://www.ntu.edu.sg/home/mmxie
How to control security?
How to control security?
Y

Y
our answer:
a)
a)
b)
Security =
1
1+ Committed Threats
c)
d)
34
http://www.ntu.edu.sg/home/mmxie
How to control security?
How to control security?
1
Deter
Intruders
1
.
Deter
2.Delay
3
Detect
3
.
Detect
4.Detain
35
http://www.ntu.edu.sg/home/mmxie
How to detect threats?
How to detect threats?
Y

Y
our answer:
a)
Vandalism
a)

b)
People
People
Asset
Terrorism
Criminal
Lieu
Activism
36
http://www.ntu.edu.sg/home/mmxie
How to detect threats?
How to detect threats?
Ililil

I
n genera
l
, secur
i
ty contro
l

i
nvo
l
ves two steps:
a)Observeanomalies
a)

Observe

anomalies
b) Analyze anomalies
Observe
Properties
Target
Under
Inspection
Observe
Properties
Inspection
Analyze
Rules
37
http://www.ntu.edu.sg/home/mmxie
How to detect threats?
How to detect threats?
El1

E
xamp
l
e
1
:
VideoFrame
I(t)
M(t)
Video

Frame
Understanding
M(t)
Understanding
B(t)
Motion Image
Understanding
B(t)
Background Image
Moving Object
38
http://www.ntu.edu.sg/home/mmxie
How to detect threats?
How to detect threats?
El2

E
xamp
l
e
2
:
39
http://www.ntu.edu.sg/home/mmxie
Q & A
40
http://www.ntu.edu.sg/home/mmxie
Q
& A
Q
•Name some examples of quality control
•Name some examples of security control
Wire-bond Inspection
Lead Inspection
Solder Inspection
Mark Inspection
41
http://www.ntu.edu.sg/home/mmxie
Part 2:
Part 2:
Vision
-
guided Manipulation
Vision
guided Manipulation
42
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Mili

M
an
i
pu
l
at
i
on

DeterministicSolution
Deterministic

Solution
•Iterative Solution
•Q & A
43
http://www.ntu.edu.sg/home/mmxie
What is manipulation?
What is manipulation?
Y

Y
our answer:
a)
a)
44
http://www.ntu.edu.sg/home/mmxie
What is manipulation?
What is manipulation?
Miliifiddibhd
fh

M
an
i
pu
l
at
i
on
i
s to per
f
orm
i
nten
d
e
d
mot
i
ons
b
y a
h
an
d
-arm
f
or t
h
e
purpose of executing a task.
45
http://www.ntu.edu.sg/home/mmxie
What is motion?
What is motion?
Y

Y
our answer:
a)
y
w
Y
a)
e
z
e
x
e
y
1+i
t
w
e
y
e
z
e
x
i
t
e
z
e
y
e
x
w
X
w
Z
46
http://www.ntu.edu.sg/home/mmxie
What is motion?
What is motion?
Mifhilid/ii

M
ot
i
on re
f
ers to any c
h
ange
i
n
l
ocat
i
on an
d/
or or
i
entat
i
on.
e
z
e
x
e
y
1
+
i
t
w
Y
e
y
e

i
t
1
+
i
t
e
z
e
y
e

e
y
e
z
w
X
w
Z
47
http://www.ntu.edu.sg/home/mmxie
How to determine intended motion?
How to determine intended motion?
Y

Y
our answer:
a)
a)
b)
c)
48
http://www.ntu.edu.sg/home/mmxie
How to determine intended motion?
How to determine intended motion?
Ilhh

I
n genera
l
, t
h
ere are t
h
ree ways:
a)manually
a)

manually
b) automatically
Perception
By
machine
c) autonomously
Decision
By
human
By
machine
B
By
machine
Action
B
y
machine
By

machine
49
http://www.ntu.edu.sg/home/mmxie
How to determine intended motion?
El
Analogue
image
Digital
image

E
xamp
l
e:
Step 1: Image sensing
by camera(s).
Step 2: Image
digitization.
Step 3: Image processing
& Object Recognition.
Image
Image

Geometry
Step 4: Measurement of object
geometry in 2D or 3D space.
Object
Geometry
Step 5: Motion planning
Motion
Series
50
http://www.ntu.edu.sg/home/mmxie
Step 6: Control of
motion mechanism.
Series
Control
Command
Deterministic Solution
51
http://www.ntu.edu.sg/home/mmxie
What is the problem?
What is the problem?
Y

Y
our answer:
a)
Left Camera Frame
Y
X
r
u
*
a
*
b
*
a
Leftimage
l
u
*b
a)
b)
*
Z
x
y
b
*
a
X
Right Camera Frame
Right image
r
v
l
v
Left

image
c)
Z
Y
x
y
*
a
b
*
* A
WorldFrame
Z
X
Y
* B
52
http://www.ntu.edu.sg/home/mmxie
World

Frame
What is the problem?
What is the problem?
•To use images to automatically determine:
a) where A is,
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
b)whereBisand
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
b)

where

B

is
,
and

Z
Y
x
y
*
a
b
*
c) how to connect A to B.
* A
World Frame
Z
X
Y
* B
53
http://www.ntu.edu.sg/home/mmxie
What should be the input?
What should be the input?
Y

Y
our answer:
a)
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
a)
b)
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
Z
Y
x
y
*
a
b
*
* A
World Frame
Z
X
Y
* B
54
http://www.ntu.edu.sg/home/mmxie
What should be the input?
What should be the input?

Theinputmustbe:
The

input

must

be:
a) Image coordinates of A and B
b) Projection matrices of the two cameras
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
Z
Y
x
y
*
a
b
*
* A
55
http://www.ntu.edu.sg/home/mmxie
World Frame
Z
X
Y
* B
What will be the output?
What will be the output?
Y

Y
our answer:
a)
Left Camera Frame
X
r
u
*
a
*
b
*
a
l
u
*
b
a)
b)
Y
*
x
y
b
*
a
Right Camera Frame
Right image
r
v
*
b
l
v
*
Left image
c)
Z
X
Y
x
*
a
b
*
* A
Z
y
Z
X
Y
* B
56
http://www.ntu.edu.sg/home/mmxie
World Frame
What will be the output?
What will be the output?
•The output will be:
a)SpacecoordinatesofA
a)

Space

coordinates

of

A
b) Space coordinates of B
c
)
Tra
j
ector
y
or
p
ath connectin
g
A to B
)jypg
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
Z
Y
x
y
*
a
b
*
* A
57
http://www.ntu.edu.sg/home/mmxie
World Frame
Z
X
Y
* B
Solution
Solution
Left Camera Frame
Y
X
x
Rightimage
r
u
*
a
*
b
l
v
*
a
Left image
l
u
*b
)
(
v
u
Leftimagecoordinates:
*
Z
x
y
b
*
a
X
*
a
Right Camera Frame
Right

image
r
v
l
)
,
(
l
l
v
u
),(
rr
vu
Left

image

coordinates:
Right image coordinates:
Z
Y
x
y
*
b
*
}12,...,2,1,{==imM
lil
}12,...,2,1,{==imM
rir
Projection matrices:
* A
Z
*
B
World Frame
Z
X
Y
*

B
),,(ZYX
www
3D coordinates of A or B in world frame:
58
http://www.ntu.edu.sg/home/mmxie
Solution (continued)
Solution (continued)
)
(
)
(
1
Q
H
H
H
Y
X
ttw
w



=







)
.
(
)
(
Q
H
H
H
Z
Y
w



=






Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b







−•−
)
(
)
(
)
(
11
3
10
2
9
1
l
l
l
l
l
l
l
l
l
ummummumm
with:
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v














•−•−•−
•−•−•−
•−•−•−
=
)()()(
)()()(
)()()(
)
(
)
(
)
(
11710695
11310291
11710695
11
3
10
2
9
1
rrrrrrrrr
rrrrrrrrr
lllllllll
l
l
l
l
l
l
l
l
l
vmmvmmvmm
ummummumm
vmmvmmvmm
H
Z
Y
x
y
*
a
b
*








4ll
m
v
mu
* A












=
8
4
8
rr
rr
ll
mv
mu
m
v
Q
World Frame
Z
X
Y
* B
59
http://www.ntu.edu.sg/home/mmxie
Iterative Solution
60
http://www.ntu.edu.sg/home/mmxie
What is the problem?
What is the problem?
Y

Y
our answer:
a)
Left Camera Frame
X
r
u
*
a
a
l
u
*
b
a)
Y
*
X
x
y
b
*
a
RightCameraFrame
Right image
r
v
*
*
b
l
v
*
Left image
b
Z
y
X
Y
x
*
a
Right

Camera

Frame
b
*
* A
Z
y
Z
X
Y
* B
61
http://www.ntu.edu.sg/home/mmxie
World Frame
X
What is the problem?
What is the problem?
Tiilldi

T
o use
i
mages to automat
i
ca
ll
y
d
eterm
i
ne:
a)thenextmove
Left Camera Frame
r
u
a
a
l
u
a)

the

next

move
Y
*
X
x
b
*
a
Right image
r
r
v
*
*
b
l
v
*
a
Left image
*
b
Z
y
b
X
Y
x
*
a
Right Camera Frame
b
*
* A
Z
y
Z
X
Y
* B
62
http://www.ntu.edu.sg/home/mmxie
World Frame
X
What should be the input?
What should be the input?
Y

Y
our answer:
a)
Left Camera Frame
r
u
a
a
l
u
a)
Y
*
X
x
b
*
a
Right image
r
r
v
*
*
b
l
v
*
a
Left image
*
b
Z
y
b
X
Y
x
*
a
Right Camera Frame
b
*
* A
Z
y
Z
X
Y
* B
63
http://www.ntu.edu.sg/home/mmxie
World Frame
X
What should be the input?
What should be the input?
Thiiil

Th
e
i
nput
i
s s
i
mp
l
y:
a)ImagecoordinatesofAandB
a)

Image

coordinates

of

A

and

B
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
Z
Y
x
y
*
a
b
*
* A
64
http://www.ntu.edu.sg/home/mmxie
World Frame
Z
X
Y
* B
What will be the output?
What will be the output?
Y

Y
our answer:
a)
LeftCameraFrame
l
u
a)
Left

Camera

Frame
Y
*
X
x
a
Right image
r
u
r
v
*
a
*
b
l
v
*
a
Left image
l
u
*b
*
Z
y
b
*
X
Y
*
a
Right Camera Frame
b
*
* A
Z
x
y
b
Z
Y
* B
65
http://www.ntu.edu.sg/home/mmxie
World Frame
X
What will be the output?
What will be the output?
ThillbhifhihfAB

Th
e output w
ill

b
e t
h
e ser
i
es o
f
moves, w
hi
c
h
converges
f
rom
A
to
B
.
Left Camera Frame
u
a
l
u
Y
*
X
x
b
*
a
Right image
r
u
r
v
*
a
*
b
l
v
*
a
Left image
*b
Z
y
b
*
X
Y
x
*
a
Right Camera Frame
b
*
* A
Z
x
y
Z
Y
* B
66
http://www.ntu.edu.sg/home/mmxie
World Frame
X
Solution
Solution

:points image of pairsfour select westage,tion initialisa At the
0,1,2,3} = i ),,({
0,1,2,3} = i ),,({
ririi
r
lilii
l
vuI
vu
I
=
=
*
0
* 1
* 2
*
3
*
0
* 1
* 2
* 3
*

0
*

3
*

0
Left image
Right image
Right Camera
Left Camera
2
P

Y
0

P

1
P

3

P

X
O
67
http://www.ntu.edu.sg/home/mmxie
0
Z
:have We
Δ•≅
×
Δ
PNI
with

34





























Δ
Δ
Δ

Δ
Δ
Δ

−−−
−−−
−−−
=
×
Z
Y
X
P
u
v
u
I
uuuuuu
vvvvvv
uuuuuu
N
r
l
l
r
r
r
r
r
r
llllll
llllll

3
2
3
1
3
0
323130
323130
34
















Δ
Δ
−−−
Z
v
vvvvvv
r
r
rrrrrr
r
r
r
r
r
r
323130
3
2
3
1
3
0
*
a
*
a
* 0
* 1
* 2
* 3
* 0
* 1
* 2
* 3
fi

a
* b

a
* b
* p
* p
Le
f
t
i
mage
Right image
Ri
g
ht Camera
Left Camera
Y
A

P
x
g
1
P•
2
P•
Y
O
68
http://www.ntu.edu.sg/home/mmxie
0

P•
3
P•
X
Z
O
B
x


Δ
••••=
Δ
−tt
k
I
N
N
N
G
k
P
)
(
]
)
[(
)
(
1






−−−−=Δ
Δ
Δ
t
rprbrprblplblplb
vvuuvvuukI
k
I
N
N
N
G
k
P
],,,[)(
)
(
]
)
[(
)
(






Δ+=

k
kPAP)(
*
a
*
a
* 0
* 1
* 2
* 3
* 0
* 1
* 2
* 3
fi

a
* b

a
* b
* p
* p
Le
f
t
i
mage
Right image
Ri
g
ht Camera
Left Camera
Y
A

P

g
1

P•
2
P•
Y
O
69
http://www.ntu.edu.sg/home/mmxie
0
P•
3
P•
X
Z
O
B
x
Example
Example
Left camera
Robot Arm
Gripper
Right camera
Position B
Position A
70
http://www.ntu.edu.sg/home/mmxie
PitiAiI
P
os
iti
on
A

i
n
I
mages
Left imageRight image
Position B in Images
71
http://www.ntu.edu.sg/home/mmxie
Q & A
72
http://www.ntu.edu.sg/home/mmxie
Q & A
Q & A
•What is hand-eye coordination?
•How to determine the motion for hand-eye coordination?
Left Camera Frame
Y
X
r
u
*
a
*
b
v
*
a
Left image
l
u
*b
*
Z
x
y
b
*
a
X
a
Right Camera Frame
Right image
r
v
l
v
Z
Y
x
y
*
a
b
*
* A
73
http://www.ntu.edu.sg/home/mmxie
World Frame
Z
X
Y
* B
Part 3:
Part 3:
Vision
-
guided Navigation
Vision
guided Navigation
74
http://www.ntu.edu.sg/home/mmxie
Learning Objectives
Learning Objectives
Nii

N
av
i
gat
i
on

PathFollowing
Path

Following
•Target Following
•Obstacle Avoidance
•Parking
•Q & A
75
http://www.ntu.edu.sg/home/mmxie
Navigation
76
http://www.ntu.edu.sg/home/mmxie
What is navigation?
What is navigation?
Y

Y
our answer:
a)
a)
77
http://www.ntu.edu.sg/home/mmxie
What is navigation?
What is navigation?
Niiiilhbd’lid

N
av
i
gat
i
on
i
s to cont
i
nuous
l
y c
h
ange a
b
o
d
y

s
l
ocat
i
on an
d

orientation in some specific ways.
78
http://www.ntu.edu.sg/home/mmxie
What are the possible ways of
navigating a body?
Y

Y
our answer:
a)
a)
b)
c)
d)
79
http://www.ntu.edu.sg/home/mmxie
What are the possible ways of
navigating a body?
Thfilfiibd

Th
ere are
f
our typ
i
ca
l
ways o
f
nav
i
gat
i
ng a
b
o
d
y:
a)tofollowpath
a)

to

follow

path
b) to follow target
c) to avoid obstacle
d) to park
80
http://www.ntu.edu.sg/home/mmxie
Path Following
81
http://www.ntu.edu.sg/home/mmxie
What is a path?
What is a path?
Y

Y
our answer:
a)
a)
82
http://www.ntu.edu.sg/home/mmxie
What is a path?
What is a path?
Ahfhilihihiiilli

A
pat
h
re
f
ers to t
h
e spat
i
a
l
po
i
nts, w
hi
c
h
connects an
i
n
i
t
i
a
l

l
ocat
i
on
to a final location.
83
http://www.ntu.edu.sg/home/mmxie
How to find a path?
How to find a path?
Y

Y
our answer:
a)
a)
b)
84
http://www.ntu.edu.sg/home/mmxie
How to find a path?
How to find a path?
Ahbdid

A
pat
h
can
b
e
d
eterm
i
ne
d
:
a)manuallybyahumanbeing
a)

manually

by

a

human

being
b) automatically from images
85
http://www.ntu.edu.sg/home/mmxie
How to determine a path in an
image?
Y

Y
our answer:
a)
a)
b)
c)
86
http://www.ntu.edu.sg/home/mmxie
How to determine a path in an
image?
Ilhldbdidbdi

I
n genera
l
, a pat
h
cou
ld

b
e
d
eterm
i
ne
d

b
y
d
etect
i
ng:
a)continuity
a)

continuity
b) discontinuity
c) appearance
in an image.
87
http://www.ntu.edu.sg/home/mmxie
How to determine a path in an
image?
El

E
xamp
l
e:
Edge map
Image
Contour Chains
88
http://www.ntu.edu.sg/home/mmxie
How to project a path from an
image to a space?
Y

Y
our answer:
Image Plane
2D space
89
http://www.ntu.edu.sg/home/mmxie
How to project a path from an
image to a space?
Ifhilfhliillb

If
a pat
h

i
s on a p
l
anar sur
f
ace, t
h
e so
l
ut
i
on w
ill

b
e:



kX
kY
k
M
u
v
















=
×











33
1


䥭慧a
Z
X
Y
䍡Ce牡
副慤⁓畲晡捥
90
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a path in space?
Y

Y
our answer:
a)
kX
k
Y
M
u
v










=







a)
b)

k
Y
k
M
v









=
×







33
1
Z
X
Y
Image
Camera
Road Surface
91
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a path in space?
Ilhli

I
n genera
l
, t
h
ere are two so
l
ut
i
ons:
a)deterministicmethod
a)

deterministic

method
b) iterative method

kX
kY
k
M
u
v














=
×











33
1
k








1
Image
Z
X
Y
Camera
RdSf
Image
92
http://www.ntu.edu.sg/home/mmxie
R
oa
d

S
ur
f
ace
How to determine the motion for a
body to follow a path in space?
El1DiiiMhd

E
xamp
l
e
1
:
D
eterm
i
n
i
st
i
c
M
et
h
o
d
.
t7
t8
t9
t10t12
t2
t3
t4
t5
t6
t7
t11
t13
t1
t2
t0
93
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a path in space?
El2IiMhd

E
xamp
l
e
2
:
I
terat
i
ve
M
et
h
o
d
.

Turn left
Motion
i
Reference
+
•Turn right
•Go straight
Execut
i
on
Path
+
-
Scene
Camera(s)
Observed path
Image
Z
X
Y
Camera
Road Surface
94
http://www.ntu.edu.sg/home/mmxie
Case 1: Turn left.
If the observed path rotates toward left, the motion command
will be “turn left”.

Turn left
•Turn right
•Go straight
Motion
Execution
Reference
Path
+
-
Scene
Camera(s)
Observed path
Image
Z
X
Y
Camera
RdSf
Image
95
http://www.ntu.edu.sg/home/mmxie
R
oa
d

S
ur
f
ace
Case 2: Turn right.
If the observed path rotates toward right, the motion command
will be “turn right”.

Turn left
•Turn right
•Go straight
Motion
Execution
Reference
Path
+
-
Scene
Camera(s)
Observed path
Image
Z
X
Y
Camera
RdSf
Image
96
http://www.ntu.edu.sg/home/mmxie
R
oa
d

S
ur
f
ace
Case 2: Go straight.
If the observed path is almost symmetric about the central line,
the motion command will be “go straight”.

Turn left
•Turn right
•Go straight
Motion
Execution
Reference
Path
+
-
Scene
Camera(s)
Observed path
Image
Z
X
Y
Camera
RdSf
Image
97
http://www.ntu.edu.sg/home/mmxie
R
oa
d

S
ur
f
ace
Target Following
98
http://www.ntu.edu.sg/home/mmxie
What is a target?
What is a target?
Y

Y
our answer:
a)
a)
99
http://www.ntu.edu.sg/home/mmxie
What is a target?
What is a target?
Afbjfibjfi

A
target re
f
ers to an o
bj
ect o
f

i
nterest, or an o
bj
ect o
f
attent
i
on.
100
http://www.ntu.edu.sg/home/mmxie
How to detect a target in image?
How to detect a target in image?
Y

Y
our answer:
a)
a)
b)
c)
101
http://www.ntu.edu.sg/home/mmxie
How to detect a target in image?
How to detect a target in image?
Ilbddbii

I
n genera
l
, a target can
b
e
d
etecte
d

b
y exam
i
n
i
ng:
a)continuity
a)

continuity

b) discontinuity
c) appearance
in image.
102
http://www.ntu.edu.sg/home/mmxie
How to detect a target in image?
How to detect a target in image?
El

E
xamp
l
e:
103
http://www.ntu.edu.sg/home/mmxie
(Video
)
How to determine the motion for a
body to follow a target in space?
Y

Y
our answer:
a)
a)

b)
?
104
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a target in space?
Ilh

I
n genera
l
, t
h
ere are two ways:
a)deterministicmethod
a)

deterministic

method
b) iterative method
?
105
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a target in space?
body to follow a target in space?
•Example 1: Deterministic Method.
Reference Image
Image at View 2
Vi2
Image at View 1
Vi
ew
2
Reference View
(R, T)
Known motion
View 1
106
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to follow a target in space?
El2IiMhd

E
xamp
l
e
2
:
I
terat
i
ve
M
et
h
o
d
.
Rotation angles and
translation vector(s)
•Translation
•Rotation
Motion
Execution
Reference
Image
+
-
Scene
Camera(s)
Observed Images
Scene
107
http://www.ntu.edu.sg/home/mmxie
Case 1: Translation.
Z
X
Y
by
axis
X
along
body
the
translate
to
necessary
is
it
center
image thefrom deviatescenter target of coordinate X theIf
C1
C2
)(
: toalproportionamount an
by

axis

X

along
body

the

translate
to
necessary

is
it

center
,
12ccxx
xxgt−=
Z
X
Y
image thefrom deviatescenter target of coordinate Y theIf
Y
C1
C2
)
(
: toalproportionamount an
by axis Y alongbody the translatetonecessary isit center,
1
2
y
y
g
t
−=
)
(

1
2
ccyy
y
y
g
t
108
http://www.ntu.edu.sg/home/mmxie
Case 1 (continued): Translation.
Z
X
Y
C1
t
l
ti
amount an by axis Zalongbody the translatetonecessary
isit value,reference a fromdifferent is target of area theIf
)(
:
t
oa
l
propor
ti
o
n
AreaAreagt
r
zz
−=
Z
X
Y
C2
109
http://www.ntu.edu.sg/home/mmxie
Case 2: Rotation.
shortens
edge
ical
rightvert
while
elongates
edge
cal
leftverti
If
lt
d
il
ihtt
hil
ht
d
l
lftti
If
axis. Yabout rotate tonecessary isit
shortens
,
edge

ical
right

vert

while
elongates

edge

cal
left

verti

If
.axis Yabout rotate tonecessary also isit
e
l
onga
t
es, e
d
ge
i
ca
l
r
i
g
ht
ver
t
w
hil
es
h
or
t
ens e
d
ge ca
l
l
e
ft
ver
ti

If
)(
: toalproportion is anglerotation The
21
y
vvk
y
Δ−Δ=
θ
Z
y
y
Z
X
X
Y
C2
1
v
2
v
X
Y
110
http://www.ntu.edu.sg/home/mmxie
Case 2 (continued): Rotation.
edge
horizontal
lower
while
elongates
edge
horizontal
upper
If
d
hitl
l
hil
ht
d
hitl
If
axis. Xabout rotate tonecessary isit shortens,

edge
horizontal
lower

while
elongates

edge

horizontal
upper

If
.axis Xabout rotate tonecessary also isit elongates,
e
d
ge
h
or
i
zon
t
a
l
l
ower w
hil
es
h
or
t
ens e
d
ge
h
or
i
zon
t
a
l
upper
If
)(
:toalproportion is anglerotation The
21x
hhk
x
Δ−Δ=
θ
Z
Z
X
h
X
Y
C2
1
h
X
Y
2
h
111
http://www.ntu.edu.sg/home/mmxie
Case 2 (continued): Rotation.
Z
X
Y
r
h
necessar
y
isit direction
,
same ain rotate ed
g
es verticalIf
Y
C1
h
1
r
h
2
r
v
1
r
v2
l
i
i
dii
i
d
hil
If
axis. about Z rotate to
y
,
g
h
2
axis. about Z rotate tonecessary
a
l
so
i
s
i
t
di
rect
i
on, same a
i
n rotate e
dg
es
h
or
i
zonta
l

If
Z
X
Y
1
h
: toalproportion is anglerotation The
C2
1
2
h
1
v
2
v
)]]
(
)
(
[
)]()([
2211z
v
v
z
v
v
z
k
or
hhzhhz
k
rr
r
r
z
×
+
×
=
×+×=
θ
θ
)]]
(
)
(
[

2211z
v
v
z
v
v
z
k
z
×
+
×
=
θ
112
http://www.ntu.edu.sg/home/mmxie
SystemSetup
Reference Image
takenatthe
System

Setup
taken

at

the

Reference View
Image taken at an
initialview
Image taken at the
final view after
movingthecamera
initial

view
moving

the

camera

with the series of
computed motion
transformations.
113
http://www.ntu.edu.sg/home/mmxie
Obstacle Avoidance
114
http://www.ntu.edu.sg/home/mmxie
What is an obstacle?
What is an obstacle?
Y

Y
our answer:
a)
a)
115
http://www.ntu.edu.sg/home/mmxie
What is an obstacle?
What is an obstacle?
Bdfiiibliihihbh

B
y
d
e
fi
n
i
t
i
on, an o
b
stac
l
e
i
s an ent
i
ty, w
hi
c
h
o
b
structs a pat
h
.
116
http://www.ntu.edu.sg/home/mmxie
How to detect an obstacle?
How to detect an obstacle?
Y

Y
our answer:
a)
a)
b)
c)
117
http://www.ntu.edu.sg/home/mmxie
How to detect an obstacle?
How to detect an obstacle?
Thhibl

Th
ere are t
h
ree poss
ibl
e ways:
a)Solutionbasedon2Dvision
a)

Solution

based

on

2D

vision
b) Solution based on 3D vision
c) Solution based on active sensing
118
http://www.ntu.edu.sg/home/mmxie
How to detect an obstacle?
How to detect an obstacle?
El

E
xamp
l
e:
a) We employ two cameras.
b)Iftheheightsandorientationsofthetwocameraswith
b)

If

the

heights

and

orientations

of

the

two

cameras

with

respect to the road are unchanged, the projection matrices
from images to road surface will remain unchanged.
c) What is the condition for a pair of image points to be
projected back to the road surface?
Z
X
Left Camera
* P1
* P2
*
Z
X
Y
RihtC
X
Y
*
Road Surface
119
http://www.ntu.edu.sg/home/mmxie
Ri
g
ht

C
amera
How to detect an obstacle?
How to detect an obstacle?
Sli

S
o
l
ut
i
on:
















•=•

:iprelationsh following thehave wecamera,left In the :1Step
3
3
1
1
Y
X
M
v
k
uk
l
l














×
1

3
3
1
1
Y
M
k
v
k
l
:
ip
relationsh
following
the
have
we
camera,
right
Inthe
:
2
Step
























•=•

×
1

:
ip
relationsh

following

the
have

we
camera,
right

In

the
:
2
Step
33
2
2
2
Y
X
N
k
vk
uk
r
r
Left Camera
*
Z
Z
X
Y
* P1
* P2
*
*
120
http://www.ntu.edu.sg/home/mmxie
X
Y
Right Camera
Road Surface
How to detect an obstacle?
How to detect an obstacle?
Sli(id)

S
o
l
ut
i
on
(
cont
i
nue
d)
:
:
y
ieldse
q
uations twoabove theofn combinatio The:3 Ste
p




















••=•


××
1

y
q
p
1
3333
r
r
l
l
v
u
NM
k
vk
uk








1
k
Z
X
Y
Left Camera
* P1
* P2
*
Z
X
Y
Right Camera
*
Road Surface
121
http://www.ntu.edu.sg/home/mmxie
How to detect an obstacle?
•Experimental Setup:
Left Camera
Right Camera
u
0
u
0
Left Image
Right Image
Y
Z
u
0
Z
u
0
Y
Wlddit
X
Y
X
Z
122
http://www.ntu.edu.sg/home/mmxie
W
or
ld
coor
di
na
t
es
v
World coordinates
v
How to detect an obstacle?
•Results:
Left Image
Right Image
Right Image Projected to Left
Edge map
Superposition
123
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to avoid an obstacle?
Y

Y
our answer:
124
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to avoid an obstacle?
Ilh

I
n genera
l
, t
h
ere are two ways:
a)deterministicmethod
a)

deterministic

method
b) iterative method
125
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to avoid an obstacle?
El1DiiiMhd

E
xamp
l
e
1
:
D
eterm
i
n
i
st
i
c
M
et
h
o
d
.
126
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a
body to avoid an obstacle?
El2IiMhd

E
xamp
l
e
2
:
I
terat
i
ve
M
et
h
o
d
.

Turnleft
Reference

Turn

left
•Turn right
•Go straight
Motion
Execution
Reference

Path
+
-
Scene
Camera(s)
Observed path
Image
Z
X
Y
Camera
RoadSurface
127
http://www.ntu.edu.sg/home/mmxie
Road

Surface
Parking
128
http://www.ntu.edu.sg/home/mmxie
What is parking?
What is parking?
Y

Y
our answer:
a)
a)
129
http://www.ntu.edu.sg/home/mmxie
What is parking?
What is parking?
Pkiibdfli/ii

P
ar
ki
ng
i
s to move a
b
o
d
y
f
rom a temporary
l
ocat
i
on
/
or
i
entat
i
on to
an intended (or authorized) location/orientation.
130
http://www.ntu.edu.sg/home/mmxie
How to determine an intended
location/orientation from image?
Y

Y
our answer:
a)
a)
b)
c)
131
http://www.ntu.edu.sg/home/mmxie
How to determine an intended
location/orientation from image?
Ildtt

I
n genera
l
, one can
d
e
t
ec
t
an
intended location/orientation by
examining:
a) discontinuity
b) continuity
c)appearance
c)

appearance
in an image.
132
http://www.ntu.edu.sg/home/mmxie
How to determine an intended
location/orientation from image?
El

E
xamp
l
e:
133
http://www.ntu.edu.sg/home/mmxie
How to project an intended
location/orientation from image to
location/orientation from image to
space?
Y

Y
our answer:
Image Plane
2D space
134
http://www.ntu.edu.sg/home/mmxie
How to project an intended
location/orientation from image to
location/orientation from image to
space?
Ifhilfhliillb

If
t
h
e space
i
s a p
l
anar sur
f
ace, t
h
e so
l
ut
i
on w
ill

b
e:



kX
kY
k
M
u
v
















=
×











33
1


䥭慧a
Z
X
Y
䍡Ce牡
副慤⁓畲晡捥
135
http://www.ntu.edu.sg/home/mmxie
How to project an intended
location/orientation from image to
location/orientation from image to
space?
El

E
xamp
l
e:
136
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a body
to park into an intended
to park into an intended
location/orientation in space?
Y

Y
our answer:
137
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a body
to park into an intended
to park into an intended
location/orientation in space?
Ilhli

I
n genera
l
, t
h
ere are two so
l
ut
i
ons:
a)deterministicmethod
a)

deterministic

method
b) iterative method
138
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a body
to park into an intended
to park into an intended
location/orientation in space?
El1DiiiMhd

E
xamp
l
e
1
:
D
eterm
i
n
i
st
i
c
M
et
h
o
d
.
Video
139
http://www.ntu.edu.sg/home/mmxie
How to determine the motion for a body
to park into an intended
to park into an intended
location/orientation in space?

Example2:IterativeMethod

Example

2:

Iterative

Method
.
Video1
Video2
Video3
Cameras
Landing Pad
Reference
Ground
140
http://www.ntu.edu.sg/home/mmxie
Q & A
141
http://www.ntu.edu.sg/home/mmxie
Q & A
Q & A
Whihd
dii?

Wh
at
i
s
h
ea
d
-eye coor
di
nat
i
on
?

Howtodeterminethemotionforhead
-
eyecoordination?
How

to

determine

the

motion

for

head
eye

coordination?
142
http://www.ntu.edu.sg/home/mmxie
End
143
http://www.ntu.edu.sg/home/mmxie