Projective Geometry for Computer Vision

Arya MirΛογισμικό & κατασκευή λογ/κού

17 Ιουλ 2011 (πριν από 6 χρόνια και 4 μήνες)

869 εμφανίσεις

3D Computer Vision Classical Problem: Given a collection of 2D images, build a model of the 3D world.

Projective Geometry
for Computer Vision
Raquel A. Romano
MIT Artificial Intelligence Laboratory
romano@ai.mit.edu
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
1
3D Computer Vision
Classical Problem:
Given a collection of 2D images,
build a model of the 3D world.
Example Applications:
•virtual/immersive environments
•robotics & autonomous vehicles
•minimally invasive surgery
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
2
Outline
1.
Projective Geometry Overview
2.
Minimal Projective Parameters
3.
Projective Parameter Estimation
4.
Motion Boundary Detection
5.
Conclusion
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
3
Image Formation
3D scene
imaging
2D images
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
4
Computer Vision
3D scene
model
data
2D images
analysis
measurement
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
5
scene point
optical center
image point
optical ray
opt
i
cal axis
Camera Geometry:
Single View
pinhole model of
perspective projection
Z
Y
y
Z
X
x
=
=
unknown depth at
each point






+



















y
x
y
x
c
c
y
x
f
s
f
y
x
1
unknown internal
camera parameters
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
6
Camera Geometry:
Multiple Views
unknown rotations and translations
jk
jk
T
R
ij
ij
T
R
ik
ik
T
R
j
x
k
x
i
x
X
T
R
+





















Z
Y
X
Z
Y
X
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
7
Measured Data:
Image Points and Lines
geometric constraint: optical rays intersect in 3D
projective geometry: express constraint in terms of
measured 2D image features
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
8
Projective Camera Model

linear model of image formation

depth-independent expression for optical
ray intersections

multilinear relations among point and line
matches
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
9
Bilinear Constraints
(Longuet-Higgins ,1981, Faugeras, 1992; Hartley, 1992)
i
i
x
X
λ
=
[
]
X
T
R
A
x

j
x
ij
T
i
x
ij
R
X
i
A
j
A
ij
i
j
j
ij
T
x
R
x
+
=
ι
λ
λ
[
]
0
=
i
ij
T
j
ij
x
R
T
x
x
j
j
j
i
i
i
x
A
x
x
A
x
1
1




0
=
i
ij
T
j
x
F
x
[
]
0
1
=


i
ij
i
ij
T
j
T
j
ij
x
F
A
R
T
A
x
4
43
4
42
1
x
fundamental matrix
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
10
Fundamental Matrix
Maps a point in one image to a line in the
other image that contains its match
j
x
k
x
i
x
ij
F
kj
F
Given matching points in two views, predict
the matching point in a third image.
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
11
Projective Models in Practice

View synthesis and interpolation: point transfer
function for dense point correspondences

Self-calibration: automatic recovery of internal
camera parameters from fundamental matrices

Bundle adjustment initialization: initial rotation and
translation for nonlinear Euclidean optimization
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
12
Outline
1.
Projective Geometry Overview
2.
Minimal Projective Parameters
3.
Projective Parameter Estimation
4.
Motion Boundary Detection
5.
Conclusion
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
13
Practical Problem

Few point matches between some
views.

Unstable for estimating geometric
relationships.
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
14
Geometric Consistency
Pairwise geometric relations may be
inconsistent.
?
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
15
Goals

Impose algebraic
geometric constraints on
stationary points seen in
arbitrarily many views.

Avoid estimating too many
parameters: depths,
rotations, translations
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
16
Geometric Dependencies
ij
F
ij
F
ij
F
ij
F
ij
F
ij
F
ij
F
ij
F

Pairwise
projective geometric relations are
interdependent.

Approach: define projective dependencies and
restrict solutions to be globally consistent
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
17
Projective Bilinear Parameters
i
x
j
x
0
=
i
ij
T
j
x
F
x
X
[
]
1


=
i
ij
ij
T
j
ij
A
R
T
A
F
x
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
18
Projective Bilinear Parameters
ij
h
ij
e
ji
e
i
x
j
x
0
=
i
ij
T
j
x
F
x
epipoles
ji
ij
e
e
ij
h
epipolar collineation
[]
[
]
[
]
x
x
ij
T
i
T
i
ij
j
j
ji
ij
e
p
q
h
q
p
e
F








(Csurka, et.al., 1997)
imaged 3D
translation & rotation
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
19
Projective Parameters
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
ij
ji
ij
h
e
e
,
,
•p
r
o
v
i
d
e
a
c
o
m
p
l
e
t
e

projective model of
camera configuration
But...

set of all pairwise
parameters are still
redundant

not all images have
sufficient overlap
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
20
Trifocal Dependencies

derive dependencies among
three fundamental matrices

correctly models degrees
of freedom in camera
configuration

geometrically consistent
parameterized model of view
triplets
ki
e
kj
e
ik
e
ij
e
ji
e
jk
e
ki
h
kj
h
ij
h
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
21
Trifocal Dependencies
trifocal lines available from two fundamental matrices

derive dependencies among
three fundamental matrices

correctly models degrees
of freedom in camera
configuration

geometrically consistent
parameterized model of view
triplets
k
t
ki
e
kj
e
i
t
j
t
ik
e
ij
e
ji
e
jk
e
ki
h
kj
h
ij
h
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
22
Outline
1.
Projective Geometry Overview
2.
Minimal Projective Parameters
3.
Projective Parameter Estimation
4.
Motion Boundary Detection
5.
Conclusion
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
23
Recovering Camera Geometry
view i
view j
view k
few
correspondences
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
24
Linear Initialization
8-point Algorithm
(Hartley, 1995)
(
)
()
{}

j
i
i
ij
T
j
x
x
x
F
x
,
Minimize
over all matching point pairs.
[]
T
ij
f
f
f
f
f
f
f
f
f
33
32
31
23
22
21
13
12
11
=
f
Rewrite bilinear constraints as
where
and solve linear system
0
Af
=
ij
[]
0
f
=
ij
j
j
j
j
i
j
i
j
j
i
j
i
y
x
y
y
y
y
x
x
x
y
x
x
1
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
25
Projection to Parameter Space
Map linear estimate of fundamental matrix
to projective parameter space:
}
,
,
{
7
ij
ji
ij
ij
h
e
e
p
=

ij
F
}
,
,
{
4
ij
j
i
ij
h
γ
γ
p
=


parameterization requires choice of projective basis

basis affects shape of error surface for nonlinear
optimization
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
26
Geometric Objective Function
point-to-epipolar-line distance ~ image reprojection
error
weighted residual of bilinear constraint
i
T
j
ij
ij
j
i
ij
w
E
x
F
x
p
x
x
p
7
)
;
,
(
7
=
2
2
2
1
2
2
2
1
)
(
)
(
1
)
(
)
(
1
j
T
j
T
i
ij
i
ij
ij
ij
ij
w
x
F
x
F
x
F
x
F
+
+
+
=
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
27
Error Surface Depends on Basis
canonical basis
geometrically defined basis
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
28
gamma(i,j) (h1,h2)
(h1,h2) (h2,h3) (h1,h3)
(e1,e2)
(e3,e4)
gamma(i,j) (h
1,h2)
(e1,e2) (e3,e4)
(h1,h2) (h2,h3) (h1,h
3)
Nonlinear Trifocal Estimation
i
j
k
1.
Initialize epipolar geometry

8-point algorithm: linear solution to fundamental
matrix for all view pairs

extract epipoles
and epipolar collineations
2.
7D nonlinear minimization: bifocal parameters for view
pairs (i,k) (j,k)
3.
Trifocally
constrained estimation for view pair (i,j)

compute trifocal lines

project parameters to trifocally
constrained space

4D nonlinear minimization for bifocal parameters
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
29
Convergence
eij
error
eji
-4000 -2000 0
Ground Truth
8-point Algorithm
7-Parameter Search
Trifocal Projection
4-Parameter Search
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
30
Ground Truth
8-Point Algorithm
7-Parameter Algorithm
4-Parameter Algorithm
Results
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
31
knossos
sequence
view i
view k
view j
few
correspondences
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
32
Ground Truth
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
33
4-Parameter Algorithm
7-Parameter Algorithm
8-Point Algorithm
Results
Summary

Imposing projective constraints on
camera geometry corrects the
estimation of epipolar geometry

Resulting camera configuration for

multiple cameras is globally consistent
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
34
Outline
1.
Projective Geometry Overview
2.
Minimal Projective Parameters
3.
Projective Parameter Estimation
4.
Motion Boundary Detection
5.
Conclusion
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
35
Camera and Scene Motion
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
36
Combining Intensity and Geometry
trifocal tensor
projective linear form relating a point-line-line
(Spetsakis
& Aloimonos, 1990; Shashua, 1994)
0
)
,
,
(
=
k
j
i
Τ
l
l
x
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
37
Tensor Brightness Constraint
(Shashua & Hannah, 1995; Shashua & Stein, 1997)

Horn-Schunk brightness
constraint is linear in
point coordinates

Defines line in each
image containing
matching point

Spatiotemporal gradient
at every pixel provides
test of rigid motion
u Ix
+ v Iy
+ It
= 0
ax + by + c = 0
(a,b,c)T

Ix






Iy
It -x
0 Ix –y
0 Iy
u = x -
x0
v = y -
y0
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
38
Motion Boundary Detection
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
39

Partition image into windows and
solve for trifocal tensor coefficients.

Sum residual error
of tensor solution.
•O
n
l
y
r
e
g
i
o
n
s
w
i
t
h

rigid 3D motion
have a good fit.

High residuals indicate regions
that cross a motion boundary.
Multiple Frame Flow

Multi-frame tracks fall into
separable classes

Track points over many frames

Robustly fit tracks to linear
approximation of
instantaneous planar motion
x(t) = x0+ t
[Ax0 + b]
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
40
Detecting Independent Motions
Residual error of estimated
motion model on all point tracks
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
41
Complexity of Motion Model
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
42
Conclusions
When possible, use domain and task
knowledge to choose model:

What type of information is needed

What aspects of the imaging conditions
are known or controlled

What types of uncertainty can be
modeled and compensated for
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
43
Future Needs
Role of learning in motion analysis:

S
upervised learning of geometric motion
classes

Data-driven model selection by flow
classification

Robust estimation of appropriate motion
model

Adaptive, time-varying estimation
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
44
END
Scientific Computing Seminar
May 12 , 2004
Projective Geometry
for Computer Vision
Raquel A. Romano
45