CSCE 641 Computer Graphics:
Image
-
based Modeling
Jinxiang Chai
Image
-
based modeling
Estimating 3D structure
Estimating motion, e.g., camera motion
Estimating lighting
Estimating surface model
Traditional modeling and rendering
User input
Texture map
survey data
Geometry
Reflectance
Light source
Camera model
Images
modeling
rendering
For photorealism:
-
Modeling is hard
-
Rendering is slow
Can we model and render this?
What do we want to do for this model?
Image based modeling and rendering
Images
user input range
scans
Model
Images
Image
-
based
modeling
Image
-
based
rendering
Spectrum of IBMR
Images
user input range
scans
Model
Images
Image
based
modeling
Image
-
based
rendering
Geometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
Spectrum of IBMR
Images
user input range
scans
Model
Images
Image
based
modeling
Image
-
based
rendering
Geometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
Spectrum of IBMR
Images
user input range
scans
Model
Images
Image
based
modeling
Image
-
based
rendering
Geometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
Stereo reconstruction
Given two or more images of the same scene or object,
compute a representation of its shape
What are some possible applications?
known
camera
viewpoints
3D modeling
From one stereo pair to a 3D head model
[
Frederic Deverney
, INRIA]
3D modeling
The Digital Michelangelo Project, Levoy et al.
Optical mocap
Vicon mocap system
Z
-
keying: mix live and synthetic
Takeo Kanade, CMU (
Stereo Machine
)
Virtualized Reality
TM
[Takeo Kanade
et al.
, CMU]
•
collect video from 50+ stream
•
reconstruct 3D model sequences
•
•
steerable version used for
SuperBowl XXV “
eye vision
”
http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html
View interpolation
input
depth image
novel view
[Szeliski & Kang ‘95]
View morphing
Morph between pair of images using epipolar
geometry
[Seitz & Dyer, SIGGRAPH’96]
Image warping
Video view interpolation
Performance Interface
Microsoft
Natal project
Additional applications?
•
Real
-
time people tracking (systems from Pt. Gray
Research and SRI)
•
“Gaze” correction for video conferencing
[Ott,Lewis,Cox InterChi’93]
•
Other ideas?
Stereo matching
Given two or more images of the same scene or
object, compute a representation of its shape
What are some possible representations for shapes?
•
depth maps
•
volumetric models
•
3D surface models
•
planar (or offset) layers
Outline
Stereo matching
-
Traditional stereo
-
Multi
-
baseline stereo
-
Active stereo
Volumetric stereo
-
Visual hull
-
Voxel coloring
-
Space carving
Stereo matching
•
Masatoshi Okutomi and Takeo Kanade. A multiple
-
baseline stereo. IEEE Trans.
on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353
--
363.
•
D. Scharstein and R. Szeliski.
A taxonomy and evaluation of dense two
-
frame
stereo correspondence algorithms
.
International Journal of Computer Vision
, 47(1/2/3):7
-
42, April
-
June 2002.
Visual
-
hull reconstruction
•
Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision,
Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23
-
32.
•
Matusik, Buehler, Raskar, McMillan, and Gortler , “Image
-
Based Visual Hulls”,
Proc. SIGGRAPH 2000, pp. 369
-
374.
Photo
-
hull reconstruction
•
Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl.
Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151
-
173.
•
Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of
Computer Vision, 2000, 38(3), pp. 199
-
218.
Papers
Stereo
scene point
optical center
image plane
Stereo
Basic Principle: Triangulation
•
Gives reconstruction as intersection of two rays
•
Requires
>
calibration
>
point correspondence
Camera calibration
From world coordinate to image coordinate
u
0
v
0
1
0
0
-
s
y
0
s
x
a
u
v
1
Perspective
projection
View
transformation
Viewport
projection
Camera parameters
3D points
2D projections
Stereo correspondence
Determine Pixel Correspondence
•
Pairs of points that correspond to same scene point
Epipolar Constraint
•
Reduces correspondence problem to 1D search along
conjugate
epipolar lines
•
Java demo:
http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
epipolar line
epipolar line
epipolar plane
Stereo image rectification
Stereo image rectification
•
reproject image planes onto a common
plane parallel to the line between optical centers
•
pixel motion is horizontal after this transformation
•
two homographies (3x3 transform), one for each
input image reprojection
C. Loop and Z. Zhang.
Computing Rectifying Homographies for
Stereo Vision
. IEEE Conf. Computer Vision and Pattern
Recognition, 1999
.
Rectification
Original image pairs
Rectified image pairs
Stereo matching algorithms
Match Pixels in Conjugate Epipolar Lines
•
Assume brightness constancy
•
This is a tough problem
•
Numerous approaches
>
A good survey and evaluation:
http://www.middlebury.edu/stereo/
Your basic stereo algorithm
For each epipolar line
For each pixel in the left image
•
compare with every pixel on same epipolar line in right image
•
pick pixel with minimum matching cost
Improvement: match
windows
•
This should look familiar..
•
Can use Lukas
-
Kanade or discrete search (latter more common)
Window size
•
Smaller window
+
-
•
Larger window
+
-
W = 3
W = 20
Effect of window size
Stereo results
Ground truth
Scene
•
Data from University of Tsukuba
•
Similar results on other images without ground truth
Results with window search
Window
-
based matching
(best window size)
Ground truth
Better methods exist...
State of the art method
Boykov et al.,
Fast Approximate Energy Minimization via Graph Cuts
,
International Conference on Computer Vision, September 1999.
Ground truth
Stereo reconstruction pipeline
Steps
•
Calibrate cameras
•
Rectify images
•
Compute disparity
•
Estimate depth
•
Camera calibration errors
•
Poor image resolution
•
Occlusions
•
Violations of brightness constancy (specular reflections)
•
Large motions
•
Low
-
contrast image regions
Stereo reconstruction pipeline
Steps
•
Calibrate cameras
•
Rectify images
•
Compute disparity
•
Estimate depth
What will cause errors?
Outline
Stereo matching
-
Traditional stereo
-
Multi
-
baseline stereo
-
Active stereo
Volumetric stereo
-
Visual hull
-
Voxel coloring
-
Space carving
Depth from disparity
f
x
x’
baseline
z
C
C’
X
f
input image (1 of 2)
[Szeliski & Kang ‘95]
disparity map
3D rendering
width of
a pixel
Choosing the stereo baseline
What’s the optimal baseline?
•
Too small: large depth error
•
Too large: difficult search problem
Large Baseline
Small Baseline
all of these
points project
to the same
pair of pixels
The effect of baseline on depth estimation
1/z
width of
a pixel
width of
a pixel
1/z
pixel matching score
Multi
-
baseline stereo
Basic Approach
•
Choose a reference view
•
Use your favorite stereo algorithm BUT
>
replace two
-
view SSD with SSD over all baselines
Limitations
•
Must choose a reference view (bad)
•
Visibility!
CMU’s 3D Room Video
Outline
Stereo matching
-
Traditional stereo
-
Multi
-
baseline stereo
-
Active stereo
Volumetric stereo
-
Visual hull
-
Voxel coloring
-
Space carving
Active stereo with structured light
Project “structured” light patterns onto the object
•
simplifies the correspondence problem
camera 2
camera 1
projector
camera 1
projector
Li Zhang’s one
-
shot stereo
Active stereo with structured light
Laser scanning
Optical triangulation
•
Project a single stripe of laser light
•
Scan it across the surface of the object
•
This is a very precise version of structured light scanning
Digital Michelangelo Project
http://graphics.stanford.edu/projects/mich/
Laser scanned models
The Digital Michelangelo Project
, Levoy et al.
Laser scanned models
The Digital Michelangelo Project
, Levoy et al.
Desktop scanner
Convenient to use
Good quality
Relatively low
-
cost
-
next engine
(about 2k)
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment