Multiview Rendering - Juraj Obert

birdsowlSoftware and s/w Development

Dec 2, 2013 (3 years and 10 months ago)

80 views

MULTIVIEW

RENDERING

Juraj

Obert

FEL
ČVUT


supervised by
Ing
. Jan
Buri
á
nek

Outline


Motivation


Related Work


Rendering Techniques


Requirements


PerspectiveCube


Results


Future Work


Discussion

2

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Motivation

Perspective (pinhole) camera is defined by a set of rays, which are

(1) incident with the center of projection and

(2) the optical axis is perpendicular to the image plane.


Relaxation of the latter requirement gives rise to
central

cameras, which are defined as sets of rays, which meet in one point.

3

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Object

Inverted image

Pinhole

Optical axis

Motivation

4

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Perspective (pinhole) camera is defined by a set of rays, which are

(1) incident with the center of projection and

(2) the optical axis is perpendicular to the image plane.


Relaxation of both requirements gives rise to
non
-
central

cameras, which are defined as sets of arbitrary rays in 3D space.

In traditional rendering systems, perspective cameras are rather well
-
explored. Our goal was to create a system for working

with non
-
perspective (also termed
multiperspective
) projections and images.

Related Work


Multiperspective

Panoramas

5


Sweeping a window across the image simulates movement
in 3D space


Used in Walt Disney
’s

Pinocchio in 1940 and researched

in 1997 by Wood et al.


Fostered a lot of new research

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Related Work


X
-
Slits

6


X
-
Slits consist of rays incident with two non
-
intersecting lines in 3D space


Yield

images, which are fundamentally different from perspective images,
but look alike (hardly distinguishable for inexperienced users)


Can simulate perspective projections by using spatiotemporal volumes

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Related Work


Digital Cubism

7

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Rendering Techniques

Spatiotemporal volumes


Stack video frames to create

a spatiotemporal volume


Define a projection surface


Shade the projection surface
by slicing the spatiotemporal
volume



Amenable to programmable
graphics hardware


Simple support for animation


Requires input video

Ray
-
based composition


Each pixel corresponds to

a different camera ray


Rays are arbitrary, no explicit
patterns such as fans of rays
as in the case of perspective
projections



Implementation on graphics
hardware an open research
problem


Aliasing problems and user
interface problems

8



Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Spatiotemporal Volume


Extract frames from input video file, stack them and build

a volume texture (x, y, t)


Three important slicing planes:


x
-
y


original frames


y
-
t


pushbroom

image


x
-
t


epipolar

plane image



Can be used to generate various classes of
multiperspective

images, such as X
-
Slits, cyclographs, etc.


Our goal was to develop a tool, which would allow a user to
do this in real time

9

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

x

y

t

Requirements


Production requirements


Capacity to handle huge amounts of data


Utilization of graphics hardware


Presets for commonly used camera classes


Interactivity


Extensibility




Naive approach


Decompress video file, create a volume texture containing every single frame and
store it in video memory


Example:


19 seconds, 640 x 480, RGB, 24 FPS


400 MB


2 hours, 1280 x 720, RGB, 24 FPS


450 GB !!!



Video memory capacity of latest ( March 2007) NVIDIA graphics cards


768 MB


Video memory capacity of upcoming (June 2007) ATI graphics cards


1024 MB



=> naive approach will not work for high
-
definition content !!!





10

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

PerspectiveCube

-

Overview


Octree
-
based spatial organization

of spatiotemporal volumes



GPU memory management based

on adaptive assembly of shuffle
textures



On
-
demand loading



Two
-
level caching mechanism and
multithreaded rendering engine

for interactivity



Predefined camera types and
animation curves



Robust, scalable and GPU
-
based



11

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

PerspectiveCube



Rendering

12

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

PerspectiveCube

-

Caching

13

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Perspective Cube


Demo

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

14

Perspective Cube


Demo

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

15

Perspective Cube


Demo

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

16

Results
-

Redmond

17

Pushbroom

X
-
Slits

Redmond X
-
Slits right

Original

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

x

t

Resolution: 360 x 240

Length: 148 seconds

FPS: 15

24
bpp

--------------------------------

STV size 548 MB

Shuffle size ~ 128 MB

Results


Langweil’s

Prague

18

Original

P
ushbroom

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Resolution: 720 x 480

Length: 30 seconds

FPS: 15

24
bpp

STV size 445 MB

Shuffle size ~ 64 MB

Future Work


Explicit camera ray representation


Depart from spatiotemporal volumes and use ray
-
based techniques


More freedom vs. higher productivity


Non
-
planar projection surfaces


Even though already supported in code, practical applications still require
thorough research


Problems with unwrapping


Integration with existing modeling systems


Access to ray tracing engine, which is necessary for ray
-
based rendering


Unwrapping for non
-
planar projection surfaces


User interfaces


Controls for perspective camera manipulation are too restrictive

for
multiperspective

cameras


How would one map one ray per pixel ?


Performance


Effect of used video compression schemes, frame extraction scheduling
algorithm, video memory management


Bottleneck is shuffle texture assembly and frame decompression

19

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Conclusion


Ray
-
based techniques amenable to ray tracing engines, but not to
programmable graphics devices



PerspectiveCube



system for GPU slicing of spatiotemporal volumes


Octree
-
based spatial organization and smart texture assembly algorithm

for video memory waste minimization


Interactivity due to two
-
level caching mechanism and multithreaded rendering


Scalability in terms of video memory size


Support for animation


Presets for commonly used
multiperspective

camera types



In the future


Push for ray
-
based techniques as ray tracing hardware gets developed


Don’t necessarily require that programmable graphics devices be used,

CPU can be faster sometimes





20

Multiview Rendering
-

Juraj Obert
-

FEL CVUT

Thank you


Questions ?

21

Multiview Rendering
-

Juraj Obert
-

FEL CVUT