Special Effects in the Real World

slipperhangingAI and Robotics

Nov 14, 2013 (3 years and 9 months ago)

68 views

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Ramesh Raskar

Mitsubishi Electric Research Labs (MERL)

Cambridge, MA, USA


Special Effects in the Real World

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Parent Organization:
MELCO


Mitsubishi Electric (MELCO), including
subsidiaries


$32B in annual sales


125,000 people


10 business sectors:


Space Development and Satellite Communications


Communications and Information Processing


Visual Equipment and Systems


Electronic Devices


Home Electronics


Energy


Industrial Equipment


Public
-
Use Systems


Transportation

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

MERL


Mitsubishi Electric Research Laboratories


Dick Waters, President & CEO


Two labs, co
-
located in Cambridge, MA:


MERL Technology Lab


Advanced systems development


~35 researchers


MERL Research Lab


Joe Marks, Director


Basic research in applied computing


23 researchers, interns, consultants


CV, HCI, Multimedia, CG, AI

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Special Effects



Photorealism and Animation


Stays on screens


Yet to affect daily life in real time


Post
-
processed content


Unlike internet,databases, UI


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Changing Appearance

Virtual light
source

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Changing Virtual Illumination

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Special Effects in the Real World


Photorealism and Animation



Limited to display screens


Yet to affect daily life in real time


Unlike other fields of info
-
tech



Fusion: real world with graphics


Next big challenge in CG ?


New opportunities in advt, gaming, animation


Believable co
-
existence


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Changing Appearance

Projector

Projector

Virtual light
source

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Changing Virtual Illumination

Raskar, Welch, Low, Bandyopadhyay,


Shader Lamps
” (2000)

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Virtual Motion

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Virtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Dynamic Augmentation

Projecting on Tracked Objects

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Virtual Reflectance

Virtual Illumination

Interaction

Virtual Motion

ShaderLamps

www.ShaderLamps.com

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Projector
-
based Augmentation

www.ShaderLamps.com

Virtual Reflectance

Virtual Illumination

Interaction

Virtual Motion

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Poor Man’s Palace

Surround CG for Daily Environment

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Maya: World is an Illusion



Fake World


We all live in one



Real
-
life Fakes


Not just photos and videos but surroundings



Privacy

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Augmented Reality (AR)

With See
-
through Head
-
mounted Display

© Andrei State

Superman X
-
ray vision

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

hand
-
held

display

spatial

see
-
through

display

head
-

mounted

display

retinal

display

Head
-
attached Body
-
attached spatial

projector

projector

projector

Spatial Augmented
Reality

Classification of Augmented Reality

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Being There


Recreate Large Environments


‘BeingThere’, walk
-
around


Museums, Exhibitions, Training


Kok
-
Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life
-
Sized Projector
-
Based
Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001
(VRST 2001), November 2001.

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Desired Virtual Model

© Andrei State

Projected Guidance for Placement

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Head
-
mounted Display

Projectors

© Andrei State

Augmented Reality

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Opportunities


Entertainment

-
Animating objects, Museums, Special Effects, Laser Shows


Advertising

-
Car showrooms, Furniture store


Design and Ergonomics

-
Prototyping products, visualize material changes


Information Augmentation

-
Office, factory or assembly
Training


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006


Spatially Augmented Reality
(SAR)

HMD
-
VR

Spatially Immersive
-
VR

AR using HMD

Spatially Augmented
Reality

e.g. CAVE

Video or Optical
see
-
through

SAR,
Shaderlamps

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Steps


Preprocessing


Scan 3D object


Roughly align projectors


Compute correction


Run time


Render images of 3D model


Correct

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Steps


Preprocessing


Scan 3D object and create virtual model



Faro arm

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Steps


Preprocessing


Scan 3D object and create model


Approximately position projector(s)


Calibrate by finding pose


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Steps


Preprocessing


Scan 3D object and create model


Approximately position projector(s)


Calibrate by finding pose


Run
-
time


Inverse Rendering of 3D Models


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Shape cue allows perceptual unwrapping

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

‘Wallpaper the surface’

Shape Adaptive Projection

Minimum Stretch Display

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Shape Adaptive Projection

Minimum Stretch Display

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines

‘iLamps: Geometrically Aware and Self
-
Configurable Projectors’,

SIGGRAPH 2003

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Spatially Augmented Reality

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines

‘iLamps: Geometrically Aware and Self
-
Configurable Projectors’,

SIGGRAPH 2003

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Mitsubishi Pocket Projector


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

A new way to play tic
-
tac
-
toe


Handheld projector.



Grid projects to a fixed position on the wall.



Cursor is guided by pointing the projector.


Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Pocket Projector Games

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Warehousing

Routing

Library

Baggage

handling

Currency

Livestock
tracking

RFID

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006


Micro

Controller

Memory

Computer

READER

Micro

Controller

Memory

Conventional Passive RFID

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Tagged Books in a Library


Id

: List of books in RF range

No Precise Location Data


Are books in sorted order ?

Which book is upside down ?

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Prototype Tag

RF tag +

photosensor

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Find tag location using


handheld Projector

Photosensing

Wireless Tags

Many geometric ops

R F I

R F I D

Interactive stabilized projection


(Radio Frequency Id & Geometry)

G

Siggraph 2004

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

AR with Photosensing RFID
and Handheld Projector

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Visual feedback of 2D position

a.
Receive via RF {(
x
1
,y
1
), (
x
2
,y
2
), …} pixels

b.
Illuminate those positions

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

RFID


(Radio Frequency Identification)

RFI
G


(Radio Frequency Id and
Geometry
)

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Curved

Planar

Non
-
planar

Single

Projector

Multiple

Projectors

Projector

j

Use
r : T

?

Pocket
-
Proj

Objects

Projectors and Surfaces

2000

2000

2002

2001

1999

2002

2003

1999

1998

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Augmentation of “five senses”


Visual Displays


Auditory Displays


Haptic Displays


Locomotion Interfaces


Olfactory Displays


Taste Displays ?


Sixth Sense ..

Extending

sensory

modalities

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Sense of Balance

Galvanic Vestibular Stimulation

Masahiko Inami (UEC)

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Scent Display

T
r
a
c
k
i
n
g
n
o
s
e
d
i
r
e
c
t
i
o
n
A
i
m
i
n
g
a
t
t
h
e
n
o
s
e
A
i
r
c
a
n
n
o
n
D
e
l
i
v
e
r
i
n
g
s
c
e
n
t
e
d
a
i
r
b
y
v
o
r
t
e
x
r
i
n
g
s
C
a
m
e
r
a
P
l
a
t
f
o
r
m
Yasuyuki Yanagida, ATR

Spatio
-
Temporal Controlled Delivery of Scent

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Air Cannon

Generating a doughnut
-
shaped
vortex ring

Aperture

Push

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Displays

Contenders

Organic

LED

Light

Emitting

Polymers

E
-
Ink

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Holodust

Perlin et al

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Fake World

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Poor Man’s Palace

Surround CG for Daily Environment

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Recap


Special Effects in Real World


Photorealism yet to impact daily life


Poor Man’s Palace


Spatial Augmented Reality


Solution for fusion: Projectors, RFID, Sensors


Games for pocket projector


Display anywhere for advertising


Future Problems


Natural phenomenon, complex reflectance, other displays


All senses: haptic, olfactory, preprioception



Next Challenge in Real World Animation


Movies and games in our surroundings


3D Advertising

Mitsubishi Electric Research Laboratories

Special Effects in the Real World

Raskar 2006

Spatial Augmented Reality


Graphics


Photorealism yet to impact daily life


Special effects in real world


Spatial Augmentation


Solution for fusion


Needs hardware components


Projectors, RFID, Sensors


Next Challenge in Animation

raskar.com