Immersive actor feedback system

yardbellΤεχνίτη Νοημοσύνη και Ρομποτική

14 Νοε 2013 (πριν από 4 χρόνια και 6 μήνες)

87 εμφανίσεις


Research & Innovation

1

Immersive actor feedback system


What sees the actor on set ?

( Answer! )


Research & Innovation

2

Immersive actor feedback system


Viewpoint dependent projection


Virtual reality: CAVE system


Integration into Chroma
-
keying environment


Retro
-
reflective cloth (BBC patent)


Research & Innovation

3

Immersive actor feedback system


Retro
-
reflective cloth


“truematte” developed by BBC

Camera with LED ring

LED off

LED on


Research & Innovation

4

Immersive actor feedback system


Projector

3D model of

Actor

3D model of

virtual scene

Viewpoint

Renderer

Head
position

Mask generation


Research & Innovation

5

Immersive actor feedback system


View dependent projection

The Ambassadors (1533), by Hans Holbein the Younger
.
National Gallery London


Research & Innovation

6

Immersive actor feedback


Head tracking

Fast template matching
filter on voxel data


Research & Innovation

7

Production Visualisation Tools


real
-
time, low
-
quality

Monitor

Actor Feedback

Director’s View

3D model of

real scene

3D model of

virtual scene

On
-
set

Planning,

On
-
set


Research & Innovation

8

Immersive actor feedback system


Research & Innovation

9

Bamzooki


BBC Children Production


Research & Innovation

10

Free
-
viewpoint
video for sports


‘Virtual Replay’ of football
scenes


Allows viewer interactive
review


Manually modelled
offline


Not
realistic


Research & Innovation

11

Free
-
viewpoint video for Sport visualisation


Piero













Overcome limitations: use multi
-
camera system,
iview project


Research & Innovation

12

Applying multi
-
camera techniques to sport scenarios


Test shoot England
-
Wales in Old Trafford 9 Oct 2004


Research & Innovation

13

Where to put the cameras?


Broadcast coverage cameras



Pro:


Works without extra setup


Cons:


Unreliable framing


Research & Innovation

14

Applying multi
-
camera techniques to sport scenarios


Calibration with wand


Problems:


Cameras knocked overnight


Cameras not rigidly mounted


Research & Innovation

15

Line
-
based calibration


Colour
-
based edge
detection


Line
-
fitting


Calibration of camera
parameters (T,R,f) against
dimensions of the pitch



Runs at video frame rate


Research & Innovation

16

Processing
-

Segmentation


Chroma
-
key against green



Research & Innovation

17

Processing


3D Reconstruction


3D representations


Billboard (flat polygon, sprite)


Visual hull (Shape from silhouette)


Research & Innovation

18

Results

From 16 cameras, visual hull + graph
-
cut shape optimisation

(results from iview project, jointly with University of Surrey)



Research & Innovation

19

Results


Research & Innovation

20

Summary


Computer vision techniques are important building
blocks for the automated generation of visual effects


For the integration of video and graphics several
optical phenomena have to be harmonised. In
descending order of importance:


Camera perspective, occlusions, shadows, ..


Visual feedback systems integrated into the
production pipeline are important to allow interaction
with virtual objects


Free
-
viewpoint video allows new insights


Goal: Stream 3D to the viewer at home




Research & Innovation

21



Research & Innovation

22

3D Modelling


Graph
-
cut












Details: A Bayesian Framework for Simultaneous Matting and 3D
Reconstruction,
J.
-
Y. Guillemaut, A. Hilton, J. Starck, J. Kilner, and O. Grau,
3DIM’07


Research & Innovation

23

Results


Video from ORIGAMI demo production


Research & Innovation

24

Applications: Requirements


I

coarse

rich detailed

3
-
D

II

III

IV

2 1/2
-
D

2
-
D

Applications:

a) virtual studio

b) On
-
set Visualisation

c) Pre
-
Visualisation

d) Interactive

e) Digital effects

Optical phenomenon:

1) camera perspective

2) occlusions

3) shadows

4) reflections

1

2

3

4

I

II

III

IV

a

b

e

Shape classes:

d

c


Research & Innovation

25

Introduction


Conventional work flow in

(feature) TV or film productions

Planning

Phase

On
-
set

Phase

Post

Production

Animations /

Virtual Set

Story

board

Final

Programme

expensive


Research & Innovation

26

Introduction


ORIGAMI approach


Mixed
-
reality techniques for visualisation through the production

Planning

Phase

On
-
set

Phase

Post

Production

Animations / Virtual Set

Story

board

Final

Programme