Virtual Reality

slipperhangingAI and Robotics

Nov 14, 2013 (3 years and 8 months ago)

65 views

Computer Graphics

-

Input & Interaction
-

Hanyang University


Jong
-
Il Park





Division of Electrical and Computer Engineering, Hanyang University

Objectives


Introduce the basic input devices


Physical Devices


Logical Devices


Input Modes


Event
-
driven input


Double buffering for smooth animations


Programming event input with GLUT



Virtual reality: interaction + reality





Division of Electrical and Computer Engineering, Hanyang University

Project Sketchpad


Ivan Sutherland (MIT 1963) established the basic
interactive paradigm that characterizes interactive
computer graphics:


User sees an
object

on the display


User points to (
picks
) the object with an input device
(light pen, mouse, trackball)


Object changes (moves, rotates, morphs)


Repeat





Division of Electrical and Computer Engineering, Hanyang University

Graphical Input


Devices can be described either by


Physical properties


Mouse


Keyboard


Trackball


Logical properties


What is returned to program via API


A position


An object identifier


Modes


How and when input is obtained


Request or event





Division of Electrical and Computer Engineering, Hanyang University

Physical Devices

mouse

trackball

light pen

data tablet

joy stick

space ball





Division of Electrical and Computer Engineering, Hanyang University

Incremental (Relative) Devices


Devices such as the data tablet return a position
directly to the operating system


Devices such as the mouse, trackball, and joy stick
return incremental inputs (or velocities) to the
operating system


Must integrate these inputs to obtain an absolute
position


Rotation of cylinders in mouse


Roll of trackball


Difficult to obtain absolute position


Can get variable sensitivity





Division of Electrical and Computer Engineering, Hanyang University

Logical Devices


Consider the C and C++ code


C++:
cin >> x;


C:
scanf (“%d”, &x);


What is the input device?


Can’t tell from the code


Could be keyboard, file, output from another program


The code provides
logical input


A number (an
int
) is returned to the program
regardless of the physical device





Division of Electrical and Computer Engineering, Hanyang University

Input Modes


Input devices contain a
trigger

which can be used to
send a signal to the operating system


Button on mouse


Pressing or releasing a key


When triggered, input devices return information
(their
measure
) to the system


Mouse returns position information


Keyboard returns ASCII code





Division of Electrical and Computer Engineering, Hanyang University

Request Mode


Input provided to program only when user triggers the
device


Typical of keyboard input


Can erase (backspace), edit, correct until enter (return)
key (the trigger) is depressed


Eg. scanf()





Division of Electrical and Computer Engineering, Hanyang University

Event Mode


Most systems have more than one input device, each
of which can be triggered at an arbitrary time by a
user


Each trigger generates an
event

whose measure is
put in an
event queue

which can be examined by the
user program





Division of Electrical and Computer Engineering, Hanyang University

Event Types


Window: resize, expose, iconify


Mouse: click one or more buttons


Motion: move mouse


Keyboard: press or release a key


Idle: nonevent


Define what should be done if no other event is in
queue





Division of Electrical and Computer Engineering, Hanyang University

Callbacks


Programming interface for event
-
driven input


Define a
callback function

for each type of event the
graphics system recognizes


This user
-
supplied function is executed when the
event occurs



GLUT example:
glutMouseFunc(mymouse)





Division of Electrical and Computer Engineering, Hanyang University

GLUT callbacks

GLUT recognizes a subset of the events recognized by
any particular window system (Windows, X,
Macintosh)


glutDisplayFunc


glutMouseFunc


glutReshapeFunc


glutKeyboardFunc


glutIdleFunc


glutMotionFunc, glutPassiveMotionFunc





Division of Electrical and Computer Engineering, Hanyang University

GLUT Event Loop


Recall that the last line in
main.c

for a
program using GLUT must be

glutMainLoop();


which puts the program in an infinite event loop


In each pass through the event loop, GLUT


looks at the events in the queue


for each event in the queue, GLUT executes the
appropriate callback function if one is defined


if no callback is defined for the event, the event is
ignored





Division of Electrical and Computer Engineering, Hanyang University

The display callback


The display callback is executed whenever
GLUT determines that the window should be
refreshed, for example


When the window is first opened


When the window is reshaped


When a window is exposed


When the user program decides it wants to change the
display


In
main.c


glutDisplayFunc(mydisplay)

identifies the
function to be executed


Every GLUT program must have a display callback





Division of Electrical and Computer Engineering, Hanyang University

Posting redisplays


Many events may invoke the display callback
function


Can lead to multiple executions of the display callback on
a single pass through the event loop


We can avoid this problem by instead using

glutPostRedisplay();


which sets a flag.


GLUT checks to see if the flag is set at the end of
the event loop


If set then the display callback function is
executed





Division of Electrical and Computer Engineering, Hanyang University

Animating a Display


When we redraw the display through the
display callback, we usually start by clearing
the window


glClear()


then draw the altered display


Problem: the drawing of information in the
frame buffer is decoupled from the display of its
contents


Graphics systems use dual ported memory


Hence we can see partially drawn display


See the program
single_double.c

for an example
with a rotating cube





Division of Electrical and Computer Engineering, Hanyang University

Double Buffering


Instead of one color buffer, we use two


Front Buffer
: one that is displayed but not written to


Back Buffer
: one that is written to but not displayed


Program then requests a double buffer in
main.c


glutInitDisplayMode(GL_RGB | GL_DOUBLE)


At the end of the display callback buffers are swapped

void mydisplay()

{


glClear(GL_COLOR_BUFFER_BIT|….)

.

/* draw graphics here */

.


glutSwapBuffers()

}





Division of Electrical and Computer Engineering, Hanyang University

Using the idle callback


The idle callback is executed whenever there are no
events in the event queue


glutIdleFunc(myidle)


Useful for animations

void myidle() {

/* change something */


t += dt


glutPostRedisplay();

}


Void mydisplay() {


glClear();

/* draw something that depends on t */


glutSwapBuffers();

}





Division of Electrical and Computer Engineering, Hanyang University

Using globals



The form of all GLUT callbacks is fixed


void
mydisplay()


void
mymouse(GLint button, GLint state,
GLint x, GLint y)


Must use globals to pass information to
callbacks








float t; /*global */


void mydisplay()

{

/* draw something that depends on t

}





Division of Electrical and Computer Engineering, Hanyang University

Virtual Reality?



Head
-
mounted Display (HMD)





Division of Electrical and Computer Engineering, Hanyang University

Virtual Reality?



BOOM (Binocular Omni
-
Orientation Monitor)





Division of Electrical and Computer Engineering, Hanyang University

Virtual Reality?



CAVE (Cave Automatic Virtual Environment)





Division of Electrical and Computer Engineering, Hanyang University

Virtual Reality?



Shared Virtual Environment





Division of Electrical and Computer Engineering, Hanyang University

Virtual Reality



Virtual


실제로는

존재하지

않지만
,
본질적으로

존재하는

것과

동등한

효과를

가지는





Reality


실제

일어난


,
현실감




Virtual Reality


실제로는

존재하지

않지만
,
실제로

존재하는

것과

동등한

사실이나

사건





Division of Electrical and Computer Engineering, Hanyang University

Def. VR



What is Virtual Reality?




A high
-
end user interface that involves real
-
time
simulation and interaction through multiple sensorial
channels.


(vision, sound, touch, smell, taste)





Division of Electrical and Computer Engineering, Hanyang University

VR



요소



Presence


사실적

reality,
몰입적

reality



Interaction


조작적

reality



Autonomy


일관성
,
자율성
.
행동의

reality


by D.Zelter (1992)





Division of Electrical and Computer Engineering, Hanyang University

Three

I

s of VR



Burdea, 1993.

Immersion

Interaction

Imagination

I

3





Division of Electrical and Computer Engineering, Hanyang University

Similar concepts
(
I)



Virtual reality (1989, J.Lanier, VPL)



계산기에

의해

합성된

인공적

세계




Artificial reality(Mid
-

70, M.Krueger)


VIDEOPLACE:
실제처럼

느끼면서

컴퓨터가

만든

세계에

참여하여

상호작용하는





Cyberspace(1983, William Gibson)



a single artificial reality that could be experienced
simultaneously by thousands of people worldwide






Division of Electrical and Computer Engineering, Hanyang University

Similar concepts
(
II)



Telepresence or Tele
-
existence


우주공간
,
심해



그대로는

체험하기

불가능한

곳의

상황을

현실감

넘치게

제시


현실의

복제라는

점에서

VR


차별화됨



Augmented reality
-

virtual objects in real world



Augmented virtuality
-

real objects in virtual world



Mixed reality



기타
:
synthetic reality, virtual environment, virtual
world, artificial life





Division of Electrical and Computer Engineering, Hanyang University

Augmented Reality

(
ARTEMIS, U.Toronto)





Division of Electrical and Computer Engineering, Hanyang University

Augmented Virtuality

(www.cyberworldcorp.com)





Division of Electrical and Computer Engineering, Hanyang University

MR Includes


virtual

AV

AR

real





Division of Electrical and Computer Engineering, Hanyang University

Reality vs. Virtuality



Reality = Unmodelled

z

Virtuality = Modelled

Reality
-
Virtuality Continuum

Real

Environment

Virtual

Environment

Extent of World Knowledge Continuum

World

Unmodelled

World

Completely

Modelled





Division of Electrical and Computer Engineering, Hanyang University

History of VR(I)



Sensorama Simulator (Morton Heilig, 1962)



head
-
mounted television(


, 1960)



Head
-
Mounted Display(Ivan Sutherland, 1966)



Scene generator(Evans and Sutherland)


simple scene about 200
-
400 polygons at 20 scenes/sec
(1973)


METAPLAY(1970) ,VIDEOPLACE(1974, Myron Krueger)


Flight simulators etc. by military(70s to early 80s)





Division of Electrical and Computer Engineering, Hanyang University

Sensorama Simulator

Heilig, US Patent #3,050,870, 1962





Division of Electrical and Computer Engineering, Hanyang University

Ivan Sutherland’s HMD (1966+)





Division of Electrical and Computer Engineering, Hanyang University

Videoplace by M.Krueger





Division of Electrical and Computer Engineering, Hanyang University

History of VR(II)


VIVED(Virtual Visual Environment Display) project (NASA)


LCD(Liquid Crystal Display)
-
based HMD (1981)


Polhemus noncontact tracker



user

s head tracking


Sensing glove (Fisher, 1985)


VIVED
-
> VIEW(Virtual Interface Environment Workstation)


3D virtual sound source (Fisher and Wenzel, 1988)


image wireframe rendering with flat shaded surface






Division of Electrical and Computer Engineering, Hanyang University

History of VR(III)


VPL, 1st commercial company selling VR products (1987)


DataGloves


EyePhones
-

1st HMD



Vision

, 1st integrated commercial VR workstation (Division
Ltd., UK, 1991)



WorldToolKit

, 1st VR development software


(Sense8, 1992)


Mixed Reality (ATR, 1994)


Plenoptic modeling and image
-
based rendering (McMillan
and Bishop, 1995)


Virtualized Reality (CMU, 1996)





Division of Electrical and Computer Engineering, Hanyang University

The VPL DataGlove (1987)





Division of Electrical and Computer Engineering, Hanyang University

NASA VIEW system (1989)





Division of Electrical and Computer Engineering, Hanyang University


VRML(Virtual Reality Modeling Language)


VRML 1.0 (1995)


VRML 2.0 (1996), VRML97: dynamic scene animation


Currently


Java 3D API, VRTP(VR Transfer Protocol)



Mixed reality applications

History of VR(IV)





Division of Electrical and Computer Engineering, Hanyang University

Recent Technologies in VR (I)


Computer vision and computer graphics


Image
-
based rendering


Large scale visualization


mosaic


Robust geometric registration


Motion/depth
-
keying


Occlusion


Image segmentation





Division of Electrical and Computer Engineering, Hanyang University

Recent Technologies in VR (II)


Devices


See
-
through HMD (ST
-
HMD)


Optical, Video ST
-
HMD


Wearable equipments


New tracking sensors for outdoor





Division of Electrical and Computer Engineering, Hanyang University

Recent Technologies in VR (III)


New interfaces


Gesture recognition


Tangible interface


Haptic devices


Hybrid interfaces





Division of Electrical and Computer Engineering, Hanyang University

Applications of VR (I)


Mobile


Outdoor navigation with wearable equipments


Military training, sightseeing


Collaborative


Simultaneous and interactive multi
-
users


Shared virtual environment


Commercial


Broadcasting: character service


Entertainment


Film industry





Division of Electrical and Computer Engineering, Hanyang University

Applications of VR (II)


Various simulator


Flight, car driving


Military training


Building, factory, City modeling


Medical application


Human body 3D modeling (CT, MR)


Tele
-
surgery


Limitations of future VR


Technology, interface, social acceptance





Division of Electrical and Computer Engineering, Hanyang University

Relation of VR, CG, and CV



VR





Division of Electrical and Computer Engineering, Hanyang University

Major Issues in MR


Accurate
registration

between virtual and real world


Realistic
display
(vision, sound, touch, smell, taste)


All should be in
REAL
-
TIME


Analysis Time(TA)


Synthesis Time(TS)


Feedback Time(= TA + TS)


Should be small enough


Should not annoy human

s feeling of reality





Division of Electrical and Computer Engineering, Hanyang University

Critical Enabling Technology


Accurate and fast registration between RW and
VW !!!


In more detail, a good
tracker


Fast


Light and small


Low power
-
consumption


Accurate


Immune to interference





Division of Electrical and Computer Engineering, Hanyang University

Where Are We Heading?

CG/CV

Computers

HCI

Mechanics

Sensors

Psychology

Human science

Art

?