brainystitchAI and Robotics

Nov 14, 2013 (3 years and 4 months ago)



Ping Gai

HFE 760



Reality Definition

Augmented Reality vs. Virtual Reality

Visual Display Systems for AR

Video Keying and Image Registration

System Design Issues

Augmented Reality Application

Augmented Reality Definition

Augmented Reality is a growing area in
virtual reality area.

An Augmented Reality system generates a
composite view for the user. It’s a
combination of the real scene viewed by the
user and a virtual scene generated by the
computer that augments the scene generated
by the computer that augmented the scene
with additional information.

Augmented Reality Definition

Typically, the real
world visual scene in an AR
display is captured by video or directly viewed.

Most current AR displays are designed using see
through HMDs which allow the observer to view
the real world directly with the naked eye.

If video is used to capture the real world, one may
use either an opaque HMD or screen
based system
to view the scene.

AR vs. VR

Virtual Reality: a computer generated, interactive,
dimensional environment in which a person
is immersed.(Aukstakanis and Blatner, 1992)

Virtual Environment is a computer generated three
dimensional scene which requires high performance
computer graphics to provide an adequate level of

The virtual world is interactive. A user requires real
time response from the system to be able to interact
with it in an effective manner.

The user is immersed in this virtual environment.

AR vs . VR

VR: the user is completely immersed in an
artificial world and becomes divorced from
the real environment. The generated world
consists entirely of computer graphics.

AR vs. AR

VR strives for a totally immersive
environment. The visual, and in some
systems aural and sense are under control of
the system.

In contrast, an AR system is augmenting the
real world sense of presence in that world.
The virtual images are merged with the real
view to create the augmented display.

AR vs. VR

For some applications , it may be desirable
to use as much as possible real world in the
scene rather creating a new scene using
computer imagery. For example, in medical
applications, the physician must view the
patient to perform surgery, in telerobotics
the operator must view the remote scene in
order to perform tasks.

AR vs. VR

A main motivation for the use of AR relates
to the computational resources necessary to
generate and update computer
scene. In VR, The more complex the scene,
the more computational resource needed to
render the scene.

AR can maintain the high
level of detail and
realistic shading that one finds in the real

AR vs. VR

NO simulator sickness. Vertigo, dizziness
introduced by sensory mismatch within
display environment can be a problem when
one uses an HMD to view a virtual world.

If the task is to show an annotation to the
real world.

Visual Display System for AR

Hardware for display visual images

A position and orientation sensing system

Hardware for combining the computer
graphics and video images into one signal

The associated system software

Visual Display System for AR

There are two main ways in which the real world
and the computer generated imagery may be
combined to form an augmented scene.

Direct viewing of the real world with overlaid computer
generated imagery as an enhancement.In this case, the
the real world and the CG images are combined

Combining the camera
captured video of the real world
with CG imagery viewed using either an opaque HMD,
or a screen
based display system.

Visual Display System for AR

Two basic types of AR system

Opaque HMD or screen
based AR.

These systems can be used to view local or remote video views
of real world scenes, combined with overlaid CG.The viewing
of a remote scene is an integral component of telepresence

Transparent HMD AR.

This system allows the observer to view the real world directly
using half
silvered mirrors with CG electronically composited
into the image. An advantage id that the real
world can be
directly viewed and manipulated.

Visual Display System for AR

Video Keying

Relevant when an opaque HMD with video
input is used to create an AR scene. Video
and synthetic image are mixed using a video
keyer to form an integrated scene.

Video Keying is a process that is widely
used in television, film production and CG.
(weather report)

Video Keying

When using video keying to design AR
scenes, one signal contains the foreground
image and the other one contains the
background image. The keyer combines the
two signal to produce a combined video
which is then sent to the display device.

Video Keying

Keying can be done using composite or
component video signals.

A composite video signal contains information
about color, luminance, and synchronization,
thus combining three piece of information into
one signal.

With component video, luminance
synchronization are combined, but chroma
information is delivered separately.

Video Keying

Chroma keying involves specifying a desired
foreground key color. Foreground areas containing
the keying color are then electronically replaced
with the background image. This results in the
background image being replaced with the fore
ground image in areas where the background
image contains chroma color.

Blue is typically used for chroma keying
(Chromakey blue) rarely shows up in human skin

Video Keying

If a video image of the real world is chosen
as the foreground image, parts of the scene
that should show the computer
world are rendered blue.

In contrast, if video of the real world is
chosen as the background image, the
computer generated environment will be
located in the foreground.

Video Keying

Video Keying

A luminance keyer works in a similar manner to a
chroma keyer, however, a luminance keyer
combines the background image wherever the
luminance values are below a certain threshold.

Luminance and chroma keyers both accomplish
the same function but usa of a chroma keyer can
result in a sharper key and has greater flexibility,
whereas a luminance keyer is typically lower
resolution and had less flexibility.



Figure is a schema of the z
key method. The z
key method requires images with both depth
information (depth map) as inputs. The z
switch compares depth information of two images
for each pixel, and connects output to the image
which is the nearer one to the camera. The result
of this is that real and virtual objects can occlude
each other correctly. This kind of merging is
impossible by the chroma
key method, even if it is
accompanied with some other positioning devices
such as magnetic or acoustic sensor, since these
devices provide only a gross measurement of

Image Registration

It’s required that the computer generated images
accurately register with the surroundings in the
real world. In certain applications, image
registration is crucial.

In terms of developing scenes for AR displays, the
problem of image registration, or positioning of
the synthetic objects within the scene in relation to
real objects, is both a difficult and important
technical problem to solve.

Image Registration

With applications that require close
registration, accurate depth information has
to be retrieved from the real world in order
to carry out the calibration of the real and
synthetic environments. Without an
accurate knowledge of the geometry of the
real world and computer
generated scene,
exact registration is not possible.

System Design Issues

Frame rate, update rate, system delays, and the
range and sensitivity of the tracking sensors.

Frame rate is a hardware
controlled variable
determining the number of images presented to the
eye per second. AR displays which show stereo
images alternatively to the left and right eye
typically use a scan rate doubler to transmit 120
frames per second so that each eye has an effective
frame rate of 60 Hz.

Update rate of the display is the rate at which new
images are presented to the viewer.

With a low update rate, if the user using an AR
display moves his head, the real and computer
generated images will no longer be registered until
the next update. Small errors in registration are
easily detectable by the visual system.

What limits the update rate is the relationship
between the complexity of the scene and the
computational power of the computer system used
to generate the scene. This relationship is esp.
important for computationally intensive
applications such as medical imaging.

System Design Issues

System Design Issues

The lag in image generation and tracking is
noticeable in all HMDs but is dramatically
accentuated with see
through HMDs. This is an
crucial problem if exact image registration is

There are two types of system delays which will
affect performance in AR: computational and
sensor delays.

As the complexity of the CG image increases, the computational
delay is a major factor determining the update of a display.

In addition, sensor delay, the time requires updating the display, is
an important variable in determining performance in augmented

Many HMD
based systems have combined latencies over 100ms,
which become very noticeable.

System Design Issues

Sensor sensitivity

The head
tracking requirements for AR displays.

A tracker must be accurate to a small fraction of a
degree in orientation and a few millimeters in position.

Errors in head orientation(pitch, roll, yaw) affect image
registration more so than error in position(x, y, z),
leading to the more stringent requirements for head
orientation tracking.

Positional tracking errors of no more than 1 to 2 mm
are maximum for AR system.

In addition to visual factors, cognitive factors
should be considered in the design as well.

Users of systems form mental models of the
system they interact with and the mental model
they form influence their performance.

With AR displays the designer must take into
account two mental models of the environment,
the mental model of the synthetic imagery and of
the real image.

The challenge will be to integrate the two stimuli
in such a way that a single mental model will be
formed of the augmented scene.

System Design Issues

System Design Issues

Integrated Mental Model

Mental Model of
real envoronment

Mental Model of

Virtual world stimuli

Auditory, haptic,

Real world stimuli

Auditory, haptic,

Reality Application



Military Training

Engineering Design

Robotics and Telerobotics

Manufacture, Maintenance and Repair

Consumer Design

Reality Application


Most of the medical applications deal with image guided surgery. Pre
operative imaging studies, such as CT or MRI scans, of the patient
provide the surgeon with the necessary view of the internal
anatomy. From these images the surgery is planned. Visualization
of the path through the anatomy to the affected area where, for
example, a tumor must be removed is done by first creating a 3D
model from the multiple views and slices in the preoperative study.
AR can be applied so that the surgical team can see the CT or MRI
data correctly registered on the patient in the operation theater
while the procedure is progressing. Being able to correctly register
the images at the point will enhance the performance of the
surgical team and eliminate the need for the painful and
cumbersome stereotactic frames currently used for registration.

Reality Application

Reality Application


Weather report

Virtual studio

Movie special effect


Reality Application

Military Training

The military has been using display in cockpits that
present information to the pilot on the windshield
of the cockpit or the visor of their flight helmet.
This is a form fo AR display.

Reality Application

Engineering Design

Distributed Coollaberation

Product visualizatoin

The scenario for this application consists of an office
manager who is working with an interior designer on the
layout of a room. The office manager intends to order
furniture for the room. On a computer monitor the pair see a
picture of the room from the viewpoint of the camera. By
interacting with various manufacturers over a network, they
select furniture by querying databases using a graphical
paradigm. The system provides descriptions and pictures of
furniture that is available from the various manufactures who
have made models available in their databases. Pieces or
groups of furniture that meet certain requirements such as
colour, manufacturer, or price may be requested. The users
choose pieces from this "electronic catalogue" and 3D
renderings of this furniture appear on the monitor along with
the view of the room. The furniture is positioned using a 3D
mouse. Furniture can be deleted, added, and rearranged
until the users are satisfied with the result; they view these
pieces on the monitor as they would appear in the actual
room. As they move the camera they can see the furnished
room from different points of view.

Reality Application

Robotics and Telerobotics

Reality Application

Maintenance, and

One application area that is currently being explored involves
mechanical maintenance and repair. In this scenario a
mechanic is assisted by an AR system while examining and
repairing a complex engine. The system may present a variety
of information to the mechanic. Annotations may identify the
name of parts, describe their function, or present other
important information like maintenance or manufacturing
records. AR may lead the mechanic through a specific task by
highlighting parts that must be sequentially removed and
showing the path of extraction. The system may also provide
safety information. Parts that are hot or electrified can be
highlighted to constantly remind the mechanic of the danger of
touching them. The mechanic may also be assisted by a remote
expert who can control what information is displayed on the
mechanic's AR system.

Reality Application

Consumer Design

House Design

Fashion, beauty industry



Virtual Environments and Advanced
Interface Design, edited by Woodrow
Barfield, Thomas A.Furness III

Augmented Reality Sites

North America


Image Guided Surgery home page

Intelligent Room project

J P Mellor's home page

Media Lab Wearable Computer page


Key project

Magic Eye project

Columbia University

Virtual Worlds research

Architectural Anatomy

University of North Carolina

Chapel Hill

Ultrasound Visualization Research

Hybrid Tracking Research

Latency in Augmented Reality

Ronald Azuma's Augmented Reality page

Telepresence Research Group

Rich Holloway's Home Page

USC Computer Graphics and Immersive Technologies Laboratory

University of Washington Human Interface Technology Lab (HITL)

Colorado School of Mines

Hazardous waste management

Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan

Augmented Reality work at the University of Toronto

Argonne National Labs

Michael E. Jebb's Augmented Reality page

NIST description of the Boeing project

Colorado State Univ.

Michael L. Croswell's Augmented Reality page

Ross Whitaker's Augmented Reality page

Mihran Tuceryan's Augmented Reality page

The WorldBoard Project

based Augmented Reality for Guiding Assembly

Rajeev Sharma, Jose Molineros, University of Illinois at Urbana

Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page

Microvision's Virtual Retinal Display