Augmented-Reality

brainystitchAI and Robotics

Nov 14, 2013 (3 years and 11 months ago)

69 views

Augmented
-
Reality

Ping Gai

HFE 760

Augmented
-
Reality


Augmented
-

Reality Definition


Augmented Reality vs. Virtual Reality


Visual Display Systems for AR


Video Keying and Image Registration


System Design Issues


Augmented Reality Application

Augmented Reality Definition


Augmented Reality is a growing area in
virtual reality area.


An Augmented Reality system generates a
composite view for the user. It’s a
combination of the real scene viewed by the
user and a virtual scene generated by the
computer that augments the scene generated
by the computer that augmented the scene
with additional information.

Augmented Reality Definition


Typically, the real
-
world visual scene in an AR
display is captured by video or directly viewed.


Most current AR displays are designed using see
-
through HMDs which allow the observer to view
the real world directly with the naked eye.


If video is used to capture the real world, one may
use either an opaque HMD or screen
-
based system
to view the scene.

AR vs. VR


Virtual Reality: a computer generated, interactive,
three
-
dimensional environment in which a person
is immersed.(Aukstakanis and Blatner, 1992)


Virtual Environment is a computer generated three
dimensional scene which requires high performance
computer graphics to provide an adequate level of
realism.


The virtual world is interactive. A user requires real
-
time response from the system to be able to interact
with it in an effective manner.


The user is immersed in this virtual environment.

AR vs . VR


VR: the user is completely immersed in an
artificial world and becomes divorced from
the real environment. The generated world
consists entirely of computer graphics.



AR vs. AR


VR strives for a totally immersive
environment. The visual, and in some
systems aural and sense are under control of
the system.


In contrast, an AR system is augmenting the
real world sense of presence in that world.
The virtual images are merged with the real
view to create the augmented display.

AR vs. VR


For some applications , it may be desirable
to use as much as possible real world in the
scene rather creating a new scene using
computer imagery. For example, in medical
applications, the physician must view the
patient to perform surgery, in telerobotics
the operator must view the remote scene in
order to perform tasks.

AR vs. VR


A main motivation for the use of AR relates
to the computational resources necessary to
generate and update computer
-
generated
scene. In VR, The more complex the scene,
the more computational resource needed to
render the scene.


AR can maintain the high
-
level of detail and
realistic shading that one finds in the real
world.

AR vs. VR


NO simulator sickness. Vertigo, dizziness
introduced by sensory mismatch within
display environment can be a problem when
one uses an HMD to view a virtual world.


If the task is to show an annotation to the
real world.

Visual Display System for AR


Hardware for display visual images


A position and orientation sensing system


Hardware for combining the computer
graphics and video images into one signal


The associated system software

Visual Display System for AR


There are two main ways in which the real world
and the computer generated imagery may be
combined to form an augmented scene.


Direct viewing of the real world with overlaid computer
generated imagery as an enhancement.In this case, the
the real world and the CG images are combined
optically.


Combining the camera
-
captured video of the real world
with CG imagery viewed using either an opaque HMD,
or a screen
-
based display system.

Visual Display System for AR


Two basic types of AR system


Opaque HMD or screen
-
based AR.

These systems can be used to view local or remote video views
of real world scenes, combined with overlaid CG.The viewing
of a remote scene is an integral component of telepresence
applications.


Transparent HMD AR.

This system allows the observer to view the real world directly
using half
-
silvered mirrors with CG electronically composited
into the image. An advantage id that the real
-
world can be
directly viewed and manipulated.

Visual Display System for AR

Video Keying


Relevant when an opaque HMD with video
input is used to create an AR scene. Video
and synthetic image are mixed using a video
keyer to form an integrated scene.


Video Keying is a process that is widely
used in television, film production and CG.
(weather report)

Video Keying


When using video keying to design AR
scenes, one signal contains the foreground
image and the other one contains the
background image. The keyer combines the
two signal to produce a combined video
which is then sent to the display device.

Video Keying


Keying can be done using composite or
component video signals.


A composite video signal contains information
about color, luminance, and synchronization,
thus combining three piece of information into
one signal.


With component video, luminance
synchronization are combined, but chroma
information is delivered separately.

Video Keying


Chroma keying involves specifying a desired
foreground key color. Foreground areas containing
the keying color are then electronically replaced
with the background image. This results in the
background image being replaced with the fore
ground image in areas where the background
image contains chroma color.


Blue is typically used for chroma keying
(Chromakey blue) rarely shows up in human skin
tones.

Video Keying


If a video image of the real world is chosen
as the foreground image, parts of the scene
that should show the computer
-
generated
world are rendered blue.


In contrast, if video of the real world is
chosen as the background image, the
computer generated environment will be
located in the foreground.

Video Keying

Video Keying


A luminance keyer works in a similar manner to a
chroma keyer, however, a luminance keyer
combines the background image wherever the
luminance values are below a certain threshold.


Luminance and chroma keyers both accomplish
the same function but usa of a chroma keyer can
result in a sharper key and has greater flexibility,
whereas a luminance keyer is typically lower
resolution and had less flexibility.

Z
-
keying


Z
-
keying


Figure is a schema of the z
-
key method. The z
-
key method requires images with both depth
information (depth map) as inputs. The z
-
key
switch compares depth information of two images
for each pixel, and connects output to the image
which is the nearer one to the camera. The result
of this is that real and virtual objects can occlude
each other correctly. This kind of merging is
impossible by the chroma
-
key method, even if it is
accompanied with some other positioning devices
such as magnetic or acoustic sensor, since these
devices provide only a gross measurement of
position.


Image Registration


It’s required that the computer generated images
accurately register with the surroundings in the
real world. In certain applications, image
registration is crucial.


In terms of developing scenes for AR displays, the
problem of image registration, or positioning of
the synthetic objects within the scene in relation to
real objects, is both a difficult and important
technical problem to solve.

Image Registration


With applications that require close
registration, accurate depth information has
to be retrieved from the real world in order
to carry out the calibration of the real and
synthetic environments. Without an
accurate knowledge of the geometry of the
real world and computer
-
generated scene,
exact registration is not possible.

System Design Issues


Frame rate, update rate, system delays, and the
range and sensitivity of the tracking sensors.


Frame rate is a hardware
-
controlled variable
determining the number of images presented to the
eye per second. AR displays which show stereo
images alternatively to the left and right eye
typically use a scan rate doubler to transmit 120
frames per second so that each eye has an effective
frame rate of 60 Hz.





Update rate of the display is the rate at which new
images are presented to the viewer.


With a low update rate, if the user using an AR
display moves his head, the real and computer
-
generated images will no longer be registered until
the next update. Small errors in registration are
easily detectable by the visual system.


What limits the update rate is the relationship
between the complexity of the scene and the
computational power of the computer system used
to generate the scene. This relationship is esp.
important for computationally intensive
applications such as medical imaging.


System Design Issues

System Design Issues


The lag in image generation and tracking is
noticeable in all HMDs but is dramatically
accentuated with see
-
through HMDs. This is an
crucial problem if exact image registration is
required.


There are two types of system delays which will
affect performance in AR: computational and
sensor delays.


As the complexity of the CG image increases, the computational
delay is a major factor determining the update of a display.


In addition, sensor delay, the time requires updating the display, is
an important variable in determining performance in augmented
reality.


Many HMD
-
based systems have combined latencies over 100ms,
which become very noticeable.



System Design Issues


Sensor sensitivity


The head
-
tracking requirements for AR displays.


A tracker must be accurate to a small fraction of a
degree in orientation and a few millimeters in position.


Errors in head orientation(pitch, roll, yaw) affect image
registration more so than error in position(x, y, z),
leading to the more stringent requirements for head
-
orientation tracking.


Positional tracking errors of no more than 1 to 2 mm
are maximum for AR system.


In addition to visual factors, cognitive factors
should be considered in the design as well.


Users of systems form mental models of the
system they interact with and the mental model
they form influence their performance.


With AR displays the designer must take into
account two mental models of the environment,
the mental model of the synthetic imagery and of
the real image.


The challenge will be to integrate the two stimuli
in such a way that a single mental model will be
formed of the augmented scene.


System Design Issues


System Design Issues

Integrated Mental Model

Mental Model of
real envoronment

Mental Model of
synthetic
envoronment

Virtual world stimuli

Auditory, haptic,
visual

Real world stimuli

Auditory, haptic,
visual

Augmented
-
Reality Application


Medical


Entertainment


Military Training


Engineering Design


Robotics and Telerobotics


Manufacture, Maintenance and Repair


Consumer Design

Augmented
-
Reality Application


Medical

Most of the medical applications deal with image guided surgery. Pre
-
operative imaging studies, such as CT or MRI scans, of the patient
provide the surgeon with the necessary view of the internal
anatomy. From these images the surgery is planned. Visualization
of the path through the anatomy to the affected area where, for
example, a tumor must be removed is done by first creating a 3D
model from the multiple views and slices in the preoperative study.
AR can be applied so that the surgical team can see the CT or MRI
data correctly registered on the patient in the operation theater
while the procedure is progressing. Being able to correctly register
the images at the point will enhance the performance of the
surgical team and eliminate the need for the painful and
cumbersome stereotactic frames currently used for registration.

Augmented
-
Reality Application


Augmented
-
Reality Application


Entertainment


Weather report


Virtual studio


Movie special effect


Advertisement

Augmented
-
Reality Application


Military Training

The military has been using display in cockpits that
present information to the pilot on the windshield
of the cockpit or the visor of their flight helmet.
This is a form fo AR display.

Augmented
-
Reality Application


Engineering Design

Distributed Coollaberation

Product visualizatoin


The scenario for this application consists of an office
manager who is working with an interior designer on the
layout of a room. The office manager intends to order
furniture for the room. On a computer monitor the pair see a
picture of the room from the viewpoint of the camera. By
interacting with various manufacturers over a network, they
select furniture by querying databases using a graphical
paradigm. The system provides descriptions and pictures of
furniture that is available from the various manufactures who
have made models available in their databases. Pieces or
groups of furniture that meet certain requirements such as
colour, manufacturer, or price may be requested. The users
choose pieces from this "electronic catalogue" and 3D
renderings of this furniture appear on the monitor along with
the view of the room. The furniture is positioned using a 3D
mouse. Furniture can be deleted, added, and rearranged
until the users are satisfied with the result; they view these
pieces on the monitor as they would appear in the actual
room. As they move the camera they can see the furnished
room from different points of view.



Augmented
-
Reality Application


Robotics and Telerobotics


Augmented
-
Reality Application


Manufacturing,
Maintenance, and
Repair


One application area that is currently being explored involves
mechanical maintenance and repair. In this scenario a
mechanic is assisted by an AR system while examining and
repairing a complex engine. The system may present a variety
of information to the mechanic. Annotations may identify the
name of parts, describe their function, or present other
important information like maintenance or manufacturing
records. AR may lead the mechanic through a specific task by
highlighting parts that must be sequentially removed and
showing the path of extraction. The system may also provide
safety information. Parts that are hot or electrified can be
highlighted to constantly remind the mechanic of the danger of
touching them. The mechanic may also be assisted by a remote
expert who can control what information is displayed on the
mechanic's AR system.




Augmented
-
Reality Application


Consumer Design

House Design

Fashion, beauty industry

….

Reference


http://www.cs.rit.edu/~jrv/research/ar/


Virtual Environments and Advanced
Interface Design, edited by Woodrow
Barfield, Thomas A.Furness III


Augmented Reality Sites
-

North America


MIT


Image Guided Surgery home page



Intelligent Room project



J P Mellor's home page



Media Lab Wearable Computer page



CMU


Z
-
Key project



Magic Eye project



Columbia University


Virtual Worlds research



Architectural Anatomy



University of North Carolina
-

Chapel Hill


Ultrasound Visualization Research



Hybrid Tracking Research



Latency in Augmented Reality



Ronald Azuma's Augmented Reality page



Telepresence Research Group



Rich Holloway's Home Page



USC Computer Graphics and Immersive Technologies Laboratory



University of Washington Human Interface Technology Lab (HITL)



Colorado School of Mines



Hazardous waste management

Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan


Augmented Reality work at the University of Toronto



Argonne National Labs
-

Michael E. Jebb's Augmented Reality page



NIST description of the Boeing project



Colorado State Univ.
-

Michael L. Croswell's Augmented Reality page



Ross Whitaker's Augmented Reality page



Mihran Tuceryan's Augmented Reality page



The WorldBoard Project



Vision
-
based Augmented Reality for Guiding Assembly

Rajeev Sharma, Jose Molineros, University of Illinois at Urbana
-
Champaign


Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page



Microvision's Virtual Retinal Display