AN OPERATING ENVIRONMENT FOR LARGE SCALE VIRTUAL REALITY

slipperhangingΤεχνίτη Νοημοσύνη και Ρομποτική

14 Νοε 2013 (πριν από 3 χρόνια και 11 μήνες)

262 εμφανίσεις

ANOPERATINGENVIRONMENT
FOR LARGE SCALE VIRTUAL
REALITY
A
THESIS SUBMITTED TO THE
U
NIVERSITY OF
M
ANCHESTER
FOR THE DEGREE OF
D
OCTOR OF
P
HILOSOPHY
IN THE
F
ACULTY OF
S
CIENCE AND
E
NGINEERING
April 1999
By
Stephen Robert Pettifer
Department of Computer Science
Contents
Abstract 12
Declaration 13
Copyright Notice 14
Acknowledgements 15
1 Introduction 16
1.1 VR or not VR?...............................17
1.1.1 The defining characteristic of VR................18
1.1.2 The invisible computer and ‘presence’.............18
1.1.3 The primacy of presence.....................21
1.2 VR applications,and the implications of presence..........23
1.2.1 All things to all people?.....................25
1.3 The current state and the future role of ‘systems’...........25
1.4 Areas for improvement..........................27
1.4.1 ‘Richness’ of behaviour in environments...........27
1.4.2 Multiple Applications and Interworking...........28
1.5 The ‘metaphysical principle’.......................29
1.6 Implications for a system’s architecture................31
2
1.6.1 Aims for a mature VR system..................32
1.7 Thesis structure..............................33
2 The current state of affairs 34
2.1 Aframework for analysing VR platforms...............34
2.1.1 Existing surveys of VR systems.................35
2.2 An alternative viewof existing VR systems..............36
2.2.1 Physical,phenomenal and the psychology of presence...37
2.2.2 Perception in the physical world................38
2.2.3 Perception in the virtual world.................40
2.3 Elements of a VR system.........................41
2.3.1 Application Interfacing.....................41
2.3.2 World management.......................42
2.3.3 Presentational mechanisms...................43
2.4 ASurvey of VR systems.........................44
2.4.1 Active Systems..........................45
2.4.2 Active toolkits and kernels...................52
2.4.3 Inactive Systems.........................55
2.5 Summary and conclusions........................59
3 Requirements of a large scale VR system 69
3.1 The world model.............................70
3.1.1 The Deva world model......................71
3.1.2 Virtual worlds fromentities and environments........78
3.1.3 The Metaphysical principle and systemindependence...80
3.1.4 Practical considerations.....................81
3.1.5 Summary of world model requirements............84
3
3.2 Interfacing.................................84
3.3 Presentation issues............................86
3.4 Summary of requirements........................88
4 Scoping an architecture for VR in the large 89
4.0.1 Methodology...........................90
4.1 Perception and reality as the starting point..............91
4.2 Issues of distribution...........................93
4.2.1 Peer to peer systems.......................93
4.2.2 Client/server systems......................95
4.2.3 Practical systems in the peer-peer/client-server spectrum.97
4.3 Perceptual solutions to synchronisation in virtual environments..99
4.3.1 Subjectivity and synchronisation................100
4.4 Degrees of subjectivity..........................100
4.4.1 Phenomenal subjectivity.....................102
4.4.2 Viewpoint subjectivity......................102
4.4.3 Presentational subjectivity....................104
4.4.4 Total subjectivity.........................104
4.5 Relating subjectivities to architecture..................106
4.6 Phenomenal subjectivity and the relaxing of synchronisation....107
4.6.1 An experiment..........................108
4.7 Exploiting phenomenal subjectivity in a ‘Kantianhybrid’ architecture109
4.7.1 Other benefits...........................110
4.7.2 Limitations of the model.....................112
4.7.3 Decoupling the subjective and objective............113
4.7.4 The Metaverse and elements of perception..........114
4.7.5 Applying ‘perceptual filters’..................119
4
4.7.6 More complex primitives....................119
4.7.7 Summary.............................119
5 Server,clients,infrastructure and content 121
5.1 The server.................................121
5.1.1 Server infrastructure:run-time support of entities and en-
vironments............................122
5.1.2 Server content:the root environments.............127
5.2 Server performance and scalability...................131
5.2.1 Parallelismfor virtual environments..............131
5.2.2 Parallelismin existing VR systems...............132
5.2.3 Extracting parallelismfromentities and environments...133
5.2.4 Distribution strategy for environments and entities.....140
5.3 The client viewers.............................141
5.4 Summary..................................142
6 Implementation 144
6.1 Selecting a platform............................145
6.2 Topology and Networking........................145
6.2.1 Inter-process communication..................146
6.2.2 Message formats.........................148
6.3 Infrastructural support for the Deva model..............150
6.3.1 Inter-entity communication...................150
6.3.2 Routing message:Names and Distributed Name Serving..151
6.3.3 The distributed name serving mechanism...........152
6.4 Translation and execution........................160
6.4.1 Compilers,interpreters and virtual machines.........161
5
6.4.2 Making Unix into a Deva VM..................163
6.4.3 Dynamic Loading:entities on demand............165
6.4.4 Lightweight threads.......................167
6.5 Content:implementing the root environments............170
6.5.1 Spatial anddisplay management:integrating with the MAVERIK
micro-kernel............................170
6.5.2 Integration with Deva......................173
6.6 The client Viewers.............................175
6.6.1 Linear dead-reckoning......................175
6.6.2 Spline-based dead-reckoning..................176
6.6.3 Controlling the viewer......................176
6.6.4 Multicasting............................177
6.6.5 Migration of the locus of control................178
6.7 Interfacing to the Deva system.....................180
6.7.1 The Application Programmer’s Interface...........181
6.8 Summary..................................183
7 Results 184
7.1 Aims revisited...............................184
7.2 Functionality................................185
7.2.1 Environments,objects and the metaverse...........185
7.2.2 The ‘Fishcages’..........................186
7.2.3 Features of the metaverse model................187
7.2.4 Other features...........................189
7.2.5 Reality failure!..........................198
7.3 Integration of multiple display techniques and interaction devices.199
7.3.1 Input and output devices....................200
6
7.3.2 AC3Dreader...........................201
7.3.3 VRAD...............................201
7.3.4 Primitive objects.........................203
7.3.5 Avatars...............................203
7.4 Performance................................204
7.4.1 Execution environment performance..............205
7.4.2 Communication layer performance..............209
7.4.3 Subjective/objective architecture performance........212
8 Conclusions and future work 218
8.1 The future of Deva............................219
8.1.1 Perceptual culling........................220
8.1.2 Multi-threaded MAVERIK....................220
8.1.3 Twines...............................221
8.1.4 Causal degradation........................222
8.1.5 Evaluation:social and technical................223
8.2 Final comments..............................224
A Example Code 226
A.1 Deva language version..........................226
A.2 Hybrid code................................227
A.3 Generated output.............................228
B Overviewof API functions 235
B.1 Name management............................235
B.2 Messages..................................236
B.3 Handler functions.............................236
B.4 General functions.............................237
7
C The Distributed Legible City 239
C.1 Exhibition guide text...........................239
C.1.1 Credits...............................240
Bibliography 241
8
List of Figures
2.1 Perceptual pipeline of objective reality.................39
2.2 Perceptual pipeline applied to a virtual environment........61
2.3 The Bamboo architecture.........................62
2.4 The DIVE3 architecture..........................62
2.5 The dVS architecture...........................63
2.6 The MASSIVE-2 architecture.......................63
2.7 The Spline architecture..........................64
2.8 The Superscape architecture.......................64
2.9 The MR architecture...........................65
2.10 The WTK architecture..........................65
2.11 The WAVES architecture.........................66
2.12 The BrickNet architecture........................67
2.13 The AVIARY architecture........................68
3.1 Defining environments and entities...................74
3.2 Instantiating environments and entities................75
3.3 An example entity class hierarchy...................78
3.4 Root environments of the metaverse graph..............79
3.5 Matrix of application types.......................86
4.1 Two people’s perception of the real world...............91
9
4.2 The perceptual model interpreted as a single monolithic process..92
4.3 Traditional peer-to-peer architecture..................94
4.4 The perceptual model interpreted as traditional client/server ar-
chitecture..................................96
4.5 Phenomenal subjectivity.........................101
4.6 Viewpoint subjectivity..........................103
4.7 Viewpoint subjectivity..........................105
4.8 Viewpoint subjectivity..........................106
4.9 Hybrid server architecture........................111
4.10 Extending the metaverse with a ‘gravity’ environment........117
4.11 Example Deva code for ‘NewtonsWorld’................118
5.1 Three environments,containing three entities each..........134
5.2 Bindings between environment and entity...............136
5.3 Partial definition of the Holospace environment...........137
5.4 Instantiation of an entity in Holospace.................138
5.5 Typical distribution of entities,and pure/impure environmental
components................................141
5.6 Relationship between viewers and the server cluster.........143
6.1 Format of an administrative message..................149
6.2 Server configuration showing Local Directory Manager.......154
6.3 Local Directory Manager with Name Cacheing............156
6.4 Virtual Server extension to name-lookup algorithm.........158
6.5 Integration of MAVERIKmodules with server and clients......174
6.6 Asimple example application using the Deva API..........182
7.1 An empty holospace environment.This is the ‘default’ environ-
ment seen if a viewer connects to an ‘empty’ server cluster.....186
10
7.2 ‘Littlefish’ with trailing bubbles,and ‘Bigfish’ seen frominside the
Bigfish cage................................188
7.3 AVRADobject is introduced as background.............190
7.4 Two avatars/users are introduced...................191
7.5 The lab as seen fromthe users’ viewpoints..............192
7.6 Riding in the legible city.........................193
7.7 AviewfromKarlsruhe..........................194
7.8 Karlsruhe fromabove...........................194
7.9 Amsterdamfromabove.........................195
7.10 Steve has been moved inside one of the cages.............196
7.11 Steve’s viewfrominside the cage....................197
7.12 The Lab model,showing names for real entities...........197
7.13 Freeware models read in fromthe AC3Dformat...........202
7.14 VRAD is used here to provide a realistically shaded setting for
three static AC3Dmodels........................202
7.15 Aprocess plant (model provided by CADCentre)..........203
7.16 Awalking avatar.............................204
7.17 Workload for 100 to 1000 entities on 1 to 5 servers..........208
7.18 Workload for 1000 to 19000 entities on 1 to 5 servers.........208
7.19 Workload for 300 entities on 1 to 5 servers against ideal scaling factor209
7.20 Single server workload plotted against increasing number of entities.210
7.21 Workload for a single server plotted against the ‘ideal’ workload..210
7.22 The particle systemwith 150 particles.................213
7.23 Framerates for stand-alone system,Deva viewer and ‘pure render-
ing’ against number of particles.....................213
7.24 Workloads for the stand-alone systemand Deva viewer according
to the number of particles managed..................217
11
Abstract
Improvements in processors and graphics hardware are making Virtual Reality
(VR) increasingly attractive as means of human/computer interaction.Although
there are compelling stand-alone demonstrations of specific aspects of the field,
there is little in the way of generic software that supports andintegrates the needs
of large or complex virtual environments.Future VR systems will require ‘large
scale’ issues to be addressed.These include complex graphics and behaviour of
objects and large numbers of applications and geographically distributed users.
However,beyond the graphics and networking challenges,a core issue is the
specification of the environments themselves.For sophisticated environments,
more appropriate methods are requiredthanthe conventional programming tools
provided by existing VR systems.For virtual environments,this specification
approaches a description of the metaphysical properties of the space being de-
scribed and the objects it contains.In such a specification,the relationship be-
tween the user and the environment is an important consideration.
This thesis argues that by building a framework for describing virtual environ-
ments upon such a metaphysically orientated specification of the environments,
that takes explicit account of the relation between perceiver and the perceived,
many of the existing problems of VR can be successfully addressed.
A VR systems architecture is developed that supports investigation of this per-
ceptual divide between perceiver and perceived,and is demonstrably capable of
facilitating these investigations in the context of ‘large-scale’ VRas definedabove.
12
Declaration
No portion of the work referred to in this thesis has been submit-
ted in support of an application for another degree or qualifica-
tion of this or any other university or other institution of learning.
All trademarks cited within this work are acknowledged by the
author.
13
Copyright Notice
1.Copyright in text of this thesis rests with the Author.
Copies (by any process) either in full,or of extracts,may
be made only in accordance with instructions given by
the Author and lodged in the John Rylands University
Library of Manchester.Details may be obtainedfromthe
Librarian.This page must formpart of any such copies
made.Further copies (by any process) of copies made
in accordance with such instructions may not be made
without the permission (in writing) of the Author.
2.The ownership of any intellectual property rights which
may be described in this thesis is vested in the Univer-
sity Of Manchester,subject to any prior agreement to
the contrary,and may not be made available for use by
third parties without the written permission of the Uni-
versity,which will prescribe the terms and conditions of
any such agreement.
Further information on the conditions under which disclo-
sures and exploitation may take place is available from the
Head Of Department of Computer Science.
14
Acknowledgements
This work has been funded by an EPSRC postgraduate studentship.
I would like to thank my supervisor Dr.Adrian West for his support and advice
throughout this project.Thanks are due also to all my colleagues in the Advanced
Interfaces Group,and in particular to Mr.Jon Cook and Dr.Simon Gibson for
their contributions to the work,and to Mr.James Marsh for his patience in proof-
reading this document.
I would like to recognise here the unfailing support that I have received frommy
family and friends.To Mum,Dad,Angela and Jane,this would not have worked
without you.
15
Chapter 1
Introduction
Sometime in the mid to late nineteen eighties a newformof interaction between
humans and computers came into public awareness.Ever since then,virtual real-
ity (VR) with its associated technologies,applications and social and psycholog-
ical implications has been the focus of much popular media attention and spec-
ulation.It is easy to see why:although the initial examples of virtual environ-
ments were—not surprisingly for a technology in its infancy—somewhat crude,
the vision being able to immerse oneself in a computer generated artificial envi-
ronment of one’s choice is clearly appealing,and technology for realizing such a
vision would have scope for endless application.
In those days of relatively low-powered personal computers,virtual reality was
seen as being a strictly specialist field,requiring powerful graphics processors
and costly input and output devices—hence the exclusive province of the larger
and wealthier research establishments.Today,however,with the ready availabil-
ity of personal computers boasting several orders of magnitude more process-
ing power than their decade-oldpredecessors andpowerful video-game consoles
with dedicatedgraphics hardware,so-called virtual reality products abound.It is
not so much that the original ideas implied by the termvirtual reality have been
realised in a form that is now commonly available to the home user,but rather
that over-exposure of the term in the popular media has made its meaning less
distinct.‘Virtual reality’ has almost become synonymous with ‘three-dimensional
graphics’,and though once referring to an exciting formof human/computer in-
teraction (HCI),the label is nowcommonly applied to any computer systemthat
16
CHAPTER1.INTRODUCTION 17
has even the most tenuous link with three-dimensional graphics.In even more
incongruous cases,any media at all that has a ‘three-dimensional’ element to it,
including books of static random-dot stereograms and the multi-frame lenticular
animation cards distributed as free gifts in cereal boxes are marketed under the
banner of ‘VR’.
The public opinion about the current state of VR is likely to be unrealistic:on
seeing the pre-computed high-quality animations that are sometimes passed off
as being virtual reality,or the latest in arcade video games,one could be forgiven
for thinking that the problems of VR have mostly been solved and that it is just
a matter of time before increase in computer power and decrease in cost of hard-
ware make VR available ‘off the shelf’ as a consumer item.
This is,regrettably,not the case,and it is to some of the many significant prob-
lems that must be solved before virtual reality becomes a realistic mechanismfor
human/computer interaction that this thesis is addressed.
This chapter begins by posing a very fundamental question:what exactly is VR?
Having suggested a guideline for distinguishing VR from other similar disci-
plines the chapter progresses by examining the implications that this guideline
has for the field.The remainder of the thesis then builds from this foundation a
framework for designing future VR systems,and a develops and assesses a par-
ticular systemwithin this framework.
1.1 VR or not VR?
There remains some debate within the research community as to exactly what
constitutes virtual reality:the term has come to be used to describe almost any
situation where a user interacts with a computer via some representation of a
three-dimensional environment.The effects of media hype and unscrupulous
marketing have not been limited to popular impressions about VR,and to a de-
gree the termappears to have fallen out of favour in the research establishments
too (it is interesting how many of the institutes that are investigating VR shy
away fromusing it in their titles – our own ‘Advanced Interfaces Group’,or Not-
tingham University’s ‘Communication Research Group’ being examples.) With
this cautionary note however,and in spite of numerous attempts to relabel the
CHAPTER1.INTRODUCTION 18
field as ‘synthetic environments’,‘artificial realities’ an so on,‘virtual reality’ still
remains a popular label.The remainder of this chapter searches for an essential
element of ‘virtual reality’ in order to distinguish it from other similar fields of
research and to define its usage for the remainder of this work.
1.1.1 The defining characteristic of VR
Hyperbole aside,as an emerging technology virtual reality may have the po-
tential to radically alter and improve the way in which humans interact with
computers.It can not,however,be characterised by the introduction of any fun-
damentally new computer technology.The ancestors of the peripherals used in
modern VR can easily be traced back to more commonly used input and out-
put devices such as light pens,mice and displays capable of rendering three-
dimensional scenes.Many of the software techniques too are importedfromother
areas of computer science such as simulation or computer animation.It is not so
much the technology that makes VRnovel,but rather their particular application,
and the psychological and philosophical challenges and questions that the result
poses for the interaction between human and machine.
It seems appropriate therefore to search for a means of describing and analysing
VR and its supporting systems not from a technological perspective,but from
some viewpoint that takes into account these more ‘human factors’.
1.1.2 The invisible computer and ‘presence’
Virtual reality is in one sense a formof human-to-computer interface.The differ-
ence between VR and other forms of HCI can be highlighted by examining the
modes of awareness experienced by users of the interface their are interacting
with.The psychologist Polanyi [Pol64] highlights two different forms of aware-
ness that people experience when interacting with objects to perform tasks (his
work predates Virtual Reality and relates to general modes of experience,but has
the foundation of later studies of HCI [J.M92]).The first of these modes focuses
on the tool and the skill,and is typical during skill acquisition.The second fo-
cuses on the objective or purpose of the task and is more common when a skill
CHAPTER1.INTRODUCTION 19
is thoroughly learnt.Polanyi suggests that tasks performed in the latter cate-
gory of awareness suddenly shift to the former if the requirements of the task
changes unexpectedly or the tool’s properties are altered.Consider as an exam-
ple of these two forms of awareness the process of learning to ride a bicycle.At
first,the bicycle itself is the object of much concentration as the rider learns to
push the pedals,steer using its handlebars and balance at the same time.Once
the skill is acquired,however,the co-ordination necessary to ride the machine,
and the awareness of the bicycle itself becomes secondary or peripheral to such
considerations as avoiding traffic,or getting to one’s destination.Awareness of
the bicycle itself only really returns if the machine fails to function in some way,
and hinders the attainment of the goal:a punctured tyre during a ride,for exam-
ple,will quickly return one’s awareness fromthe destination to the bike itself.
Analogies can be drawn fromthis to the use of most modern computer interfaces:
learning to use the interface (for example,a ‘desktop’ environment controlled by
a mouse) involves a degree of enskilling during whichthe interface itself is the ob-
ject of awareness.Once learned however,the interface becomes simply a means
to an end,a way in which it is possible to perform the necessary manipulations
on files,say,that is the real task to hand.It is only when the metaphor of the
interface breaks down and some non-intuitive action is required by the user that
the user’s awareness is returned fromthe task to the existence of the interface.
With most traditional forms of HCI the user is aware of their position as user
external to the computer and the interface presents a metaphor allowing themto
manipulate structures that are very much ‘inside’ the computer.There are three
clear factors:
1.The user
2.The interface and associated metaphors
3.The data.
As an interface mechanism,VR is unique:the metaphor is that the user inhabits
the same space as the data or other structures with which they are interacting.
Both data and user co-exist.Whether the user is really sitting at a workstation or
is fully dressedin specialisedinput andoutput devices,the VRremoves the third-
party of ‘interface/metaphor’ and co-locate the user with their data and task.
CHAPTER1.INTRODUCTION 20
The computer,in effect,becomes ‘invisible’.The extent to which this metaphor is
convincing,i.e.the extent to which the user experiences themselves as co-existing
with the data they are manipulating,canbe seenas both a fundamental anddefin-
ing characteristic of VR and as a means of measuring the success of various VR
mechanisms and systems.
The feeling of existing in or being part of a world that is entirely ‘virtual’ is known
as ‘presence’.The psychologist J.Loomis describes the psychological mecha-
nisms by which this is made possible as being a continuous extension of a process
called ‘distal attribution’ by which humans refer sensory experiences to external
objects[J.M92]—the attribution in this case being the most extreme form possi-
ble of resolving ‘self’ as ‘virtual self’ (described more dramatically by M.Pesce
[Pes93] as ‘the final amputation’).
There are a number of significant benefits to defining VR in terms of the expe-
rience of presence,rather than in terms of any particular hardware or software.
Psychologist and computer scientist J.Struer [Ste92] suggests that
Defining virtual reality in this way will provide
(a) a concrete unit of analysis for VR,
(b) a set of dimensions over which VR can vary,and,perhaps most impor-
tantly,
(c) a means for examining VR in relation to other types of mediated experi-
ence.
This more psychological rather thantechnological approachto VRalleviates many
of the difficulties in distinguishing between what is ‘virtual reality’ and what is
not.Leaving behind the details and minutiae of a particular installation,system
or hardware it provides a broader context within which to examine the nature of
VR systems,and to decide whether something is deserving of the label ‘virtual
reality’ or whether it is something else (such as ‘computer animation’ or ‘physical
simulation’).
CHAPTER1.INTRODUCTION 21
1.1.3 The primacy of presence
Taking this experience of presence as defining the essence of virtual reality,it is
worth at this point examining briefly how existing VR technology and systems
induce this experience.
Presence and the senses
VR induces the experience of presence by presenting artificial stimuli to some or
all of the human senses,to ‘fool’ them into experiencing artificial realities.The
degree to which VR technology stimulates the senses varies considerably from
systemto system.Generally speaking there is an emphasis on the visual element
of worlds,and all VRsystems generate at least a visible representation of their en-
vironments.Although neglected in the early days of VR,the auditory element of
environments is now considered also to be important,and is being investigated
with much enthusiasm.As a consequence many systems support,in addition
to the graphical element,the generation of spatialised audio to accompany and
enhance the visible world.Although devices exist that stimulate the skin (for ex-
ample,using vibrating eccentric motors,force feedback devices or inflatable air
bladders) to simulate the tactile element in a virtual world,these are less com-
monplace and are used more in ‘high end’ research systems than in ‘consumer
VR’.Rarer still are means of stimulating smell and taste.The paper presented
by Henry et.al.[HYH
￿
92] on the ‘Nose Gesture Interface Device’ whilst being a
classic piece of spoof-literature,nevertheless highlights an area of VR peripheral
technology that has attracted relatively little research.
Immersion and non-immersion
Whether the experience of presence is achieved via head-mounted displays and
‘three dimensional’ mice (so called ‘immersive VR’) or via a window on a world
controlled by a standard mouse and displayed on a desktop workstation screen
(‘non-immersive’,or ‘desktop’),the technology is of secondary importance with
regard the degree to which presence is experienced by the user.Although an in-
crease in the functional autonomy of the user,and the potential for the transfer of
‘real-world’ psycho-motor skills between real and virtual environments suggest
CHAPTER1.INTRODUCTION 22
that immersive VR may well be more effective at enabling the user to experi-
ence presence in a virtual world,there are other important factors at play.Steuer
[Ste92] suggests that,amongst other things,
Reading a letter from a distant friend or colleague can evoke a sense of pres-
ence in the environment in which the letter was written,or can make the
distant party seemlocally present.
and
Listening to live recordings of music (recordings made during a performance)
gives the listener a sense of presence in the room (e.g.concert hall) in which
the recording was made.
Such observations have an important implication:though the technology in-
volved in generating virtual realities must clearly play a vital role in the quality
of their experience,presence is not a totally technology driven experience (it can
not be if some forms of presence can be experienced by reading a letter or listen-
ing to music).It is clear,for example,that the formand content of virtual worlds
is at least as important as the means used to display it:if the content of a world
does not conduce to the experience of presence,no amount of high-resolution
head-mounted-display equipment and high-end graphics engines will convince
the user that they are present in a virtual world.
This second observation of the importance of the content of a virtual world is
taken as a fundamental principle throughout the main body of this thesis,and is
used to guide the work developed in later chapters.
Presence and direct manipulation
A user of VR,no longer an external bystander interacting with a world via a
computer interface,becomes a participant in a situation or place and as such is
able to more directly engage in a task such innate and powerful human skills as
spatial awareness,perceptually basedinformation filtering and psycho-motor co-
ordination:this in a sense is the central VR thesis.It is expected that these skills
CHAPTER1.INTRODUCTION 23
can be far more effectively engaged than would otherwise be possible using more
traditional interface techniques.Thus ‘direct manipulation’ takes on a deeper sig-
nificance than when attributed to interaction with computers via metaphors such
as the ‘desktop’.In a sense the computer,although responsible for the behaviour
of the virtual world becomes invisible.Indeed,the interface mechanism itself
could be thought of as having disappeared.Interaction with objects in a vir-
tual world becomes unmediated by representations which are perceptually dis-
tinguishable fromthe objects they represent.
1.2 VRapplications,andthe implications of presence
Do presence and direct manipulation etc.solve all the HCI problems?Entering
for a moment the realm of fantasy,it can be conceived of a system so advanced
that it could fool the human senses completely so that they transmit information
suggesting that virtual objects are completely real in all regards.Such a system
could reproduce all the methods of HCI used in their existing forms (consider
Star Trek’s fictional ‘holodeck’—perhaps the ultimate VRinstallation—which can
reproduce completely realistic situations fromcomputer programs:it is conceiv-
able that a user could request a desk,computer,VDU and keyboard should they
wish,though there are doubtless more imaginative and effective ways of achiev-
ing their final goal).Whether such reproduction is a worthwhile thing to pursue
is one question.In this sense,VR can be considered as a ‘superset’ of other forms
of HCI (though,it must be emphasised that not only is it likely that this is en-
tirely the wrong way of achieving some goals,but it is also not realistic in terms
of today’s technology).
VR,even in its current form,however does have some unique properties stem-
ming frompresence that can be applied beneficially to existing application areas
that are more commonly dealt with using other interface mechanisms:
Enhanced interaction Because the experience of presence makes possible the di-
rect interaction of a real user with virtual objects,applications that already
contain some notion of objects (in particular those objects that already exist
in some formthree-dimensional space) can be represented particularly well
in virtual worlds,the traditional example of such an area being Computer
CHAPTER1.INTRODUCTION 24
Aided Design/Manufacture.Case studies of such applications abound in
the web and paper based advertising for existing VR systems,and plenty
of informal descriptions are available that suggest that VR is being used
successfully in this field.There are also numerous papers describing trials,
prototypes and techniques for enhancing computer aided design using VR.
Companies such as Boeing,Daimler-Benz,Audi,Volkswagen and BWMall
report on some activity in the area of VR [Con95] [Dai98],and virtual en-
vironment suitable for concurrent engineering tasks are also under devel-
opment [MFD95].Surprisingly however there is little formal documented
evidence to suggest that something that could definitely be termed as VR
rather than,say,computer simulation or animation is being used success-
fully in these fields
￿
The benefits of direct interaction in any case are clear–at
least in principle–andin situations of training this formof interaction can be
used as a means to the physical enskilling of trainees;something otherwise
only possible using the ‘real thing’ or a physical simulator;a unique benefit
of VR over other forms of interaction.Prototypes for such training already
exist in industrial and medical situations [HDL
￿
93] [SJ97] [DLBR97].
‘Sensualisation’ (being there,not just seeing it) In some applications the sense
of ‘really being there’ can add much to the experience of ‘visualising’ some-
thing (‘visualising’ in this sense must be extended to include hearing etc.)
VR has been used as a means to visualising real entities as well as abstract
data sets.For example models of buildings and other architectural mod-
els in particular are present in the field,and realistic simulation of lighting
effects coupled with the ability to navigate through a structure can give
the ‘inhabitant’ a powerful experience of ‘being’ in a place that is yet to be
constructed in the real-world [SWKvdM95] [NSBD95] [FWI
￿
95] [FTSK96].
VR is also seen as a way to visualise abstract data–the experience of di-
rect manipulation adding much more to the discipline than just a ‘third-
dimension’) [CSBD97] [CC92] [ACR96] [Mar98].
Communication and co-operative working Although numerous technologies ex-
ist for bringing together geographically distant participants (and have done
￿
In private conversations with various companies that produce VR systems,it has been sug-
gested to the author that the client companies such as the mechanical engineering companies
mentioned are currently reluctant to release information regarding their use of VR since in such
as young discipline almost any ‘innovation’ can be a great financial advantage.
CHAPTER1.INTRODUCTION 25
so for some time since at an extreme,any communication technology such
as the telephone does this),the sense of shared presence and the possibility
of more direct interaction with other participants in a virtual environment
adds much that is new to the field of Computer Supported Co-operative
Working (CSCW).Distributed Virtual Environments (DVEs) have become
the subject of much research interest both with respect to the social implica-
tions they raise [BBF
￿
94] and the more pragmatic implementational issues
that they present [BBC
￿
93b] [ZPF
￿
93].
1.2.1 All things to all people?
In spite of its potential,VR is not the HCI panacea however,and although almost
any application could be made to work in a virtual world,not everything would
benefit fromsuch a transposition.Indeed,current technological limitations mean
that some tasks would be considerably more complex than they would in the real
world.For example (and for the time being at least),limited by display resolu-
tion and input device encumbrance and the lack of tactile feedback,typing on a
‘virtual keyboard’ is unlikely to be as suitable a means of interaction with a word-
processor as simply using the real thing.Virtual reality does,nevertheless,hold
the key to revolutionising the approach to HCI for many application areas.
1.3 The current state and the future role of ‘systems’
Froma technical point of view,a large body of evidence is gathering that demon-
strates the viability of virtual worlds as a means for interacting with a variety
of different application areas.Although much remains to be done,significant
progress has already been made in a number of the component disciplines.
In particular the field of VR hardware is changing rapidly and showing signs of
a healthy improvement.The processing and graphics engines,equally applica-
ble to both ‘pure’ computer-graphics and virtual reality,are becoming increas-
ingly powerful.The home PCcan nowbe fitted with facilities to accelerate three-
dimensional graphics,and the ‘rawpolygon’ power required to display a useful
virtual world at the appropriate rate now available as a consumer item.With
CHAPTER1.INTRODUCTION 26
less than a decade of development,machines such as the 1992 Silicon Graphics
Crimson VGXT – then over £80,000 – are being outperformed by £80 PC graph-
ics cards.In terms of display hardware too,significant improvements have been
made over the past few years.Though far from presenting a perfect solution,
head-mounted-displays are nowavailable that through improved resolution and
field-of-view give a much stronger sense of immersion in a virtual environment
than those of a few years ago (which tended to present the virtual world as
though viewed through two cardboard tubes covered in smudged cling-wrap).
Peripherals for input—including those with haptic feedback—are also being de-
veloped,and though in areas such as position tracking there is still much work to
be done,large improvements in the specifications of input devices are apparent.
In the field of VR software many of the more pragmatic and immediate compo-
nents of VR software are being examined.Still,there is much work to be done
in the integration of these components to achieve an all encompassing integrated
system—the field of VR system’s architecture,as it may be called,is relatively
untrodden.Interaction techniques,for example,ranging fromthose required for
simulating the delicate surgery performed on the human eye [HDL
￿
93] to those
suitable for manipulating massive and cumbersome spools of piping on offshore
oilrigs are under development,as are mechanisms for overcoming network de-
lays to allow effective multi-participant virtual environments [ZPOM93] as well
as all manner of methods for rendering increasingly complex and visually realis-
tic environments.All these ‘low-level’ issues (details of which form the greater
part of the documentation relevant to VR) are vital to the success of virtual envi-
ronments both in specific applications and,more generally,without solutions to
all these VR is reduced to a set of contrived demonstrations and speculation.
Many of these issues are addressed to varying degrees in the VR systems avail-
able today.However,these tend to have evolved fromspecific application areas,
to have been based around specific hardware or peripherals,or to have been de-
signed with a particular type of application in mind.In most cases their heritage
is visible in their architecture.MASSIVE [GB95] and MASSIVE2 [Gre97],for ex-
ample,are systems designed primarily for large scale,multi-participant DVEs,
and while they have much support for communication metaphors,they are less
suitable for those situations requiring interaction with and accurate manipula-
tion of objects.Superscape,on the other hand,coming from a background of
smaller-scale PC based single-user virtual environments,though nowcapable of
CHAPTER1.INTRODUCTION 27
multi-participant interaction provides little of the richness of MASSIVE in this
respect.Current VR systems will be examined in detail in 2,however,a general
observation can be made beforehand:although there is no doubt that most sys-
tems could be extended to include all of the features described above,no existing
system is comprehensive in facilitating the range of attributes generally agreed
as being desirable in a VR system.
Even if this were the case,however,and the ‘low-level’ features mentioned pre-
viously came to be all supported in a single VR system,this would not of its own
accord be the ideal situation.The simple aggregation of the lower-level require-
ments of VRinto a single package does not in itself necessarily constitute an ideal
VR‘system’.It is the effect on the concrete design of a VRsystemhad by the more
exotic considerations (‘the human factor’,discussed in further detail in chapter
2 and chapter 4) rather than just the more pragmatic and technical requirements
that will be examined in this thesis.
1.4 Areas for improvement
The great diversity of architecture exhibited by existing VR systems makes it dif-
ficult to initially justify the claim that the current designs lack a particular fun-
damental property necessary for VR to become a mature technology.However,
in spite of this diversity,there are two common traits which are revealed to vary-
ing extents by most VR applications and systems which are symptomatic of a
common immaturity in current implementations.
1.4.1 ‘Richness’ of behaviour in environments
Although all systems contain mechanisms for describing the visual or spatial ele-
ments of their contents (i.e.the position and geometries of objects,surface prop-
erties,lighting,transparency etc.),there is little coherent means for attributing
dynamics or behaviour to objects or their environments.Although there is still
work to be done in the fieldof computer graphics,andthe visual presentation of a
virtual environment rarely matches our visual experience of the real world,there
is a greater gulf by far between the richness of interaction possible with objects
CHAPTER1.INTRODUCTION 28
in the real world and the limited narrowly prescribed interactions possible with
the content of virtual ones.In our real world,most objects are available for ma-
nipulation or inspection,they have some function or capability,and behave in a
manner that is consistent both with our expectations of them,and with the basic
rules of our physical universe.Current VR demonstrations,impressive though
they can be,achieve visual realismthrough techniques such as texture mapping
which give the illusion of complexity where in fact only a small number of mean-
ingful interactions are possible–and these only for a limited range of options for
which explicit provision has been made.Thus in most exiting demonstrations
the emphasis is on navigating in a textured environment with the ability to phys-
ically manipulate a pre-specified range of objects in a constrained manner.The
user is often all but an observer in a world of very limited possibility.
Although many systems nowprovide for some degree of behaviour using mech-
anisms such as VRML [BPM],Tcl [Ous94],or Java [HC97] ‘applets’ this is not a
complete solution to the problem,since it effectively shifts the issue of managing
dynamics out from the VR system and leaves it to the application programmer.
While of course the flexibility that such a mechanism gives is good,the work it
creates for the application programmer and inconsistent behaviour it allows is
not.It could be said that current systems enable rich behaviour for their virtual
environments but do not support it.This distinction is made clear by author of the
C++ language,Bjarne Stroustrup [Str86]:
A language is said to support a style of programming if it provides facilities
that makes it convenient (reasonably easy,safe and efficient) to use that style.
A language does not support a technique if it takes exceptional effort or skill
to write such programs;it merely enables the technique to be used.For
example,you can write structured programs in Fortran and object oriented
programs in C,but it is unnecessarily hard to do so because these languages
do not directly support those techniques.
1.4.2 Multiple Applications and Interworking
It is nowstandard practice in normal desktop windowing environments to have
several applications active at once and to transfer data with relative ease between
CHAPTER1.INTRODUCTION 29
them.The low level functionality necessary to support this is becoming an in-
creasingly fundamental aspect of modern operating systems and the sharing of
data between applications using such mechanisms is commonplace.VR systems
have not yet reached this level of maturity,and are much more analogous in
this particular respect to the operating systems of the early days of PCs where
only a single application was active at any one time and transfer of data between
applications was,if supported at all,done by explicit importing and exporting
mechanisms.For many of the same reasons that the desktop environment revo-
lutionised the acceptance of personal computers by the masses,a coherent mech-
anismfor handling multiple applications and the interaction of their data is nec-
essary for virtual worlds if they are to become a practical means of interacting
with applications.
Although it is possible in most existing VR systems to have more than one appli-
cation running at once,the concept is again mostly enabled,rather than supported,
and,in particular the specific implications of interaction between multiple appli-
cations in virtual worlds are yet to be addressed.In the words of Myron Krueger
‘...these are aesthetic issues with engineering consequences’ [Hei93],and have
an impact on the fundamental design of VR systems and the usability of their
environments.This will be discussed further in chapter 3.
1.5 The ‘metaphysical principle’
These ostensibly different weaknesses can be traced back to a single common
cause,which can be referred to as as the missing ‘metaphysical principle’.De-
scribed at greater length in [Pet95],the argument is this.Virtual Reality is much
more than just three-dimensional computer graphics:it is an attempt to convince
a participant that they exist in a reality generated entirely by computer.As such,a
VR systems needs to provide not only mechanisms for describing the visible and
spatial relationships between objects,but additionally must cater for the other
properties that virtual objects may posses and the relationships between these
properties (rather like the ‘laws of physics’ in our world).Because of the diverse
nature of applications,it is not possible to legislate for a fixedset of properties and
physical laws that can be applied in all cases and are suitable for all applications,
CHAPTER1.INTRODUCTION 30
therefore a mature VR system must provide some mechanism for extending,al-
tering and managing the properties of its worlds and the effects they have on
their content.This can be seen as a formof ‘metaphysical framework’.
It could be argued that the modelling of world properties,and perhaps also meta-
physics are issues that lie outside the scope of VR systemdesign,and are better
left to the individual applications.This,is in effect the situation presented to
the application designer using most existing systems.All systems provide,to
some extent,a ‘world model’ for virtual environments,i.e.they have some no-
tion of what properties objects in a virtual world have and howthey interact with
each other,the environment they are in and the systemitself.The problems arise
when this model is hard-wired into the architecture of the systemand there is no
‘metaphysical’ model to allow this to be changed (a system that has no explicit
awareness of the metaphysical principle could be said to express the metaphysi-
cal concept that ‘all forms of virtual environment have this particular set of laws
and never any others’).It is exactly this lack of awareness of ‘virtual metaphysics’
that causes the two problems described in section 1.4.
In the first case,since there is no sense within a VRsystemof general object prop-
erties beyond a fixed default set (usually those to do with the graphical repre-
sentation of an object),the relationships between the objects must be coded ‘by
hand’ in each application.This tends to mean that objects must either be pro-
grammed specifically to respond to each particular stimulus individually or will
become inert and lifeless under most circumstances.Consider in our own world
what we call the ‘laws of physics’.Even if an object is not imbued with specific
significance to us,we can say something about its behaviour in our world under
different conditions because we understand to some extent the laws of physics
(and even if our understanding of these laws is crude,and perhaps incorrect,
they nevertheless provide a framework within which we can reason about and
predict the behaviour of things that we interact with).
In the second case,virtual reality applications are unable to exchange information
mainly because there is no mediating mechanismto allowthemto do so:both ap-
plications and systemwill knowthat objects have three-dimensional representa-
tion,may be solid bodies andso on,but only the individual application will know
anything further about any of its objects’ properties (e.g.the fact that an object’s
colour is somehow related to its monetary value etc.).Since there is no common
CHAPTER1.INTRODUCTION 31
basis to relate physical descriptions of the simulated realities,the VR systemcan
not support the transfer of the object fromone application to another (other than
by trivially exporting its geometry,since it can be assumed that both systems
support ‘three dimensional representations’).Co-operating between such appli-
cations,or between users of such applications is difficult since the ‘physics’ of one
application may be very different to the ‘physics’ of the other.Without a higher
‘metaphysical’ framework,such differences can not be resolved meaningfully by
the system.
The metaphysical principle is important,since it points towards much of what
distinguishes virtual reality from just three-dimensional computer graphics.A
much more detailed discussion of the benefits of the awareness of the metaphysi-
cal principle are contained in [Pet95],and its implication for the design of virtual
reality systems will be examined later in this thesis.
1.6 Implications for a system’s architecture
What implications,if any,does the metaphysical principle have for the design of
virtual reality systems?
There can be little doubt that the concept of an overarching structure for describ-
ing the properties of virtual worlds could be added to many,if not all,of the
existing VR systems.However,such modification is unlikely to produce satisfac-
tory results:while it is relatively straightforward to modify a systemto take into
account a specific new‘low-level’ feature (such as a newinput device),it is much
more difficult to make changes that will bring it in line with a new and much
more general principle (fundamental aspects of the system’s architecture,which
to some extent can be seen as an expression of a particular ‘physical’ model,are
likely to intrude).The alternative is to design a systemaround the principle from
the outset,and it is such a design that will be proposed and developed in the
remainder of this thesis.
Many implications for the design of a general-purpose virtual reality systemfol-
low on directly from taking the metaphysical standpoint.In addition,there are
various assumptions implicit in this position that must be made clear from the
outset.
CHAPTER1.INTRODUCTION 32
1.6.1 Aims for a mature VR system
Specific requirements incumbent upon a ‘mature’ VR technology will be consid-
ered in detail in later chapters.However,several general observations are ax-
iomatic to the central propositions and motivation of this thesis.It is appropriate
therefore to define themat this stage.
Support for different ‘world models’ Taking as read the basic VR idea of inter-
action within a computer generated three-dimensional world,it is envi-
sioned that a mature virtual reality systemwill support the notions of mul-
tiple users and applications,simply because this is the way in which people
work in the real world (i.e.we use many tools,sometimes in combination,
to address numerous problems,and often do so in co-operation with other
people).Although in our real world we exist within a single,on the whole
coherent,‘world model’ explained by the laws of physics (and therefore do
not experience differences in the laws of reality),we commonly move be-
tween environments in which the social rules and conventions change rad-
ically.To this extent,our interaction with our surroundings differs depend-
ing on our location in our world.It is not so much the distinction between
social andphysical norms that is important,but rather that in virtual worlds
all laws,social and physical,may be different in different environments.In
a virtual world,it is possible to imagine situations where the distinction be-
tween ‘social norms’ and ‘physical norms’ is irrelevant,and such a notion
should be supported by the system.
Multiple users and applications Given that a number of real people will be us-
ing such as system,unless it is possible for all the users to agree to exit the
system at the same time,there are two important implications:first,the
system must be persistent to the extent that it must provide a continuity
of service to the users regardless of the presence of other users or applica-
tions;second,that in order to do this it must support the dynamic creation
and destruction of applications and their objects as well as the joining and
leaving of users.
The support for multiple users has two further implications:
CHAPTER1.INTRODUCTION 33
1.unless all users are to be located physically around the same computer,the
engine that drives the virtual world or worlds must cater for DVEs,i.e.the
remote location of participants.
2.unless all users are to be forcedto use exactly the same hardware,the system
must be designed not only independent of input devices (mice,data-gloves
and the like),but also of output devices (which in this case is taken to in-
clude the power of any graphical rendering equipment that might be used
at the users’ sites,and not only peripherals such as headmounted displays).
While there may always be scope for a single-user VR interface,the above sug-
gests also a place for a system of grand scale;large numbers of users and ap-
plications interacting with each other within what is effectively a single ‘virtual
universe’,and this thesis deals particularly with issues arising from this latter
form.Whether or not this is managed by a single system or not,there needs to
be some high-level framework that hides away fromthe participants any purely
pragmatic issues that make such a universe work.
1.7 Thesis structure
This thesis consists of eight chapters.Chapter 2 begins by looking in more depth
at the implications of ‘presence’,and develops fromthis a framework for under-
standing existing VR systems.Chapter 3 then builds upon this understanding a
set of requirements for a ‘large scale’ VR system,starting fromhigh-level princi-
ples,and following through the implications they hold for a realistic implemen-
tation using today’s hardware.Chapter 4 develops an architecture that addresses
these requirements,and Chapters 5 and 6 describe a prototype implementation
of that architecture,first at a general level,and then detailing novel algorithms
and techniques employed.Chapter 7 presents the results of the prototype,and
the thesis finishes in Chapter 8 with conclusions and directions for future work.
Chapter 2
The current state of affairs
A bewildering variety of VR systems already exists,from shrink-wrapped com-
mercial offerings to more experimental research platforms.The different architec-
tures and the flavours of virtual reality that they support are numerous and some
structure is necessary if one is to understand their function and relationships–an
understanding that is important if one is to distinguish novel approaches to fun-
damental research issues fromfeatures that are essentially cosmetic.This chapter
begins by developing such a structure,then proceeds with an analysis of the cur-
rent state of VR platforms within the developed framework.
2.1 Aframework for analysing VR platforms
There are,broadly speaking,two approaches to the construction of the kind of
framework necessary to be able to understand the relationships between differ-
ent VRsystems.One method,which could be described as being ‘top down’,is to
start from first principles and to try to extend from the theoretical into the prac-
tical a set of criteria and metrics against which existing systems can be measured
and compared.The other method,which could be described as ‘bottom up’,is
to examine a set of existing systems and to drawfromthemthe criteria and met-
rics by observation of their architecture and features.Both approaches have their
benefits.The former lends itself better to reasoning both about the nature of the
resulting structure as well as the organisation of its contents.The latter,being
34
CHAPTER2.THECURRENTSTATEOFAFFAIRS 35
generated ‘empirically’,is likely to be a better fit to the data set but has less to say
about the various metrics and categories other than that they are convenient for
the taxonomy.
The framework developed in this chapter is of the ‘top down’ variety.The mo-
tivation for this is twofold.First,this is because the main aimof this thesis is to
develop a VR systemfromfirst principles and such a framework forms the most
sensible basis for this development.Second,the few formal surveys of VR sys-
tems currently available are ‘bottomup’ taxonomies and a different approach to
their classification is more likely to yield fresh insight into interesting properties
and areas for research.
2.1.1 Existing surveys of VR systems
Rather like formal descriptions of the applications of VR,reviews of existing sys-
tems are fewandfar between.Watson [Wat94] has presenteda reviewof VRwork
in Japan (although this is more of an overviewof the different projects running in
that country rather than a technical reviewof the various systems in use),Snow-
don includes a survey of VR systems in his thesis describing the AVIARY system
[Sno96],and in The Science of Virtual Reality,Kalawsky [Kal93] describes eight of
the more mature VR systems of the time.The most rigorous and recent survey of
VRsystems andtheir technical issues is that by Snowdon et.al.[SGB
￿
96] fromthe
University of Nottingham,and although this review considers only ‘distributed
architectures’,all current VR systems of note have some notion of distribution in
their makeup and are included in this work.
The Nottingham paper is interesting not only for its coverage of numerous sys-
tems,but because unlike the other surveys mentioned (that are descriptions of
individual systems with little linking infrastructure),it contains a development
of what the authors describe as a ‘general VR reference architecture’.Their pro-
posed framework consists of six sections:distributed systems,security,user in-
terface,object support,core VRandnon core VR,each section being broken down
into subsections depending on the nature of the VR systembeing examined.The
paper concludes by presenting three categories:
￿
Components that are considered ‘essential’ to a distributed VR system.In
CHAPTER2.THECURRENTSTATEOFAFFAIRS 36
this category come collision detection,input/output handling for peripher-
als,visualisers,simple behavioural and object models,auralisers and other
audio support.
￿
‘Advanced Features’ that some systems provide,such as more complex be-
havioural models (under which title is include scripting languages),world
building facilities,support for multiple users,support for distribution and
user-demons for the explicit representation of a user within a world’.
￿
two ‘major areas in which all current systems are deficient’,i.e.security
and trading and resource discovery,both of which are elaborated upon in
the paper.
The survey presented in the following sections takes a very different approach,
and is essentially ‘top down’,as compared with the Nottinghamreviewwhich is
more ‘bottom up’.Although many of the same issues will be considered and in
some cases similar conclusions will be reached,the material is examined froman
alternative viewpoint.Taking as its basis the principle of the induction of pres-
ence as the primary distinguishing feature of virtual reality systems,this chapter
builds a framework fromthis principle downwards towards the supporting tech-
nology.
2.2 An alternative viewof existing VR systems
The approach taken in building this framework is to examine in some detail what
it means to experience presence in a virtual environment.What it means to ex-
perience ‘presence’ in a real environment has long been an issue of philosophical
debate,and whether fundamental or absolute truths about such a complex area
can be stated,a number of useful models of perception and experience can be
built that shed light upon the ‘basic requirements’ of a VR system.The following
sections are based upon one such model that has been used (with subtle varia-
tion) throughout the history of philosophy.
CHAPTER2.THECURRENTSTATEOFAFFAIRS 37
2.2.1 Physical,phenomenal and the psychology of presence
Attempting to explain the psychological processes that enable presence to be ex-
perienced,J.Loomis [J.M92] proposes a model of perception in which a ‘strict
separation between the phenomenal and physical worlds’ must be observed.There are,
he suggests,two distinct worlds,one (the ‘physical’) that objectively exists inde-
pendent of our experience of it,the other (the ‘phenomenal’) that is an internal
representation of that experience.The former,impinging on our sense organs
causes sense impressions that are in turn referred by our minds to create the lat-
ter:a notional or conceptual space that we perceive to exist beyond our physical
bodies.
Although not usually associated with the experience of presence in virtual envi-
ronments,models of reality in which the phenomenal and the physical are con-
sidered distinct abound in philosophical works [bJM34,She81,Rus48].
In such models,we are only directly aware of the mind’s constructs that formthe
phenomenal world,and have but indirect knowledge of the physical world by
inference from observation and critical reasoning.In addition to a fundamen-
tal division between physical and phenomenal,suggests Piaget [Pia54],there
is a ‘natural’ (i.e.human) tendency to divide the phenomenal into a ‘self’ that
includes the phenomenal body as well as subjective experiences such as pain
or hunger,and ‘nonself’ which is everything else phenomenal.For the greater
part the subjective division between phenomenal ‘self’ and phenomenal ‘non-
self’ parallels the more concrete division between physical body and everything
else physical.The crux of the issue with regard to the experience of presence in
virtual environments is that although it may be common to associate phenome-
nal self with physical self,there are cases such as the phantomlimb phenomenon
[vF66],brain dysfunction [Bra59],psychopathology [Lai65] and intoxication with
psychoactive drugs [Klu66] where the phenomenal self’s boundaries are altered,
and ‘self’ comes to be identified with something other than or in addition to the
physical body.The experience of presence,it could therefore be said,is another
situation where phenomenal self becomes removed from physical body.In this
case,the association being with some form of ‘virtual body’,whether explicit or
implicit in the virtual world.
Many of the technical requirements such as high graphical update rate and low
CHAPTER2.THECURRENTSTATEOFAFFAIRS 38
interaction lag that VR systems must satisfy in order to cause this association
are understood (see [Rob92],[J.M92],[MAB92] and [Kal93]).Without elaborating
on these any further however,there are a number of important factors leading
towards a general presence-driven framework for VR that can be drawn out of
this physical/phenomenal model:
￿
First,the division between the phenomenal world and the physical world
itself is significant.The question that must be asked is:‘given that even in
the real world we do not have direct perception of physical reality,what
implications does this have for modelling of virtual realities?’ Are there
observations that can be made about the physical world that have concrete
design implications?Similarly,are there pertinent observations that can be
made about the phenomenal?Furthermore,can anything be said about the
process of interpreting one as the other?
￿
Second,there is de-coupling of phenomenal from physical,i.e.although
they may commonly be interpreted as being identical they are in terms of
this model fundamentally distinct,the latter being but an interpretation of
the former.For example,the sense of ‘self’,although often related to the
physical body is not based on an immutable physical situation,but is sub-
ject to change under a variety of circumstances (the experience of a virtual
reality being posited as one of these).This suggests that the interpretation of
the physical as phenomenal is,in part at least,a subjective process (though
the consistency of experience between different observers implies that this
the process nevertheless has some objective basis).
2.2.2 Perception in the physical world
Figure 2.1 is a diagramshowing a simple mapping of a person’s experience of the
physical world according to the model suggested previously.
In the interpretation of the physical world,there can be seen three distinct stages:
first the impingement of the world upon the sense organs,second the translation
of that impingement into electro-chemical signals by the body’s sense organs,and
third the interpretation of those signals by the mind.
CHAPTER2.THECURRENTSTATEOFAFFAIRS 39
Translation into biochemical signals
Interpretation
Senses
Mind
World
Phenomenal
Objective
Reality
Impingement upon the sense organs
Figure 2.1:Perceptual pipeline of objective reality
These processes involve no conscious effort and,mind altering drugs aside,there
is little that can be done to affect our phenomenal experience once stimuli have
been received.There are some general observations that can be made about the
kinds of phenomenal world that will arise fromvarious kinds of ‘physical’ stim-
uli.At one extreme lies our experience of the everyday real world.Regardless
of our understanding of the processes that go to make up this world,its effect in
terms of our phenomenal experience is very rich,and we are able via our senses
to ‘process’ large amounts of information and to respond intelligently (when we
cross the road,for example,the amount of ‘raw data’ in terms of image,sound,
and object dynamics that we effortlessly process to avoid being hit by a fast mov-
ing car is very large indeed).At another extreme,one could imagine being pre-
sented via some ‘artificial medium’ a barrage of random stimuli at each of the
senses,e.g.arbitrary patterns of colour,sound and smell etc.which though per-
haps some representationof a data structure bear no obvious perceivable relation-
shipto one another.Although in a sense,a similar degree of stimulation is present
at the sense organs as when we cross a road,the phenomenal world experienced
by such an unfortunate subject is unlikely to be comprehensible.While it may
be possible to pick out specific sounds or patterns fromthe mˆel´ee,it would seem
unlikely that the whole experience could be formed into a coherent phenomenal
world,or that meaningful response to the environment would be possible.
Without the necessity of an in-depth investigation,these two extremes suggest
two general things about the mechanisms of human perception.First,given the
CHAPTER2.THECURRENTSTATEOFAFFAIRS 40
right kind of stimuli,our perceptual mechanisms are able to channel large quan-
tities of information to our minds which can then formulate a coherent phenom-
enal world,and our ability to interact with one another sensibly suggests that the
channel is one of both high capacity and fidelity.Second,this channelling and in-
terpretation is dependent on certain features that are true of physical reality and
that it is not a mechanismfor creating phenomenal worlds fromarbitrary stimuli.
2.2.3 Perception in the virtual world
In one sense VR can be thought of as a vehicle for exploiting this innately hu-
man,high fidelity,high bandwidth information channel as a means of present-
ing complex application data to a user in an easily interpretable format.Rather
than experiencing physical reality,the VR user’s senses are stimulated by artifi-
cial means with representations of a world the structure of which is influenced
by the semantics of some application.
Figure 2.2 shows a similar pipeline as 2.1 but for perception in VR.
The latter two stages of the process are identical to those for physical reality:once
stimuli have been presented to the senses,their interpretation by the mind and
their translation into a coherent (or otherwise) phenomenal world is automatic.
The role of a virtual reality systemcan therefore be seen as directing the process
during its first stages:translating the semantics and data of an application into a
representation of a world,and then rendering that world to appropriate output
devices.Specifically this leads to three main elements in a VR system:
Application.The applications represented in a virtual environment provide its
purpose.In effect they describe why the world behaves as it does.
World management.This provides a coherent basis for experience,and hence a
place capable of being experienced and engendering presence.It provides
the structure for the world within which the purpose of the underlying ap-
plications can be expressed.
Presentation.This is the perceptual window onto the virtual world,a trans-
lation of the world model into stimuli that can be presented to the body’s
senses.
CHAPTER2.THECURRENTSTATEOFAFFAIRS 41
These three categories arise primarily fromthe consideration of a single user in a
virtual world.The extension of the ideas to cover multiple users and multiple ap-
plications does not however affect the classification at this most principular level
(although it certainly does introduce many complications as these principles be-
come reified into the more practical concerns of software engineering,and these
will be considered later).
2.3 Elements of a VR system
These three headings of application,world model andpresentation can be further
examined to reveal many of the more pragmatic issues that must be considered
in the design of a VR system.
2.3.1 Application Interfacing
While there is great scope for experimentation with the features of a VR system
itself,for ‘real world’ VR applications there is a clear need to interface the VR
system to external applications which must then be able to affect the behaviour
of the virtual world,its contents and its inhabitants according to the applications’
semantics.Failure to provide such a mechanismmeans that any functionality that
is currently available in a non-VRapplication must be re-implemented within the
structures of a particular VR systemif it is to benefit fromthe unique affordances
of VR.Such interfacing is not a trivial task since,as well as VR systems,applica-
tions themselves come in many different architectures.
One thing at least is clear in the area of interfacing that is common to all systems,
and that is that the continuity of perceptual experience of the VRsystemmust not
be dependent on the constraints of interaction with the represented application.
Many applications such as batch processing mechanisms can not execute in ‘real-
time’;i.e.they do not respond to user interaction immediately.To maintain the
experience of presence in a coherent artificial reality,the system’s presentational
mechanisms and possibly world management elements too can not be stalled by
CHAPTER2.THECURRENTSTATEOFAFFAIRS 42
delays in the application code otherwise the virtual world will become discon-
tinuous and the experience of presence adversely affected.The system must to
some extent de-couple itself from the behaviour of the application so that it can
maintain some degree of coherence in the world and perceptually significant fac-
tors such as a constant graphical frame rate regardless of the thread of execution
of the application.For example,a single object being managed by a specific ap-
plication may not be able to respond immediately to a user request in the virtual
world because its controlling application is busy.However the world manage-
ment element of the system if de-coupled from the application may be able to
signal this to the user and allow interaction with other objects.At all times a
de-coupled presentational process should be able to allowthe user to move their
viewpoint,‘body’,‘hands’ etc.in order to maintain the minimum‘low level’ re-
quirements for the experience of presence [HMWH95,Ste92].Such de-coupling
in the system/application interface is important for single user/single applica-
tion systems (where,at worst,the user may understand and be able to accept
that the world will stall if a particular application function is selected),but is vi-
tal where multiple applications and especially multiple users are concerned:the
albeit temporary breakdown of the world for all users because a single user has
activated a particular function is totally unacceptable in a multi-user system.
2.3.2 World management
World management can be considered as the core function of a VR system to
which tasks such as graphics and device driving are literally peripheral.It is
the area of function that distinguishes this VR from others in computer science.
Other fields of have interfacing issues,and many HCI-type applications have
presentational requirements.VR however is unique in its need for mechanisms
that represent arbitrary application data in forms that can be presented to a user
or users in such a way that they will experience presence with that data.It is,in
short,the requirement of managing the underlying ‘reality’ of a virtual reality.
Existing VR systems have a variety of mechanisms for managing application in-
formation.Some,such as DIVE [ACHS94] and dVS [Gri91] use a (in both these
cases distributed) database of arbitrary tuples to represent application informa-
tion,the interface to the controlling applications being primarily in the form of
CHAPTER2.THECURRENTSTATEOFAFFAIRS 43
updates to this database (the distribution of which is mostly managed automati-
cally).Others,such as AVIARY [WHH
￿
93] and VEOS [Coc92] have an interface
in which a programming language (Cand XLISP respectively) is used along with
a library of functions to represent the objects and worlds.Numerous alterna-
tives and combinations of these approaches are sported by existing systems and
will be examined in the following survey.A specific observation can be made
here nevertheless,that the role of a VR system is not just to provide a flexi-
ble programming environment or general database distribution mechanism but
rather to provide methods of representing application data as perceivable reali-
ties.The requirement can be summarised as the provision of a world model rather
than just a programming model.There are numerous overlapping categorisations
of programming paradigms,including imperative,declarative,object-orientated,
multi-threaded and so on,and as has already been suggested many of these are
employed by existing VR systems.A world model however is a framework that
includes at its lowest level a programming model,at its highest a particular view
of the nature of virtual worlds and their contents,and linking these together clear
relationships between the high level ‘metaphysical’ concepts and the program-
ming paradigmthat is to be used to represent them.
2.3.3 Presentational mechanisms
What are here referred to as the presentational requirements of VR have been
the subject of much research (and are often considered to be the area where the
main difficulties in VR lie).This component of a VR systemhas two major tasks:
the generation of representations of the world in software,and the translation
of these representations into formats suitable for the output peripherals.At first
glance,these tasks appear to be independent of one another:for example,the
same graphical representation of a scene can be displayed by numerous different
display devices such as workstation monitors,large-screen projectors or head-
mounted displays.On further examination however,it can be seen that the ren-
dering algorithms are in some cases affected by the peripheral technology used
to present them.A stereoscopic display,for example,will require two images
from slightly different viewpoints to be rendered per frame.Similarly the algo-
rithms used to spatialise sound that is to be heard using stereo headphones are
very different to those used to generate ‘surround sound’ using multiple speaker
CHAPTER2.THECURRENTSTATEOFAFFAIRS 44
systems.Indeed both would be unnecessary if the final sound was to be played
via a single speaker.
As well as being influenced by peripheral devices,the rendering methods suit-
able for one application may differ fromthose required by another.For example,
although insome cases such as a virtual art gallery or architectural previewwhere
photo-realistic graphics may be necessary it is not clear that such mechanisms are
important or even beneficial to all applications.
Such diversity in both peripheral and application requirements means that ide-
ally a VR system’s presentational component should have access to a variety of
different mechanisms for rendering the ‘reality’ that is contained in its world
management components.Regardless of the exact mechanismused however,the
presentational process must at all times aim to support the experience of pres-
ence,and at this level this means attempting to maintain a realistic ‘update rate’,
which in the case of graphics is usually quoted as being greater than 30 frames
per second [Kal93].These lowlevel real-time issues are of the highest priority for
a VRsystemand must be serviced at the expense of other systemfunctions.Gen-
erally speaking,for complex scenes,maintaining a high update rate will require
culling mechanisms to reduce the amount of work required.Standard graphics
techniques such as back-face,frustum and occlusion culling are frequently used
to cut down on the number of polygons and objects drawn in a scene,and similar
techniques can also be applied to artefacts for the other senses.More interesting
than such ‘brute-force’ techniques however are those that take into account that
the final product is to be experienced by the human senses and ultimately by the
mind.Hence limitations in any stage of the perceptual mechanisms can be ex-
ploited to formculling methods that cut down on the processing requirements of
a VR system’s presentational processes on the basis that if the human mind does
not eventually experience a phenomenon,there is little point in wasting time ren-
dering it [GH97].
2.4 ASurvey of VR systems
As well as the framework developed in this chapter,there are a number of other
ancillary considerations that can be used to bring order to a survey of existing
CHAPTER2.THECURRENTSTATEOFAFFAIRS 45
VR systems.A rather broad yet nevertheless useful distinction can be made be-
tween that which is a complete ‘system’ for virtual environments and that which
is a toolkit or kernel.The former,for the purposes of this review,provides both
function and structure whilst being an active ‘process’ in its own right.The latter
can be considered as more akin to a library of functions with application struc-
ture being delegated to the application programmer.It is neither straightforward
nor productive in this situation to differentiate too rigidly between these,since
many ‘systems’ have ‘kernel-like’ features,and conversely some ‘toolkits’ and
‘kernels’ display some qualities of ‘systems’.However,on what is a continuous
spectrumbetween systemand toolkit,the instances reviewed here tend towards
one extreme or another and are categorised as such.
Orthogonal to this can be applied a broad measure of activity,distinguishing be-
tween ‘current’ software that is either used by third parties or is still under de-
velopment and that is either incomplete or has fallen out of current usage but
nevertheless has interesting properties.
With regard to the category of toolkits and kernels,the line is drawn at those
that explicitly address virtual reality issues,thus excluding other more general
purpose packages such as OpenGL or 3D Studio,that contribute to the field and
are commonly used to build proprietary VR applications.
2.4.1 Active Systems
The starting point for considering VR support systems is the provision of VR
environments that provide run-time support for the world model.In essence,
these systems allow a range of defined objects to exist within the virtual world
and to maintain the view of objects in this world.Each of the following systems
take slightly different approaches to realising virtual environments.Often these
different approaches are the basis for the research techniques being investigated.
CHAPTER2.THECURRENTSTATEOFAFFAIRS 46
Bamboo
Bamboo [WZ98] is a recent portable VR system from The Naval Postgraduate
School in California.Released free of charge for PCs and Silicon Graphics ma-
chines,Bamboo provides a core of object-orientatedfunctions anddefines a frame-
work for application specific ‘plugs-ins’ that extend the capabilities of the basic
system(cf.MAVERIK,in section 2.4.2).
The system as released consists of a collection of header files,dynamically link-
able libraries,Java class files and the runtime environment.In combination these
provide enough basic rendering,device and world management features to con-
struct simple virtual environments.The intention with the Bamboo system,how-
ever,is that applications extend the features of the various components to suit
their specific needs,and each of the default Bamboo classes contains numerous
‘hook points’ where callback functions may be inserted in order to modify the
standard behaviour with application-specific functions.The runtime environ-
ment contains functions that allow the inclusion and rejection of new classes at
runtime using dynamic loading.
Inline withthe system’s application-extensible philosophy,the basic worldmodel
provided by the default classes is simply a three-dimensional space in which ob-
jects may be created.Rather than prescribe specific features of that space,the sys-
temhas functions that allowapplications to register and then reference attributes
and code that describe the dynamics of the shared space.
Bamboo is built on top of Washington University’s ADAPTIVE Communication
Environment (ACE) [Sch94],which provides the system with networking,con-
currency (threading) and synchronisation functions,as well as aiding integration
with CORBA [BN95] and Java [HC97].The Bamboo rendering classes are built
on top of the scene-graph based OpenGL++ [Gra97] fromSilicon Graphics.
DIVE 3
The Swedish Institute of Computer Science’s DIVE (Distributed Interactive Vir-
tual Environment) [CH93] systemis one of the most popular and mature VR sys-
tems in use today.Available freely on the Internet for a number of platforms
CHAPTER2.THECURRENTSTATEOFAFFAIRS 47
including SGI IRIX machines and more recently PCs running WindowsNT,DIVE
has almost become a de facto standard platform for research in distributed VR
applications.
Architecturally DIVE is a loosely coupled peer-to-peer system,managing the
contents of its worlds using a shared database distributed amongst the appli-
cation processes with communication supported by standard Internet protocols.
ADIVE world consists of a number of discrete objects positioned in threedimen-
sional space,the geometry of which may be defined using files in DIVE’s own
proprietary format or VRML 1.0.Support is provided to allow these objects to
be considered as ‘solid’ (i.e.collision detection routines will be invoked on such
objects to prevent them from inter-penetrating).Applications attach behaviour
to the world’s contents either by manipulating the shared database themselves
using C functions,or by providing Tool Control Language (Tcl) [Ous94] scripts
that are triggered by events in the world.Objects in DIVE worlds have their
behaviours entirely governed by the applications that create and manage them:
DIVE itself provides only the shared space in which they exist and the facility to
communicate between applications.
DIVE can be considered as having a sparse world model that is simply a shared
three-dimensional space of geometric (and potentially solid) objects.Multiple
worlds are supported as disjoint spaces between which users and objects can be
moved by means of portals.Further richness in the world is provided by applica-
tions that govern behaviour on a per-object basis.The presentation of the world is
usually provided by a special DIVE process namedvishnu that represents a user’s
body and interfaces with a wide variety of input and output devices.
dVS
Initially built for their own PROVision [Ltd91] parallel processors and graphics
hardware,the UK based Division Ltd.’s Distributed Virtual-reality System(dVS)
[Gri91] has since been ported to run on Silicon Graphics IRIX machines.It is a
client/server based architecture with independent parallel processes (known as
‘actors’) in the server cluster providing such functions as handling input and out-
put devices,managing space and maintaining consistency of the shared database
that represents the world and its artefacts.Worlds in dVS are often constructed
CHAPTER2.THECURRENTSTATEOFAFFAIRS 48
using dVISE,a world-builder program supplied by Divison that allows simple
applications to be generated using a simple event model and requiring little pro-
gramming.
The world model in dVS consists of a three-dimensional space of geometric ob-
jects that are derived fromfour basic ‘elements’:
Cameras that represent a user’s viewpoint,
Lights that provide an illumination model for the scene,
Control points which represent object positions