"Perspectives and Priorities for Virtual Environments Research"

creepytreatmentAI and Robotics

Nov 14, 2013 (4 years and 8 months ago)


"Perspectives and Priorities for Virtual
Environments Research"

Professor R. A. Earnshaw

Dept of Electronic Imaging and Media Communications

University of Bradford

Bradford BD7 1DP





This paper summarises current trends, developments and issues in virtual reality and outlines areas
needing further research to make the technology more usable by hu
mans or more effective in
serving particular application domains. Recent EPSRC and European/NSF Workshops have sought
to identify the research areas that need to be addressed and these are summarised in this paper.

1. Current Trends, Developments, and I

1.1 Industrial Applications

Andreas Roessler, Virtual Reality Competence Centre, Fraunhofer, reports the following results from
industrial applications.

Current trends in real world applications are not to use head
mounted displays or gloves, but r
immersive or wall
based projection technology and simple input devices. Head
mounted displays have the
following problems


Limited resolution


Wires and cables limit human movement


Difficult to engage with other users in the real world


Ergonomic iss
ues of wearing a helmet

Gloves have the following problems


Mean time between faults is low


One size has to fit all



30 values in 30 Hz



no force feedback for the hand when “touching” something


Lack of comfort and ease of use

nt experiments with the use of a Powerwall at BMW for CAD evaluation of car shapes has produced the
following results


6D user interface is closer to reality, without forces


There is a big difference between laboratory trials and productive usage


small di
fferences in user interface make a big difference for the user


User should drive the process and iterate with the developer

Issues which need addressing


Users of Virtual Reality


Understanding the nature of VR


From technology push to application pull (a
nd better meet the user’s needs)


How to learn to use VR (rather than leaving it to the VR experts)


User acceptance that it is useful



Development of minimal user interfaces with high usability (3D mouse?)


Standard user interfaces


Complex user i
nterfaces (hierarchical are not easy to use)

User Effects


Long term usage effects


Health and safety issues


Visual systems


Haptics without mechanical linkages



High level development tools

reusable interaction toolkits


based Immersive Proje
ction Technology


Wireless interaction


Precise tracking


A recent EPSRC Workshop (21
22 February 2000) on defining the future research agenda for human factors
of virtual reality identified the following areas where work needed to be done.

More N
atural User Interfaces

Movement and Haptics

Interaction with complex materials and properties is very difficult

Precision movements versus gross movements and rotating objects are difficult

The “haptic glace” conveys a lot of information to the human user

from a brief interaction, but
context affects the sense of touch, which in turn determines the quality of the interaction

Other modalities such as vision affect haptic interaction.

3D audio and speech/natural language processing

3D sound representation h
as applications in submarines, alarms, architectural

Matching speech to the visual appearance of an avatar and embodying emotion

Synchronising the visual with speech when the user gives priority to the visual

High fidelity (large vocabulary) and low fideli
ty in speech

allow for accents and multiple
human speakers

Visual representation

Very high quality rendering gives a good appearance but does not help the user maintain a good
judgment of space. This seems to depend on very small cues

Stereo vision

ize, shape etc



What are the ways the user can interact with space to improve perception?

It would be useful to integrate the above modalities, but there is no standard software.

Evaluation Metrics

We want to do things in Virtual Environme
nts that we can’t do in real life

Trying to measure presence in VE’s makes us realise we don’t really know what it means in the real


We need a taxonomy, guidelines, corpus, and framework

Dependence on the application domain, for example


high fidelity
/high precision


general visualisation (e.g. games) where appearance is important,


exploratory versus interacting with objects

Software Tools

Existing tools are inflexible, closed and labour intensive

The gap between those who want to study applications an
d the starting point to get there

there is a
high application development overhead

Flexible, open, interoperable, generic tools and interfaces are needed

Design in the light of User Psychology

What is the relationship between technological, physiologica
l, and cognitive elements?

What are the factors that account for individual differences?

When is immersion useful?

There are low level interaction issues and high level learning issues

Large Scale and Small Scale Design

System Integration

time modif

Database integration

Supporting the whole design process

Intelligent Level
Detail control

Status control

Common components/file formats

Human Interaction

Human in the loop

Manipulation from within

user manipulation

sire cooperation

Easy exploration and manipulation


Simulation of physical forms


Design approval


Real World Integration

An example application domain is high precision surgery

Human Factors Needs

Need for outcome measures

developed from re
al tasks with real users

Need to validate training systems with skilled end users

Need to identify where the added value occurs

Perceptual Fidelity Issues


high frequency, stiffness, temperature

which is most relevent?


individual diffe
rences in cue use (shadows, texture) and response to cue conflicts

The visual/haptic interaction

Larger Scale Simulation

There are a number of perceptual issues and questions


off between visual information and between vision and non
visual informa

Difference between effects on appearance and behaviour

What perceptual information is essential to support and effective VR environment?

Abstract Visualization

Flexible representations

Mapping from abstract to spatial (what works for what task?)

oring (navigation)


where am I (and where have I come from?)


2. Generating a Research Agenda

A recent joint European Commission/National Science Foundation Workshop was organised to bring
together leading researchers from Europe, USA

and Japan. The overall theme was

and Human
Computer Interaction”.

This joint Workshop was set up under the auspices of the Joint European Commission/National Science
Foundation Strategy Group that had its first meeting in Budapest,

4 September 1998. The meeting derived
from a joint collaboration agreement between the EC and NSF in August 1998, signed by Dr George
Metakides (Director, Information Technologies, EC) and Prof. Juris Hartmanis (Director, CISE, NSF). The

aims to facilitate the joint development of knowledge and applications in key emerging science
and technology areas of mutual interest. Successful cooperation holds the promise of more cost
investment of research funds in the USA and the Europe
an Union.

National initiatives in the USA and many European countries are recognizing the benefits to scientific
research in supporting larger groupings, often with interdisciplinary teams of researchers. It is possible to
achieve results with a national

grouping that it is not possible to achieve on the same time scale with an
institutional one. This model has also been used for a number of years by the European Commission to
facilitate research development in European countries, and accomplish faster t
echnology transfer to
European industry by company participation in projects. These initiatives have recently been extended to
include non
European partners on a self
funded basis. Collaborative links have also been established with
Japan. It is clear t
hat with the increasing globalization of research and development there is a need for
companies to develop products that are viable in world markets. Thus what is being proposed by the EC and
NSF is a logical extension of existing paradigms for securing s
ignificant progress in key research areas.

It was felt desirable to arrange a series of research Workshops to enable early identification of key research
challenges and opportunities in information technology. It was intended that each Workshop should br
together eminent scientists and technologists in the US and Europe in the area being addressed, and that the
themes would emanate from the research community.

At the meeting of the Joint EC/NSF Strategy Group on 3
4 September, a number of possible the
mes were
identified. These included "human
centered computing and virtual environments", "large
scale scientific
databases", and "intelligent implants". Scientists on this Strategy Group included: Prof. Andy van Dam
(Brown University, USA), Prof. Paul Me
ssina (California Institute of Technology, USA), Prof. Rae
Earnshaw (University of Bradford, UK), Prof. Giorgio Baccarani (University of Bologna, Italy), Prof. Rolf
Eckmiller (University of Bonn, German) and Prof. Gilles Kahn (Inria, France).

It was agree
d that the first joint research Workshop should concentrate on the themes of human
computing and virtual environments. Human
centered computing is perceived as an area of strategic
importance because of the move towards greater decentralization a
nd decomposition in the location and
provision of computation. The area of virtual environments is an area where increased collaboration should
facilitate more rapid progress in solving some of the more intractable problems in building effective
ons. It is intended that further Workshops should follow this one, either on separate topics or on
specific issues arising out of this first Workshop.


The objective of the Workshop was to concentrate on the research frontiers of human computer interaction

and virtual environments. Of particular relevance are the desires that interaction be more centered around
human needs and capabilities, and that the human environment be considered in virtual environments and in
other contextual information processing a
ctivities. The overall goal is to make users more effective in their
information or communication tasks by reducing learning times, speeding performance, lowering error rates,
facilitating retention and increasing subjective satisfaction. Improved design
s can dramatically increase
effectiveness for users who range from novices to experts and who are in diverse cultures with varying
educational backgrounds. Their lives could be made more satisfying, their work safer, their learning easier
and their health

better. Research areas to be addressed included:


high level content descriptions and their access, such as metadata and MPEG7


reducing cognitive load and providing more scope for creativity


disciplinary interaction and how to make it work



interaction in specific social contexts and with cultural differences


dealing with universality and the problems of the differently


interaction styles and their implications


consistency of cognition models across information appliances


paradigms for

emerging new kinds of interaction; beyond WIMP interfaces: multimodal and perceptual
user interfaces


challenges for virtual environment technology and interfaces


usability issues and measuring the effectiveness of symbiosis


design and evaluation of online

communities for intranet and internet


scaling online communities to support millions of people


universal access, social and ethical issues

Detailed results are at

3. Diversity of Technology and Users

The current diversity of the field, such as in displays, is both a challenge and an opportunity. Current
developments in technology and content generation, and the rapid rise of new uses and applications, req
diverse kinds of interdisciplinary expertise in order to exploit the technology effectively. There is a diversity
of technology (hardware, software, and networking), a diversity of users (especially in areas where
technology has not yet made signific
ant inroads) and an increasing gap between what users know and what
they need to know to use current systems effectively. All this bears testimony to tools and systems being
driven rather than user
driven. Much more attention needs to be given

to end
end design and
integrating the needs of the user from the very beginning. Critical parameters in the design and evaluation
process need to be much more firmly identified, quantified and rigorously upheld. Research is needed in this


Research Integration

The breadth of the field is not being taken into account by current research. The experts in converging areas
are not working together, and research programs are not getting the right kind of interdisciplinary expertise,
or support,
to give added
value integration. Indeed, the need for a greater degree of integration and greater
attention to scalability pervade many of the research issues.

5. Multiple Disciplines

One important area of future work is the behavior of individuals and
communities in their relationship to
each other and to the world. There is a long history of educational, psychological, social psychological and
sociological studies, but methodological innovations are needed to capture and understand the complex
of individual and group behaviors that occur while using technology. Analytic and descriptive studies
can provide useful insights, but there is a strong need for more prescriptive outcomes that can guide
designers of new technologies. Guidelines are avail
able for basic user interface design, and these need to be
extended to accommodate new technologies. In addition, validated metrics, user surveys, task taxonomies,
ethnographic methods of observation, participatory design methods, usability testing strate
gies, expert review


techniques and software development methodologies would all help produce more orderly development
processes for new technologies. Social impact statements prepared in advance of implementations could
facilitate broad discussions of cri
tical technologies and thereby minimize the number and severity of
unanticipated side effects.

Understanding community relationships becomes even more critical when it is a community of users
interacting in a shared world or information space, such as on
the World Wide Web. The interface needs to
be appropriate to the task to be performed, the social behavior of the user (or groups of users), and the
maintenance of relationships. Research programs should be developed in this area.

Strong encouragement s
hould be given for universities to support multidisciplinary activities and to reform
traditional computer science departments so that they include a human
centered approach throughout their
research and educational programs. A specific suggestion for mov
ing the center of gravity in this direction
would be to fund graduate fellowships in human
centered systems.

Much stronger promotion of evaluation and empirical testing of systems in the context of work is needed.
This evaluation and testing should be bo
th controlled and ethnographic, in the laboratory and in the field.
Needs, requirements and behavior of the users, as well as the range of problems they need to solve, should be
considered. Progress in VR, online communities, universal usability and othe
r user
centered areas will be
dramatically increased if the funders insist on some form of assessment.

6 Key Application Drivers

Problems to be solved can be key drivers in the domain. These represent in some sense "pull" requirements
from the user tha
t need to be considered alongside the more normal "push" technology from the vendors.
One possible agenda for collaboration between industry, academia, and government, is to use the "Content"
age as the key driver for 2010. Content is needed for human me
dia technology, augmented reality, digital
story telling, interactive broadcasting and multimedia workspaces. Indeed, the whole nature of the human
computer interface may move away from one operating on a model of sequential task definition and
. The new human
computer interface could operate on a model of behavior, context, cultural
background, information awareness and imagination, namely "story
telling" at the interface, thus drawing on
its own values of context and history.

A working model
and methodology is needed for the Content Age. The technology should be user
and mobile with new types of interaction technology and information display. Three key application domains
have requirements for this technology:

• Health and continuin
g medical education (both doctors and patients)

• Environment

• Cultural heritage

These involve digital content, but more importantly the user perceptualisation of this content and interaction
with it.

7. A Taxonomy of Human
Centered Systems

An initial
taxonomy for human
centered computing in the context of virtual environments is proposed. This
provides a framework for an understanding of multi
channel input and output, the skills of the user, the
particular technology selected for a task and the task
to be performed (whether simple or complex). A
foreground and background task model is proposed, and it highlights two key issues for the future:

• How to get foreground and background to assist each other

• How to increase the effective contribution of
the background (i.e. to make the computer more aware of the
user's context, needs, and requirements at any point in time).



This is a short list of useful references and background reading. It is not intended to be an exhaustive

An NSF Workshop produced a special issue of Behaviour & Information Technology Vol 12, No. 2, March
April 1993.

A useful Canadian policy document that thoughtfully deals with Universal Access


A SIGGRAPH agenda document that deals with graphics and user interfaces



"Virtual Reality: Scientific and Technological Challenges",
National Research Council Report, 1995.

An NSF Workshop on
"Human Centred Systems"

February 17
19, 1997

This has many useful position papers and recommendations from four working groups:


"More than Screen Dee

Toward Every
Citizen Interfaces to the Nation's Infrastructure",

Research Council Report, 1997.


"Modeling and Simulat
ion: Linking Entertainment and Defense"

National Research Council Report, 1997


"The Unpredictable Certainty: White Papers",

"Internetwork Infrastr
ucture Requirements for Virtual Environments",

pp 110
122, National Academy
Press, 1998


The following document is a recent publication outli
ning a basic taxonomy of approaches to VR with special
consideration to their affordances to support collaborative work, both same
place and remote.


The following two pointers are to articles describing research from Bill Buxton's group dealing with
technology mediated collaborative work at a distance, reactive environments and ubi



Electronic Visualization Laboratory:


CAVE Research Network Users Society


"Funding a Revolution: Government Support for Computing Research"

Chapter 10,
"Virtual Reality comes of Age",

pp 226
249, National Academy Press, 1999.

A recent and influential US Presidential Information Technology Advisory Council Report, PITAC report,


President Gore Announces C
linton Administration's $366 Million IT



"What Exactly is the Grid?"


ACM CHI99, Workshop 8,
"Development of an HCI Research Agen



"Networked Virtual Environments

n and Implementation",

Sandeep Singhal and Michael Zyda, ACM Press, July 1999

Contents at

Author Biography

Rae Earnshaw is Professo
r of Electronic Imaging and Media Communications and Head of the Electronic
Imaging and Media Communications Dept at the University of Bradford, UK. He is also Head of the School
of Informatics. He has been a Visiting Professor at Illinois Institute of T
echnology, Chicago, USA,
Northwestern Polytechnical University, China, George Washington University, Washington DC, USA, and
the Swiss Federal Institute of Technology. He was a Director of the NATO Advanced Study Institutes on
computer graphics and CAD in

the UK in 1985 and in Italy in 1987. He is a member of ACM, IEEE, CGS,
EG, and a Fellow of the British Computer Society. He is an Expert Adviser to the European Commission, a
Reviewer for Framework 5 proposals, and an Evaluator of ESPRIT LTR and ACTS pr

Prof Earnshaw has authored and edited 28 books on computer graphics, visualization, multimedia, design,
and virtual reality, and published 128 papers in these areas. He is a member of the Editorial Board of The
Visual Computer, a member of the Ed
itorial Board of IEEE Computer Graphics and Applications, Editor In
Chief of Virtual Reality: Research, Development and Applications, Vice
President of the Computer Graphics
Society, and Chair of the British Computer Society Computer Graphics and Displays
Group. He is also a
Director of Advance Visual Communications plc.

20 March 2000