CWE22 Tomek-Giles-Collaboration and augmented reality

granddetourfannieInternet and Web Development

Feb 2, 2013 (4 years and 6 months ago)

119 views

Collaboration in the
Age
of Augmented Reality

Ivan Tomek, Rick Giles

Jodrey School of Computer Science, Acadia University, Wolfville, Canada

ivan.tomek@acadiau.ca
,
r
ick.giles@acadiau.ca

Abstract

Groupware and socialware
focus on

human interaction and

usually provide document sharing

and team
management
support
.
During

the last few years
, however,
people and documents have been joined by

Internet
-
connected electronic
devices
that

produce
and

consume data and allow humans to observe
and control real
-
world processes. This c
o
-
presence
of humans, data, and electronic devices on Internet
constitutes

a virtual universe
with
a new

dimension,
one

that

parallels

the physical
wo
rld

and

merges with it seamlessly.

Unfortunately
, no applications
currently support
this new reality
while
providing sophisticated

groupware and socialware

features

at the same time
.

W
e
propose a list of properties of an ideal environment supporting
human
, document, and
device interaction,
and
describe

a

design that would satisfy these requirements.

Keywords

Augmented reality, co
llaboration,
social interaction,
software framework
.

1

Introduction

The goal of groupware (s
oftware

for collaborative work)
is to
s
upport activities essential for
cooperation within work teams. It
generally focus
es

on

communication
,

shared
information access
,
and support for group management

and provides at least

textual chat

and
access

to d
ocuments
. More
recent environments also prov
ide integrated phone, audio and video conferencing, whiteboards for
real
-
time brainstorming and annotation, support for groups, user
-

or group
-
controlled
information

access,
document
versioning, and other functions.
Besides

stand
-
alone clients,
it

also
off
ers

mobile
clients, either rich
(standalone)
or thin (based on Web browsers).

Because socialware has similar
requirements as groupware, it
usually provides

similar features but
typically

in a more
limited form
.

Because socialware features
basically

form a
subset of groupware features, we will not usually refer
to
socialware
explicitly in the rest of this paper.

Although groupware has a long history, even its current implementations are not flawless.
Shortcomings
include scalability, limited functionality
,

p
ortability and

lack of adaptability

to varying
and evolving user requirements
. A
n almost universally missing feature

is lack of
integration
of
groupware with

Internet
-
based
electronic
devices
,

an important characteristic of
modern
Internet

and
the

subject
of much recent research

in
pervasive, ubiquitous
,
and embedded computing.
And while
groupware research mostly ignored
Internet
-
connected devices
,

work in augmented reality has


symmetrically
-

largely
misses

support for

collaboration,
[
Cawood
,

2008
]
,
[San
tos, 2007],
[
Hainich
,

2008]
.

The need
for integrated

sophisticated groupware and support for augmented reality is thus
unfulfilled.
The
main goal

of this paper
is to
offer a solution that is based on the concept of a flexible
groupware framework
.

Before pr
esenting our proposal, however,

Section 2
sets the stage and
describes

several situations in which such software is needed
. Section 3
lists

features that
we
consider essential for
environments of this kind. Section 4 discusses a possible approach to
a
n

aug
mented groupware
framework
based on an existing framework for conventional groupware
.
Section 5 summarizes our arguments.

2

The need for

a groupware

framework

W
e will now present several examples of collaboration and social interaction to demonstrate
the
sub
stantial overlap of functionality on one hand, and significant variety on the other. With a
flexible and extendible

software framework providing a set of fundamental functionality
, support
for such specialized needs

would
be

much easier to develop
,

even

in

new directions, such as
support for collaboration in augmented reality.

2.1.

Scenario 1: Support for a
social

community

I
-
Neighbor
s

[
Faber 2006
]
,
[
i
-
neighbors
, 2008
]
is an experimental socialware application. It
was
developed
to examine whether Web
-
based
commu
nity
software could contribute to

the variety of
community life.
I
-
Neighbor
s

offers a set of functionalities that resemble traditional groupware
and provide
s

some
additional
unusual features
, such as

community
-
based polls, reviews of local
establishments,
government links, neighborhood
-
based e
-
mail
by subscription
,
neighborhood
directories, reviews, and polls.

According to
I
-
Neighbor
s

report, the
experiment

demonstrated

that
Web
-
based collaborative
software can
have a positive effect
on

communication and co
mmunity awareness.
The
experiment

was thus successful but
its implementation
ineffective: The
software was designed from scratch,
even though
most

of the modules have already been implemented in
other products

and functionality
falls short of
typical
group
ware
standards
. Had a
groupware
framework been in place, the
conventional functionality would
have been present
and
tested
, and new functionality could be
added by plugging in new modules or by using framework runtime programmability.

Even after
completion
, the
framework
-
based
application could dynamically evolve to satisfy new needs as they
would undoubtedly emerge.

2.2.

Scenario 2:
Support for
a work

community

Imagine

a group of environmentalists including biologists, engineers, politicians,
members of
general

public,
and others.
The community includes people from several neighbouring towns, and
people who work on different schedules and cannot always physically meet at times suitable to
others. The life

of such a community includes meetings where issues are di
scussed, data presented
and evaluated, documents produced and shared, groups
dedicated to specific tasks
formed,
data
generated by electronic devices at various
physical sites accessed

and assessed
, measurements taken
,
etc.

This

community is typical of a
team

that could be well served by appropriate groupware, were it not
for the fact that
some of its

specific

collaboration

needs are not provided by any existing groupware
product, and that no groupware product on the market allows real
-
time long
-
distance a
ccess to
physical data collecting devices and actuators. A groupware framework that allows relatively easy
addition of
new

groupware features and customization of device interfaces would enormously
simplify the creation of such an environment.


2.3.

Scenario
3
:

Support for
a
gaming

community

With the spread of Internet and growing range of computer
-
based devices, gaming has acquired new
dimensions.

New game categories include
alternative reality

games

[
Kim
,

2008
]
,

ubicomp game
s

such as
Catch Bob
[
Nova
,

2007
]

or
Treasure [Chalmers
,

2005]
, and others
.

Players of
many such
games usually form teams whose members communicate with one another using mobile devices,
access shared data stored on a communal server
, draw maps, and access devices

such

as RFID
-
tagged objects.

Group size may reach

thousands
or even

millions

and the environment and goals
of
the

game may be

controlled by design teams operating in real time and
using

large amounts of
information coming from the field.
The

nature of such
events

has many features of

collaboration but
also exhibits its particular requirements.
Inheriting

the shared functionality from a well
-
tested
groupware framework, and adding special functions or tuning existing
ones

would greatly ease the
development of the software of the game

an
d improve its quality
.
Runtime programmability

would
make it possible to respond to inevitable emergent needs as soon as they arise.

These

brief
examples demonstrate the need for a

flexible extendible groupware / socialware /
augmented reality framework

is

obvious.

Before addressing possible design principles, we must

ask
what
are the requirements
that such

an
environment

should
satisfy
.

The following section
proposes

a
list of
the most
desirable features
.

3

Requirements

We argued that the best
tool for devel
oping groupware and socialware is

a
flexible
software
framework
.

What
requirements should
such an environment satisfy?

We see the following features
as most important.

3.1.

Support for standard collaboration features.

T
he framework
should include built
-
in modul
es providing
, at the minimum,

functions such as text
communication

(chat)
, audio
-

and video
-
confere
ncing, real time collaboration
whit
eboard
,
event

logging, shared access to an extendible variety of document

formats, support for group management,

access co
ntrol
, presence awareness,
application sharing,
and
support for privacy and security
,

and
administrative support,
.

3.2.

Support for software agents.

Software agents
can play numerous useful
roles
, including those performed in physical
environments by support st
aff. They can be used to provide help, implement expert
recommendation, deliver periodical as well as spurious information to groups of subscribers,
check document
usage
characteristics

and act on them (garbage collection), and other functions
requiring a
certain amount of specialized intelligence.

3.3.

Support for interaction with Internet
-
based devices.

Certain types of w
ork

teams

and social communities will increasingly require access to data
dynamically generated and consumed by Internet
-
based devices. Thes
e devices are much more
varied than static documents and cannot be expected to conform to predefined standard
s
. A flexible
general
-
purpose interface
framework
is thus required.

3.4.

Scalability

The
framework should

be suitable for groups ranging from very small
, such as a book reading club,
through

large, such as a university community of
thousands of
administrators, faculty, and students
,
to

very large, such as ubicomp game players
. To make this possible, the
framework

must provide
means
for handling,

distribut
ing and balancing large loads.

3.5.

Performance

If they can, users

avoid software that does not provide almost instantaneous response. Groupware is
used by
geographically separated users and
satisfactory performance is thus especially challenging
.
The architect
ure of its design and its implementation must therefore pay special attention to
satisfactory response.


3.6.

Persistence

Work and social interaction are ongoing processes that require continuity and maintenance of
history. Work and social encounters should not

be one
-
of
-
a
-
kind events and their traces should not
disappear when the event is over. In addition, work environments, whether ‘real’ or virtual should be
‘always on’. One should always be able to go to a library to view a document, or to one’s office to
w
ork on a project


especially because a virtual engagement may be a distant one, for example
crossing several time zones.

3.7.

Reliability

Related to persistence is reliability. Results of one’s work and social interactions should not
disappear or be corrupted
unless a major catastrophe occurs.

3.8.

Portability

Today’s users are exposed to

a large

variety of computing devices with
a

broad

ran
ge of resources.
On
the
hardware sid
e,
computing platforms

range from desktop and laptop computers
to

PDAs,
intelligent
phones

and other

devices. On the software side, there
are several popular

operating
systems, programming languages
,

communication
protocols, an
d Web browsers. The
framework
should simplify creation of
application
s

that can

run on
any
of these platforms and should

be
relatively easy to extend to interface to new platform
s
.

3.9.

Support for
a variety of user interfaces

Users
cannot
always
be guaranteed access to a portable or desktop computer. Support for mobile
devices is thus necessary.
Even
if the user has access to a

suitable device, it
may not
have
the
required

standalone client
software
installed on it. Web
-
based
(thin client)
support is thus necessary

but rich clients have their advantages too
.

3.10.

Module pluggability

As a groupware or socialware application is used, t
he need for f
unctions
not anticipated by the
designers
will arise.

Moreover,

existing functionality may not be implemented in the most efficient
or most suitable form
, and new technologies may require or enable new approaches
.
S
upported
functionality shoul
d
thus
be modularized
,

and modules should be easy to add, delete, or replaced.

3.11.

User programmability

Modularity satisfies many needs related to adaptability and innovation but

not all of them. This is
because modules are typically larger functional units th
at require significant development time that
may not always be available. Some required functions may be so simple that they don’t require new
modules, and modifying existing modules may be too laborious. Direct programmability of the
environment using wel
l
-
known programming languages and bypassing application shutdown is thus
a necessity.

3.12.

Structuring support

Different
sub
groups
working on parts of larger tasks
may be meeting at the same time in closed
meetings, different
virtual
meeting places may require
different
virtual
‘equipment’,
and
special
purpose

places


such as document libraries, help desk

office
s, private personal ‘offices’, and
geographical or other grouping
s

of data
-
generating and data
-
consuming devices may be desirable.
This requires support

for
‘spatial’ structuring that allows hierarchical organization
providing spaces
with individual characteristics and access
control
.

3.13.

Ease of learning and ease of use

T
o attract
users
, software must not only provide
required

functionality but
must
also be
easy to learn
and use. Ease of learning is improved when users are familiar with the metaphor underlying the
application's design and when
its

implementation does not violate their experiences

and
expectations
. Because all users are not the same, a usable
application should

allow customization
and adaptation to
user
-
specific needs.

After listing features

desirable in a framework for augmented reality groupware
, we will now
describe how such an environment could be constructed.

4

Possible
Framework
Design and
Implementation

We will now

outline

an approach to the design of a framework postulated above, one

that
satisfies
the
requirements enumerated in the previous section.

The design is based on our current
FVE

project
and we show
how

relatively simple modificat
ions
can

extend
its current

groupware/socialware
orientation to incorporate a
ugmented reality.

FVE (Federated Virtual Environment)
[Tomek, 2008]
is based on the metaphor of a virtual universe
that has it
s origin in MOO applications [
Haynes
,

2001
]
. When use
rs
log into

FVE they find
themselves in
a

home room in a virtual building on a virtual multi
-
building 'campus'. They can
navigate around the building

or campus
, 'carrying' objects such as documents, sharing them with
other
s
, communicate, and form groups.
C
ampus navigation is transparent,

without
the users
knowing
that different 'buildings' may
be running

on different servers

and physical machines
. Through its
persistent
federated implementation of the virtual space, FVE provides structuring of the
environme
nt, increases scalability

and reliability
, provides a
familiar

metaphor, and its principle is
easy to understand and use.

Users with

proper authorization can also program the environment using one of several well known
scripting languages including BeanShe
ll (a derivative of Java)

[
BeanShell,
2008], [
Bosanac
,

2007
]
,
Jython (Python interpreted on the Java Virtual
Machine or JVM
)

[
Bosanac
,

2007
]
,
[
The Jython
project
, 2008
]
, and JRuby (another JVM
-
based scripting language)

[
Bini
,

2007
]
,
[
Bosanac 2007
]
,
[JRuby,

2008]
.

This makes the environment rather easily adaptable to new applications and
individual users. Because FVE is implemented with IBM's Eclipse platform

[
Carlson
,

2005
]
, it
supports modularity and modules are runtime pluggable.
Due to its

use of Java, F
VE is also portable
among many ha
rdware and software platforms.
At present, FVE provides only essential
functionality and features

not implemented at present
include

support for mobile devices
, software
agents, Internet
-
based devices,

and
certain

groupware

features.


In the following, we will explain how FVE can be extended to support augmented reality. To do
this, we must first explain
how it works

and this will be achieved by tracing execution of a simple
function triggered on a client interface (Figure 1
).































Fig
ure 1. Command execution in FVE

When a user activates a

function such as a chat command, the client converts it into the form
required by the MCP client
-
server MOO Communication Protocol [
MCP
, 2008
] and sends it to the
server. The server stores the command in its Task Queue and eventually retrieves it

for execution
. It
then parses the command and determines the required
function

(‘verb’) and its arguments
,
retrieves
the verb’s declaration
,

and identifies

the software obj
ect on which the verb is defined. The server
then
determines

which
language the declaration uses

and sends it to the appropriate interpreter. In
executing the
declaration
, the interpreter uses JVM and
forwards

any results that may need to be sent
to client
s or other
FVE
components to their destinations. The server then executes the next message
from the Task Queue, etc.

The scheme just described is currently limited to conventional groupware/socialware components.
How could
it

be extended to
handle

Internet
-
based devices in the augmented reality of a new FVE
universe?
The satisfying answer is that this is relatively trivial and requires only the following two
interventions:

1.

Tracing

of the execution sequence shows that if suitable objects
with appropriate v
erbs
are
defined
on the server
to represe
nt each required type of device,

any operation
needed

to retrieve
data from sensors, process them, and use them, for example to trigger new device operations
,

can be incorporated in an FVE server without great diffi
culty.
This does not require any effort
beyond that normally necessary to create new virtual objects or new functionality; in particular,
it does not require any new FVE modules.
And because
script declaration can be
performed

at
runtime

without shutting d
own the server and interfering with the running environment, new
operations can be created
on the fly

and new devices integrated
seamlessly
into an existing
environment.

2.

The only
intervention required to make Internet devices part of FVE
is client
-
server
communication
. This is

because

clients
now
may

not be restricted to programmable devices but

include

Internet devices
, which

cannot

be expected
to conform to the MCP protocol
-

or any
other fixed protocol for that matter
.

Client
-
server communication must t
herefore use the factory
pattern
[
Gamma
,

1995
]
to
instantiate proper protocol interfaces depending on the protocol
needed interpret to
‘commands’ coming from clients

and sending commands back to them
. This
is a relatively
simple

FVE
generalization

that has
, in fact, been planned even for FVE’s
conventional groupware/socialware use to enable use of alternative protocols, for example based
on XML.


This brief discussion shows that

extension
of FVE (or any other interpretation
-
based application)
from its
conve
ntional
groupware/socialware
scope

to augmented groupware

/

socialware is relatively
simple.


5

Conclusion

One of the best known characterizations of the

evolution

of computing is in terms of three
successive
computing paradigms by
M
ark Weisner
: “First were
mainframes, each shared by lots of
people. Now we are in the personal computing era, person and machine staring uneasily at each
other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when
technology recedes into the back
ground of our lives.” The second of these

paradigms

is often
associated with virtual reality,
characterized as

‘excluding the infinite richness

of the universe’
[Weisner, 1991
]
or

‘leaving the real world behind’ [
Weisner
,
1993].

In this view, virtual reali
ty is an
intermediate
stage

preceding the more sophisticated

ubiquitous ‘calm technology’. It has, however,
been suggested that this is an
overstatement

and that the third computing paradigm should be a ‘real
virtuality’


a blending of virtual and physica
l

[Castells, 2000]
. Conventional groupware and
socialware is
one of the expressions

of the second paradigm
(virtuality), collaborative augmented
reality
represents

its
maturation

into the third computing paradigm
;

it

combines virtuality
of
conventional gro
upware and socialware

with the real virtuality

of physical devices
that can
communicate

with groupware and socialware via electronic signals.
Discussion of desirable
properties and
an examination of a possible

design of a framework that would make
developm
ent of
such environments
easier
and better
are

the
main
subject
s

of this paper.

The paper

first demonstrated

the need for collaborative augmented reality on examples from
communal interaction, team work, and gaming
.
We then proposed a list of

requirements
that such an
environment should satisfy

and argued

that
these

environments should be
built

from a software
framework. Finally, we
showed

that
design combining

concepts
such as

federated virtual space,
command interpretation, runtime programmability, softwa
re factory principles,
flexible protocols,
and modular design
is

ideally suited for
such a framework
.

References

i
-
neighbors:
http://www.i
-
neighbors.org/

BeanShell


Lightweight Scripting For
Java:
http://www.beanshell.org/
; accessed in 2008.

Bini O.:
Practical JRuby
On
Rails Web 2.0 Projects: Bringing Ruby on Rails to Java, Apress, 2007
.

Bosanac D.:
Scripting in Java: Languages, Frameworks,
and

Patterns Addison
-
Wesley P
rofessional, 2007
.

Carlson D.: Eclipse
Distilled
,
Addison
-
Wesley Professional, 2005.

Castells M.:
The Rise of the Network Society
,
Wiley
-
Blackwell, 2000.

Cawood
S.,

Fiala

M.
: Augmented Reality: A Practical Guide, Pragmatic Bookshelf, 2008.

Chalmers M., et
al:
Gaming on the Edge: Using Seams in Ubicomp Games
, PerGames 2005.

Eclipse


An Open Development Platform
: http://www.eclipse.org/

Faber, J.:
I
-
Neighbors.org: A Study
of Technology and Community

,

Annual Meeting
of the American Sociological
Association, Canada
,
2006.

Feiner, S. K.
:

Augmented Reality: A New Way of Seeing: Computer scientists are developing systems that can enhance
and enrich a user's view of the world
.

. Scientific American, April 2002
.

Gamma

E.
, et al.:
Design Patterns: Elements of Reusable Object
-
Oriented Software
, Addison
-
Wesley, 1995.

Hainich,

R.

R.
:

The End of
Hardware:

A Novel Approach to Augmented Reality
,
Booksurge
, 2006.

Haynes

C. A.
,
Holmevik

J. R.
: HighWired: On
the

Use
,
Design,
and

Theory
of
Educational
MOOs, University of
Michigan Press, 2001.

JRuby


Java
Powered
Ruby
Implementation
:
http://jruby.codehaus.org/
; accessed in 2008.

Ki
m J.Y., et al: Alternate Reality Gaming, Communications of ACM, February 2008.

MCP


MUD
Client Protocol
:
http://mcp.cubik.org/
; accessed in 2008.

Nova N.:
The Influences of Location

-

Awareness on Computer
-
Supported

Collaboration
, PhD Thesis 2007,
http://tecfa.unige.ch/~nova/img/phd_defense.pdf
; accessed in 2008.

Pedroni S., Rappin N.:

Jython Essential, O'Reilly Media, 2002.

Santos P., et al:
IMPROVE: Collaborative Design Review in Mobile Mixed Reality
, HCII, Beijing,

2007.

The Jython
Project
:
http://www.jython.org/Project/index.html
; accessed in 2008.

Tomek I., et al:
FVE


A
Virtual Environment for
Collaboration
,

AINA, Japan, 2008.

Weisner M.: The Computer
fo
r

the

21
st

Century, Scientific American, 1991.

Weisner M.: The World Is Not A Desktop, ACM Interactions, 1994.