A virtual and augmented reality approach to collaborative product design and demonstration

creepytreatmentAI and Robotics

Nov 14, 2013 (3 years and 9 months ago)

49 views

A virtual and augmented reality approach to
collaborative product design and demonstration

Konstantinos Smparounis
1
,
Dimitris Mavrikios
1
,
Menelaos Pappas
1
, Vagelis Xanthakis
1
,
Giovanni Paolo Viganò
2
, Katharina Pentenrieder
3

1
Laboratory of Manufacturing Sys
tems& Automation, Departm
ent of Mechanical Engineering &
Aeronautics, University of Patras,
26110 Rio
-
Patras
, Greece,


{ksmpar,

mavrik, pappas
,

xanth}@lms.mech.upatras.gr

2
Institute of Industrial Technologies and Automation


National Research Council, via

Edoardo Bassini 15, 20133 Milano, Italy, giampaolo.vigano@itia.cnr.it

3
metaio GmbH, Infanteriestraße 19, 80797 Munich, Germany, katharina.pentenrieder@metaio.com

Abstract


This paper describes the framework and the implementation plan of a web
-
based platf
orm
for
support
ing

collaborative product design evaluation, demonstration and customization.

The relevant industrial needs are first
outlined
.

A

short overview
is given on current
Research and Development (
R&D
)

activities aiming at

addressing
these needs
. The conceptual framework and the functionality of the suggested
web
-
based platform

are

then
presented
, mainly focusing on the Virtual / Augmented Reality and collaboration features. The design and
implementation of the suggested platform are discussed,

providing an insight
on the architecture and communication
aspects.
An indicative pilot case is outlined to demonstrate how the suggested approach and tools will be applied in
the collaborative review,
demonstration and customization

of products coming f
rom the textiles industry.
Finally, the
potential of the web
-
based platform in addressing the needs of collaborative product development activities is
briefly
mentioned.

Keywords



C
ollaborative design review, distributed cooperative design, web
-
based col
laboration, virtual reality, shared virtual
environments

1

Introduction


Manufacturing companies need to innovate, both by designing new products and by enhancing
th
e quality of th
e existing ones [Chryssolouris
2006]
. Usually, during product design, all th
e
persons involved share a great number of drawings
-
files and assembly models. Often, different
components or sub
-
assemblies of the product are designed by different groups of designers at
geographically different

regions
. Companies are frequently out
-
so
urcing engineering activities,
in order to accelerate the design and the product development process. Nowadays
,

50
-
80% of all
the components
produced
by original equipment manufacturers are out
-
sourced to external
suppliers
[Rezayat

2000
]
. This
action
of
ten creates problems due to the lack of distributed and
collaborative design and manufacturing systems, which would effectively disseminate product
design knowledge. The

respective

problems are typically resolved through meetings or via e
-
mails and phone
discussions. Colleagues are not capable of collaborating and exchanging their
ideas easily, if they work in different places
,

particularly different countries. In the past decade
,

many research approaches and applications
were
focused on the use of VR for

overcoming the
complexity of
product design and manufacture [
Chryssolouris
,

et al
. 2002]
. A lot of them also
included human simulations in order to perform

an

ergonomic analysis of virtual
products or
assembly processes [
Chryssolouris
,

et al
.

2004]
.

In
recent
years, research has also presented
several applications of Augmented Reality, ranging from games and education to military,
medical and industrial applications
[Azuma 2001]

[Pentenrieder
2007
]
. The potential of
Augmented Reality for product present
ation was discovered long ago. Through the
combination
of real and Virtual Reality

products can be shown in their future real environment
during the
design stage, before they are even produced. In 1999, Dempski presented the idea of context
-
sensitive e
-
Co
mmerce using Augmented Reality. She used furniture shopping as an example to
show 3D and full sized representations of virtual objec
ts in
a
physical living room [Dempski
2000]
. Around the same time, Zhang and Navab developed a direct marketing system, base
d on
AR technology. A marker plate is shown in a live video stream. The system recognizes this plate
and superimposes anima
ted 3D models of real products [Zhang 2000]
.

In additio
n
,

the other
hand, various Web
-
based manufacturing systems have
also
been deve
loped for supporting
collaborative activities, in different life
-
cycle phases of
the
product development. These include
marketing, design, process planning, production, distribution, service, etc. Distributed product
development life
-
cycle activities
,

in

a globally integrated environment
,

are associated with the
use of internet as well as web technologies. Many product development software systems, such
as Computer
-
Aided Design (CAD), Computer
-
Aided Manufacturing (CAM), database
management and intelligen
t knowledge
-
based, have also been integrated, through web
technologies, into these web
-
based collaboration sy
stems [
Yang
,

et al
. 2003]
. Some
characteristic research approaches to the field of a Web
-
based collaborative product design are:
Immersive Discuss
ion Tool (IDT)
, cPAD

[
Shyamsundar
et al
.
2002]
, Detailed

Virtual Design
System (DVDS) [
Arangarasan
,

et al
. 2000]
, Virtual
-
based Co
llaborative Environment (VRCE)
[
Kan
,

et
al
.2001]
, Web
-
enabled

Product Data Management (PDM) [
Xu
,

et al
.2003]

etc.
Moreover
,

a

few commercial tools are also available such as

the

OneSpace.net
[
Cocreate,

2008
]
,
IBM’s Product Lifecycle Management (PLM) Express Portfolio
, eDrawings Professional
[
Solidworks, 2008
]
, Matrix10
[
Matrixone,

2008
]
.

Despite the investment made in the last years,
both in research and in industrial applications, the global market still lacks in collaboration tools,
capable of providing AR and VR techniques with the possibility o
f product design evaluation.
Most collaborative tools are more related to a PLM
environment
and less to shared VEs.

Thus,
the development in the context of this work

is

focused on the conceptualization and pilot
implementation of a web
-
based platform for

product collaborative design and evaluation in full
co
-
operation with
an
interactive and immersive VE to support advanced visualization of the
product prototype, as well as to provide multi
-
user navigation and interaction capabilities.

2

Web Platform Functi
onality

The innovative concept of this

web

platform l
ies

in the integration
of

VR and AR
-
based multi
-
user
functionality
, for the development of
a
working (virtual) environment, which
will
provid
e

the collaborative users
with
realistic and simultaneous nav
igation

as well as

with
immersion and
interaction capabilities.

During a multi
-
user collaborative session, each participant will have
his
own copy of the graphical user interface, which will present a rendered 3D view of the virtual
product, through a VR
-
based or AR
-
based environment. All users will be able to interact with
the virtual product at any time, with no restriction on the number
of
simultaneous interactions.
Any
changes
made by a user
on the virtual prototype
will
be

seen immediately

by
the othe
rs.
Real
-
time chat capabilities will enable the continuous communication among the online users.
Any number of users
will be able to
join a collaborative session using TCP/IP over Local or
Wide Area Networks.

The basic functionalities that are provided th
rough the
Collaborative
Product Reviewer

(CPR)

web platform

are
summarised hereafter:

2.1

Virtual Reality Function

A tool has been developed to provide a shared collaborative Virtual Environment to aid a group
of users with different roles to evaluate and cust
omize a

digital prototype of a
product. The
GIOVE Viewer is a Virtual Reality tool, used for visualizing and reviewing the product
prototype. A set of software libraries and tools have also been developed in order to provide a
modular approach for the prog
ress of interactive Virtual Environments

[Viganò, 2007]
.

The
GIOVE Viewer is used during collaborative sessions for designing and evaluating a prototype in
a shared Virtual Environment and it is integrated into the web platform as a prototype design
tool.

It can load and display the product inside a 3D scene and allows users to interact with scene
objects or to change their appearance in the same shared environment.

To allow the integration
of the Viewer into a web platform, the 3D scene (including the pro
totype and the surrounding
environment) is loaded and displayed as part of the selected project. Project files include XML
documents, 3D models and textures. Each project can be generated by using proper tools and
having it uploaded to the web platform. Th
e web platform allows
users to select a project,
download

all

the required files and start

the Viewer in the proper mode.

The Viewer is designed
to run in three different modes: standalone, client or server. In the standalone mode the Virtual
Environment i
s available as soon as the project is automatically loaded, while in the client and
server modes a GUI (Graphical User Interface) is presented to the user to manage the
collaborative session before all the participants are connected and ready to start.
In
b
oth
,

the
client and server mode
s
, a list

of participants, including profile

details, is available together with
a chat console

(cf. Figure
1
)
.
The console is also used for broadcasting session communications
(e.g. in c
ase of connection/disconnection

of us
ers). In the client mode, the user is automatically
connected to the selected server. In the server mode, the user awaits all participants to connect
and then starts the sess
ion to let each member, as well as the server,

to
automatically load the
selected
project and start the Collaborative Virtual Environment.




Figure
1
:

T
he session management GUI

and the

Virtual Environment GUI

Once the Virtual Environment has been loaded, a context menu is available, providing function
s
such a
s session management and object

editing. Both windowed and full screen visualization are
supported. Each user can interact with the VE while other users can see changes in real time. The
user gains exclusive control on the object selected until it
is released in order to avoid conflicts
among users interactions (while an object is locked no one can select it until it is released). The
user can freely navigate so as to explore the scene or can move to a viewpoint choosing from a
set of default or use
r
-
defined viewpoints. It is possible to select objects, move and rotate them or
change their appearance (material properties and texture). Stereoscopic graphics and different
types of background are supported. The GIOVE Viewer is built on GIOVE library and

is part of
the GIOVE Toolkit.
The GIOVE library is the basic facility upon which higher level libraries
and tools are built. It has a Core module providing basic common utility functionalities for the
other modules: Graphics and the Network.
The
GIOVE Too
lkit
is

based on the GIOVE library
and provide
s

a higher level of programming interface for the tools developed on it: Viewer,
Scene Editor, GUI Editor, Application Editor.

2.2

Augmented Reality Function

Augmented Reality (AR) is an innovative technology which

enhances the real environment by
overlaying virtual information. The superimposed virtual elements are seamlessly integrated into
the user’s view of the real world to provide additional information both at the right place and
time. The combination of virt
ual information with real environments is applicable to many
scenarios.

Within the DiFac

research project

[Sacco 2007]
, the idea of Augmented Reality based
product presentation was integrated in the
Collaborative Product Reviewer

module.

Next to a
virtua
l presentation, the products can thus also be visualized and reviewed in real environments.
Two versions are available, a light
-
weight online application and a more powerful offline
application

(cf. Figure
2
)
.
They are both based on the
metaio

Uni
feye SDK®

Professional
Edition [
Metaio, 20
08]

and create mixed reality scenes by using real world digital image data
and virtual 3D models.

For the
online

version, the basic AR functionalities
have been
wrapped into a light
-
weight
ActiveX plugin. The system is ac
cessible through a web
-
site and requires only a few steps
in
order for
Augmented Reality scenes

to be created
.



Figure
2
:
Unifeye Product Presentation within the

CPR: online and offline versions

The 3D models are seamlessly

integrated in the digital view of the real world. This is realized
through the underlying
metaio
marker tracking, which detects the paper marker in the image and
uses this reference to place the virtual model data in the correct perspective. Next to the A
R
view, additional information on the product can be presented (e.g. size, availab
le colours and
textures, price etc
)
.

The

offline

tool based on the
metaio

Unifeye SDK® is a powerful interface
for creating various mixed reality applications. Next to the di
gital images, the offline version can
also use video data or real
-
time camera streams for visualization and offers a large selection of
configuration and measuring features. Very beneficial, in the context of the Collaborative
Product Reviewer, are the wor
kflow
-
oriented functionalities. The offline version provides tools
for creating and running AR
-
based workflows in an easy and intuitive way. For product
presentation, this feature can be applied to present the composition of more complex products by
visual
izing their step
-
by
-
step assembly. A concrete application scenario within the DiFac project
is the assembly of parts of a laser machine.

2.3

Collaboration Functions

The DiCoDEv

platform
[
Pappas
et al.
, 2006, 2007]

served as a basis for the
definition of the
co
llaboration framework and functionality of the DIFAC project web
-
based platform.
Major
collaboration functions include:

User/Project Management:

Through the personal home page
,

each user is given the capability
not only to process his/her personal profile

but also to see the messages sent to him/her by the
other collaborative users with reference to the projects involved. He has also the option to access
an existing project or create a new one and assign the other users that will be capable of having
acce
ss to its files and naming their rights.

Messages/Chat:

Asynchronous communication

among the users is accomplished via written
messages. They will be able to send out a message in groups to all the involved members
directly, as
well as the notification
to

others, who are just supervising the project’s progress. In
order for the electronic collaboration among the users in real time, to be facilitated and improve
d,
the platform will enable

the synchronous conferencing through the exchange of instant messa
ges
among all the on
-
line users, in different chat rooms and different modes: public/private chat. All
conversations will be recorded and saved so as for the user to revert to them whenever necessary
after the conclusion of the chat.

Project Versioning:

T
here is a mechanism for automated project file versioning. All new
versions created are stored making it easier for the users to keep track of all current
modifications, thus facilitating the design procedure, since they will have the ability to retrieve
both the old and the new versions and notice the changes.

Calendar/Scheduler:

A c
alendar onto which the programmed meetings, via
the web, will be
recorded and
to which all the users will have access to, but each user will have the right to see
only the ann
ouncements or programmed meetings of the projects in which he/she is an active
member. There
will
also be a mechanism that automatically send
s

messages to the parties
involved

so as to remind them, in advance,
of the upcoming meetings or in order to notif
y them
of an announcement related to some project.

File Sharing/Browser:

It is the platform’s user
-
friendly web
-
based interface that allows
authorized users to download and upload files. It has been designed in such a way so as to
provide rapid adoption

throughout an organization, requiring little or no training
for
familiar
isation.

3

Web Platform Implementation

3.1

Platform Architecture


The platform is designed based on an open architecture and
a
Browser/Server technology.

The
Collaborative Product Reviewer

workflow
,
is
being a broad description of the key phases of any
product design application, inside the digital factory, and
specifies the way this

module answers
to design needs

(cf. Figure 3)
.

The innovation aspect of the platform lies on the critical
co
nsideration of presence, collaboration and ergonomics, throughout its
development
.


Figure
3
:
Collaborative Product Reviewer workflow

The Collaborative Product Reviewer architecture follow
s

the three
-
tier example and include
s

thre
e
layers

(cf. Figure
4
)

namely: (a) the data layer, (b) the business layer and (c) the
presentation layer. These layers communicate through
the
internet or
an
intranet, depending on
the type of communication



Data layer
(1st tier):
includes the application’
s database and the connections with all the
other external systems
, namely

an external database for the recovery and storing of data.
Some characteristics, such as data locking, consistency and replication ensure the integrity
of data. Oracle 9i
is
used fo
r the platform’s database implementation.



Business layer
(2nd tier):

consists of the business logic. The architecture of this level can
be divided and analyzed further

into: The connection mechanism between the mainframe
PC and the application (JavaServer)
, the Java Bean Architecture, which contains the work
-
division planning algorithm and the database interactions.



Presentation layer

(3rd tier):

concerns the clients and consists of a standard Web browser
(i.e. Internet Explorer, Mozilla, etc.) as well as t
he GIOVE and Unifeye viewer that will be
integrated into the Web browser.





Figure
4
:
Platform
3
-
tier architecture and conceptual framework

3.2

Communication

The web interface provides access to the p
ortal. Through this portal,
authorized users can
upload/download the required virtual project environment and information. By the time the
project
-
related files are uploaded, a new version of the selected project
will be

automatically
created into the database. The communication betw
een the front
-
end and the GIOVE application
enables authorized users to open and modify a virtual project environment realized through the
XML
protocol

(cf. Figure
5
)
.

Communication between the DiCoDEv platform and other

external
applications, such as

data
bases
,

is also feasible.


Figure
5
:
Communication interface between DiCoDEv platform and GIOVE/Unifeye viewer

4

Pilot Application

In order
to demonstrate the

proposed framework, a pilot scenario
,

related to

collaborative review,
cu
stomization and demonstration of
carpets
,
is

used
.

The
structure of the framework
provide
s

functionality around two major activities
,
namely

the product promotion and the design
procedure. More specifically, it enable
s
:



The

interactive w
eb
-
based product
demonstration

of
existing
textile
products
, by providing
a virtual showroom for potential customers, and



The

online collaboration during design and customisation, between key actors of the
production proces
s from designers to end users.

In the case of ne
w products, the information input to the development process of a prototype
,

is
usually a market

trend analysis or a specific customer demand.

The
framework

will provide a
graphical environment that will facilitate
the
exchange of ideas among the company’
s relevant
departments and its customers,
regarding the
product
s’

properties and characteristics.

While
connected to this virtual platform, all parties will be abl
e to interact, suggest changes in the

product design and finally
,

reach a binding agreement
on
the
product customization.

In the
following schema

(cf. Figure
6
)
, is shown the concept of

th
is

framework
.



Figure
6
:
The concept of the virtual framework for textiles

The
workflow

of this scenario
is:
A
consumer

enters
a car
pet shop in order to buy new carpets
for a

room he wants to furnish
.

The

consumer is
show
n by the salesman

into the

virtual
showroom.

Initially, he selects

a photo of a room that looks
similar to his.


The virtual
showroom

system asks
him

for
the decorati
on he prefers
, the
dimensions

of his apartment
as well
as any

other information

required
.

While the
consumer

goes through

this questionnaire
,

the
system presents some new proposals of curtain
-
carpet
-
sofa
combinations and presents sets of
carpets
matching

w
ith
curtains in different colours.

After selecting one of the proposed textiles,
the photo of th
e selected room starts changing.


The selected

carpet
is

integrated
so as for
the
consumer

to be able to recognize the changes
that were
made according to his n
eeds.

5

Conclusion

The paper presented a

web
-
based collaborative platform for
product review, demonstration and
customization
. The Coll
aborative Product Reviewer

platform
will

serve as a multi
-
user real
-
time
collaboration tool with VR and AR integration an
d
can

be used as an efficient tool by designers,
engineers and managers.

It
will

enable multiple users to work in a collaborative and distributed
way, by decreasing considerably the time required for the designing phase to

be completed. This
work
has been
focused

on improving team productivity, on providing the infrastructure
necessary to make engineering teams efficient, even if they
were
dispersed over different sites,
without changing the existing design environment.

The benefits of using the proposed
co
llaborative platform include:

Quick and easy exchange of design data/information among
distributed users (user and data management capabilities)
,

m
ulti
-
user visualization and
interaction with the shared product prototype
, r
eal
-
time collaboration for on
-
lin
e decision making
on the same product design
,

u
ser friendly environment for multi
-
disciplinary assessment

of
product design outcomes
, i
mprovement on the communication among the participating working
groups throughout the life cycle of the products

and

e
ffi
cient

review and
evaluation

of alternative
virtual designs.

Acknowledgement

This study is partially supported by the IST research project “DiFac


Digital Factory for Human
-
Oriented
Production System” (FP6
-
2005
-
IST
-
5
-
035079), funded by the European Commiss
ion within the priority 2.5.9
Collaborative Working Environments (
http://www.difac.net/
).

References

Arangarasan R, Gadh R. (2000)
“G
eometric modelling and collaborative design in a multi
-
modal multi
-
sensory
virtual e
nvironment

.
Proceeding of the ASME 2000 Design Engineering Technical Conferences and
Computers and Information in Engineering Conference
, 10

13.

Azuma R, Baillot Y, Behringer R, Feiner

S, Julier S, MacIntyre B.(2001) “
Recent Advances in Augmented
Reality

,
IEEE Computer Graphics and Applications
, Vol. 21, No. 6, (Nov. 2001), pp. 34
-
47

Chryssolouris

G
.(2006)


Manufacturing Systems
-

Theory and Practice

,
2
nd

edition, Springer


Verlag, NY, 2006

Chryssolouris G, Mavrikios D, Fragos D, Karabatsou V, Pistioli
s K
, (2002)

A Novel Virtual Experimentation
Approach to Planning and Training for Manufacturing Processes
-
The Virtual Machine Shop

.
International
Journal of Computer Integrated Manufacturing
,
15(3), 2
14
-
221.

Chryssolouris

G., Mavrikios D., Fragos D., Kar
abatsou V. and Alexopoulos

K. (2004).

A hybrid approach to the
verification and analysis of assembly and maintenance processes using Virtual Reality and Digital Mannequin
technologies

. In:
Virtual Reality and Augmented Reality Applications in Manufactur
ing
, Nee A.Y.C. and
Ong S.K. (eds), Springer
-
Verlag, London.

Cocreate, WWW page. http://
www.cocreate.com
, accessed 1.3.2008

Dempski K.(2000)

“Context
-
Sensitive eCommerce”,
Proceedings of the Conference on Human Factors in
Computing Systems (CHI)
, The Hagu
e, Netherlands, (April 2000)

Kan H. Y, Duffy V. G,
Su C. J. (2001)

An internet virtual reality collaborative environment for effective product
design

.
Computers in Industry
,
45
, 197

213.

Matrixone,
WWW page.

http://
www.matrixone.com
, accessed 1.3.2008

Me
taio GmbH, “Augmented Reality
-
Software, Systems and Consulting from metaio”,

WWW

page.

http://www.metaio.com,
accessed 1.3.2008

Pappas M, Karabatsou V, Mavrikios D, Chryssolouris G.(2006)


Development of a web
-
bas
ed collaboration
platform for manufacturing product and process design evaluation using virtual reality techniques

,
International Journal of Computer Integrated Manufacturing
, Vol. 19, No. 8, (Dec. 2006), pp. 805
-
814.

Pappas M,
Karabatsou

V,
Mavrikios

D,
Smparounis

K,
Chryssolouris

G.(2007)


A Collaboration Platform for
Product Design Evaluation, Demonstration and Customization

,
Proceedings of the IFAC Workshop on
Manufacturing Modelling, Management and Control
, Budapest, Hungary, (November 2007), pp. 14
5
-
150.

Pentenrieder K, Doil F, Bade C, Meier
P.
(2007)

“Augmented Reality based factory planning
-

an application
tailored to industrial needs”,
Proceedings of the IEEE and ACM International Symposium on Mixed and
Augmented Reality
, Nara, Japan, (Nov. 200
7)

Rezayat

M. (2000) “
The enterprise
-
web portal for life cycle support

.
Computer Aided Design
,
32(2)
, 85

96.

Sacco M,
Redaelli

C,
Constantinescu

C,
Lawson

G,
D'Cruz

M,
Pappas

M.(2007)


DiFac: Digital Factory for
Human Oriented Production System

,
Proceedi
ngs of the 12th International Conference on Human
-
Computer
Interaction (HCII 2007)
, Vol.4, LNCS 4553, Beijing, P.R. China, (July 2007), pp. 1140
-
1149.

Shyamsundar N, Gadh R.

(2002) “
Collaborative virtual prototyping of product assemblies over the Internet

.
Computer Aided Desig
n
, 34, 755

768.

Solidworks,

W
WW page.
http://
www.solidworks.com
,
accessed 1.3.2008

Viganò
G, Sacco M, Greci L, Mottura S, Travaini E.

(2007)

“A Virtual and Augmented Reality tool for supporti
ng
decisions in motorbikes design: Aprilia application case”.
3
rd

International VIDA Conference
, 28
-
29 June
2007, Poznan, Poland

Xu X.
W
, Liu T. (2003) “
A web
-
enabled PDM system in a collaborative design environment

.
Robotics and
Computer
-
Integrated Manufa
cturing
,
19(4)
, 315

328.

Yang
H
, Xue D. (2003) “
Recent research on developing Web
-
based manufacturing systems: a review

.
International Journal of Product Research
,
41(15)
, 3601

3629.

Zhang X, Navab N, Liu S
-
P.
(2000)
“E
-
commerce directed marketing using au
gmented reality”,
Proceedings of the
IEEE International Conference on Multimedia and Expo (ICME 2000)
, Vol. 1, New York, USA, (July
-
August 2000), pp. 88
-
91