IRG on Interface Technologies

estonianmelonAI and Robotics

Oct 24, 2013 (3 years and 9 months ago)

61 views

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






1

IRG on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments


1.

Summary

This report builds on the experience of the different key actions in FP5 which have been involved in the
technology and application areas related to
user interfaces. Research addressing Interfaces is based on
advances in simulation and visualisation technologies, speech and language technologies, multi
-
modal
interaction, computer graphics, virtual and augmented reality, body area networks, ubiquitous
c
ommunication etc., and is particularly relevant in key mainstream application areas, such as access to
networked audio
-
visual information, culture, e
-
learning, entertainment, mobile communications, e
-
work, design and engineering, and for people with spec
ial needs.

There is a wide agreement that future systems will have to take user behaviour into account, based on
awareness of how people communicate and interact, what they want to do and achieve, in order to
design truly usable systems that provide easy,

efficient and effortless interaction and use. Devices,
systems and applications must operate in harmony, seamless to but taking full advantage of available
technologies. To achieve natural interaction with networked sensors embedded in our environment,
“t
echnology” needs to understand and interpret our senses in an integral way.

This report also summarises the significant work in this direction already done in FP5, although with a
high degree of fragmentation between different action lines and application
areas. A common effort is
required now in order to reach the necessary critical mass to involve all interested actors and build on
the Europe’s strengths in mobile communications, multilingual and multi
-
modal interaction.

Three main areas of possible resea
rch are defined:
understanding the user

(in particular his

expectations
and behaviour in relation to the mobile applications and services which will be made possible by the
introduction of new interaction and communication technologies)
,
enabling technolog
ies

(such as
speech recognition in adverse conditions, virtual reality, wearable devices, which will make it possible
to create new services), and
new interaction systems and paradigms

(such as the use of avatars,
conversational interfaces, all
-
senses comm
unication).


2.

Scope


From the SP text:

Intelligent interfaces and surfaces: The objective is to provide more effective ways of accessing
ubiquitous information and easier and natural interaction modes with intelligence that surrounds us.

Research will foc
us on interfaces and interactive surfaces that are natural, adaptive and multi
-
sensorial,
for an ambient landscape that is aware of our presence, personality and needs, and which is capable of
responding intelligently to speech, or gesture or other senses.

The aim is to hide the complexity of
technology by supporting a seamless human interaction between humans, between humans and with
devices, virtual and physical objects and the knowledge embedded in everyday environments. This
includes research on virtual

and augmented reality

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






2

Work will also address technologies for multilingual and multicultural access and communication that
support timely and cost effective provisions of interactive information
-
rich services meeting the
personal, professional and busine
ss requirements of all members of linguistically and culturally diverse
communities.

The vision formulated by ISTAG for FP5,

"start creating an ambient landscape (for seamless delivery of services and applications) in Europe,
…develop user friendliness…",

has been further expanded and exemplified in the report "ISTAG scenarios for ambient intelligence in
2010" which, among other things, stressed that a central challenge of Ambient Intelligence (AmI) is to
create systems that are intuitive in use
.

Such syst
ems will have to take user behaviour into account and adapt to it over time, based on
awareness of how people communicate and interact, what they want to do and achieve, in order to
design truly usable systems that provide easy, efficient and effortless in
teraction and use. We would
like to see natural interfaces that are, for instance, based on natural language speech recognition,
gesture interpretation and even brain wave control. The systems should be able to, if not read our
minds, know what we want. In
formation browsing and retrieval should be intuitive and effortless. The
user experience should be built upon more senses than sight and hearing, and should be constantly
tailored and personalised to the user’s preferences, habits and situation. All wirele
ss services of the
future should be provided with the highest degree of integrity and security.

The solutions must offer holistic, converged communication appliances for fixed as well as mobile
information and communication. Devices, systems and applicatio
ns must operate in harmony, seamless
to but taking full advantage of available technologies.

Several different concepts are included in this vision, all converging to the goal of achieving natural and
intuitive interfaces. Human
-
centred interfaces enable p
eople to use knowledge based services and
appliances in an easy and natural way, hiding the complexity of the underpinning technology.
Multilingual interfaces cope with the need to share information in a multicultural and multilingual
environment. Intellig
ent agents are used as intermediaries in communication, seeking and retrieving
relevant information. Virtual environments provide an intuitive interaction paradigm with a complex
network of information provided by web servers and distant users. Natural int
erfaces therefore are an
important part of a chain that enables us to communicate with each other in our language of choice,
access information and aquire knowledge. Interfaces are also a key component in building an all
inclusive knowledge society.

Resea
rch addressing Interfaces is based on advances in simulation and visualisation technologies,
speech and language technologies, computer graphics, virtual and augmented reality, ubiquitous
communication, multi
-
modal interfaces, body area networks etc. and i
s particularly relevant for the
following key application areas:



Mobile communications (3G/UMTS and further evolution, ambient intelligence)



Access to web
-
based and broadcasted audio
-
visual information, culture, e
-
learning, entertainment
(e.g. mixed reali
ty, immersive TV, networked and mobile games)



E
-
work (cooperative and remote working techniques),, engineering (CAD, simulations, etc.)



People with special needs (intelligent assistive devices and systems for persons with disabilities and
elderly people)


Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






3

3.

Rationale for EU action


Whereas developments in technologies for access to information, mobile communications and even
entertainment have insofar based mainly on a “technical” vision (debating, say, new data classification
techniques, network concepts and

protocols, radio interfaces, virtual reality techniques), a common
effort in a much wider context is now needed, in order to involve all these areas to create the necessary
critical mass at European level. Such a common effort will furthermore give import
ant support to the
e
Europe objectives of creating a socially inclusive and digitally literate Europe by providing natural and
easy to use ways of interacting with information and services.

Such a common effort will be based on an user centred approach, loo
king at the new ways in which
users will interact with the systems. Indeed it is expected that the major innovative thrust in all the
different application areas will come from new ways of interaction with the system or among systems.
The new development s
hall be truly user centred, and shall provide the user with new sensations and
experiences opening up new possibilities. Given the increasing bandwidth of the new Broadband
networks, applications will gradually start to exploit new ways of interaction. Soo
n the bandwidth
bottleneck will not be in the transport of data but rather be determined by the “Bandwidth to the Brain”,
requiring innovative solutions for interfacing and communicating.

In this context, wearable terminals and deviceless interactions will

become commonplace. The
communication space of the user will be populated by avatars, and the reality he sees will be augmented
by useful information.

To achieve natural interaction with networked sensors embedded in our environment, (e.g. in home
appli
ances, in car devices, around us in the office and in wearable devices, etc.) “technology” needs to
understand and interpret our senses in an integral way. Until now focus has been purely on independent
senses rather than on combined senses that optimises
natural interaction with technology. Europe needs
to mobilise its strong research community to intensify research in this domain. In addition Europe is in
a prime position to improve multilingual services such as accurate context dependent automatic
transl
ation systems. Associated work includes interactions within virtual environments.

3.1.

Priority Application Areas

This research effort, in order to reach the necessary critical mass, needs to be carried out at European
level, involving all the application area

mentioned above which are potentially concerned by such
developments.

Let us consider specifically each targeted application area:



Mobile communication

is seeing the advent of the third generation of mobile cellular systems,
UMTS. With UMTS it will be pos
sible to use high bit
-
rate data communication on the move and
ubiquitously. It will enable mobile Internet access with its information and e
-
commerce transaction
services. It is expected that mobile access to the Internet will outnumber fixed access withi
n two to
five years. However, the transition from a voice
-
oriented system to a data
-
oriented system is not
going as smoothly as anticipated by the industry. In Europe the market for GSM type
communication is not growing as rapidly as in the last few years,

and new applications that could
attract significantly more traffic, and motivate a rapid take
-
up of UMTS, are hard to find. Basing
the further developments on the real user needs and exceeding his expectations through innovative
interfaces (augmented rea
lity, wearable communication devices, avatars) will be key to attain a
successful deployment of the new systems. In
-
vehicle applications (e.g. the car environment) is an
area where the introduction of natural and easy
-
to
-
use interfaces will have positive e
ffects, e.g. on
road safety, especially in the light of the ever increasing number of features and services made
available to the driver.

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






4



The
audio
-
visual

(a/v) service industry has already a long standing experience with user interfaces
such as the remote

control of the television and video recorder. The programming of the latter
device has been a notorious difficult problem for which no generally accepted solution exists. The
convergence of technology has led to increased functionality of the a/v devices
and until now it has
increased the burden of the use and maintenance of these devices for the end
-
user. Interface
technology will be relevant for all applications of a/v services, such as entertainment, e
-
Learning
and e
-
Culture.
Mixed Reality

(MR) systems
still have a number of unresolved key technical issues
such as the integration of mobility, real
-
virtual visualisation aspects and communication, the
development of light and non
-
intrusive sensors and displays. Once these problems are overcome, a
technolog
y such as MR is likely to lead to the creation of completely new markets. In this context,
it may lead to the "revaluation" of manual work. In the media area, MR will be very appealing to
service providers in need of differentiation and to operators in nee
d of bandwidth
-
hungry
applications. MR is also likely to be an integral part of the next generations of interactive TV, a
mixture of broadcasting and Internet services.



Several areas of
e
-
learning

can only really take off if appropriate interfaces are avai
lable. This is
particularly the case for realising the “
ambient learning
” concept, for the provision of ubiquitous
access to learning, an instantiation of the “ambient intelligence” paradigm, which requires natural
access to learning facilities and should
go seamlessly by combining wireless access and broadband
communications to access the required resources. It also applies to other concepts such as:
advanced distance learning
, implementing the concept of virtual classroom, where teachers and
learners com
municate through collaborative learning (which requires advanced visualisation
facilities enabling virtual presence and natural ways for application sharing facilities) and
training
areas

requiring the use of virtual and mixed reality simulations and of au
gmented reality (e.g.
training of surgeons, training of maintenance staff of complex equipment, training of mechatronics
engineers…).



The development of appropriate interfaces is also crucial for supporting
e
-
work

and for the
realisation of the workplace o
f the future. In particular there are a number of changes in work
organisation, workplace tools and workplace design that have to be considered for a number of
priorities: improved
health and safety

in work (notably to reduce levels of stress and increase
creativity in a supportive work environment);
competitiveness and efficiency

of businesses and
public administrations;
city and rural development

(enabling “sustainable city” development with a
balance between physical and “network” mobility). Research is
also required to improve the
technology and to incorporate augmented reality in applications such as surgery, industrial design
and visualisation of computation results in all science and engineering field.



Innovative interfaces will be key to making techn
ologies barrier
-
free, so as to promoting the social
integration of
persons with disabilities and elderly people
. The focus is on mainstream
Information Society products and systems, including public services, which must be designed in
such a way that they
can meet the requirements of the widest possible range of users and use
situations. One particular area of work should be highlighted:
the convergence between
telecommunications (including mobile), information technologies and more conventional electronic
products such as radio and TV. Developments in this area will continue to have a major impact on
European society including disabled and older people. Research into the impact of technology
convergence on marginalised groups or groups in danger of margina
lisation should be addressed
from the barrier
-
free perspective. Focus is also in empowering technologies and systems specially
designed to ensure access for users with as

example physical, sensory or cognitive impairments.
Priority is to be given to techn
ologies promoting personal autonomy and enabling people with
special needs to live independently, as well as systems for professional and informal carers.



Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






5


3.2.

Europe’s strengths and weaknesses in the domain of user interfaces: a SWOT analysis


Strengths

-

Te
chnological excellence in multi
-
modal
interaction, multi
-
lingual dialogues, speech and
gesture interpretation, avatar control and other
critical component technologies (e.g. computer
graphics, MPEG)

-

Strong VR simulation expertise

-

Strong telecom, mobile ter
minal and content
industry (and demand)

-

Strong expertise in eAccessibility and solutions
empowering users with special needs


Weaknesses

-

Difficulty in understanding real user needs and
identifying potentially successful services

-

Fragmented research scene a
nd dispersed
European effort

-

Linguistically and culturally fragmented market

-

Weak computing and mass market software
industries, absence of large industrial players

-

No military budget is devoted to visualization
interfaces as in the US

Opportunities

-

Wide
spread user adoption of new services thanks
to their accessibility through natural interfaces, and
consequent reduction of costs

-

Turn the multilingual weakness (fragmentation)
into a strength (strong multicultural experience in a
globally even more linguis
tically and culturally
fragmented market)

-

Reaffirm a possible world leadership of the
telecom, mobile terminal and content industry

-

Improving traffic safety by use of low cognitive
load interfaces in the car

-

Strong potential in some application fields such

as:

-

distance learning (EU's political commitment
to pedagogical innovation through the e
-
Learning action plan, long history of
cooperation among Universities, strong
knowledge on cognitive issues and
multidisciplinary research, possibility to
implement ne
w pedagogical paradigms, e.g.
distributed collaborative learning)

-

European strength in research on e
-
work and
in related disciplines (e.g. psychology and
design)

-

Very high potential growth in some
application areas (e.g. mService/mCommerce)

-

Introduction of

interactive digital TV services,
UMTS, networked appliances


Threats

-

Delays in take
-
up of new devices and services
(see for instance WAP flop due to a deficient
infrastructure and lacking services, GPRS’s slow
瑡te
-
up due to a lack of services, and 3G’s
d敬ey
du攠 瑯 r敬u捴慮捥 in inves瑩ng on infras瑲u捴ur攠
慮d s敲v楣is d敶敬epmen瑳)

-

To be overtaken by competing economic regions
(as it is already happening for the deployment of
3G networks)

-

Complex application development required

-

Technology not reliable
enough to be accepted by
the market

-

Too long term return on investment, unclear
business models

-

Domination by non
-
European market players
(e.g.
in visualisation software)

-

Resistance to change



Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






6


4.

State of play


The IPPA4 report (covering IST calls 1
-
6) obs
erved a significant fragmentation of
multimodal/multilingual interfaces (and virtual environments) across the IST programme.

"In important areas such as advanced multimodal interfaces, although the volume of the effort seems
appropriate, fragmentation, as

highlighted before (…)
[32 different action lines in the IST programme]
,
might limit the contribution of the programme in these fields."

The report furthermore recommends:

"The impact of the programme towards a realisation of an ambient intelligent landsc
ape could be
significantly improved by further concentration and aggregation of the effort spread across the
programme"

Due to the generic nature of "interface technologies" it is inevitable and even desirable that there is a
certain dispersion of the area

across multiple application domains. Mature or semi
-
mature interface
technologies would be expected to quickly find their way into novel and advanced applications.
Nevertheless, there is a recognised need for a concentration of the effort in design and te
chnology
development where such work also needs to be closely linked to exemplary applications.

With the aim of concentrating the effort in multimodal interfaces, a Cross Programme Action was
created (CPA 2) which covered user friendliness, human factors,

multilingual and multimodal dialogue
modes. So far 12 projects have been selected for funding. The effect can however only be evaluated at a
later stage as a) the first projects started only in mid 2001 and b) the management of the projects is
distributed

on the different key actions.

The portfolio of relevant projects addressing "interface technologies" is large and varied and covers
everything from pure technology development to projects focusing strongly on application aspects.

Current FP5 participants
in the area represent the whole spectrum of European players form academic
institutions and research institutes to large enterprises and SMEs, including IT
-
industry, car
manufacturers, telecom operators, media conglomerates, educational and cultural instit
utions.

Coverage of IST projects focusing on application aspects:



Multimodal, multilingual interaction and virtual environments for applications in e
-
culture, e
-
commerce, e
-
learning, e
-
work including intelligent meeting rooms/virtual meetings, intelligent

homes, in
-
vehicle command and control etc.;



Intelligent systems for independent living and intelligent assistive technologies.

Coverage of IST projects focusing on technology issues:



Conversational interfaces and language technologies such as machine tra
nslation and cross
-
lingual
information management including affectional interfaces;



Virtual/augumented reality, avatar and agent technologies, detection and modeling of human
motion and gestures, multimodal display technologies (optical, tactile, aural, et
c.);



Interfaces for mobile services and networked audio
-
visual services;

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






7



Novel integration of interaction modalities and implementation in applications for testing
interaction scenarios such as for artistic expression.


The
broader scene

(industrial allian
ces, national programmes, worldwide initiatives)

European excellence in multimodal interfaces, multilingual services and communications, speech and
language technologies as well as strong application areas such as e
-
work, e
-
learning, and cultural
content h
as attracted significant participation from non
-
EU countries.



US cooperation (participation funded by DARPA, NSF or by industrial partners themselves) is
significant in the areas of spoken language interaction, in e
-
work and in e
-
learning.



NAS countries p
articipate actively in FP5 projects on interface technologies and application areas.
Hungary has a significant strength in language technologies and the Baltic states have cooperated
successfully in IST projects on e
-
work.



Swiss, Norwegian and Israeli orga
nisations participate in FP5 projects on spoken language
interaction and multimodal systems.

A number of important international and national initiatives address multimodality and multilinguality.
Several EU member states and non
-
EU countries are running l
arge national initiatives in the IT area
which devote substantial efforts to human interface issues There are also a number of important
international standardisation efforts for www
-
markup languages specifically addressing mobility and
ubiquitous web
-
acce
ss.

Smartkom

(DE)

(
smartkom.dfki.de/start_en.html
) D
ialog
-
based Human
-
Technology Interaction
by Coordinated Analysis and Generation of Multiple Modali
ties. A German
national project 1999
-
2003

IT2006 (DE):

A new German national IT research programme 2002
-
2006, with a significant
effort devoted to "Human
-
technology interaction" including language engineering,
virtual and augmented reality.

DARPA (USA)

Ubi
quitous computing and Communicator programs strongly emphasises user
interfaces, especially for adverse environments. Multilinguality is a priority.

NSF (USA)

ITR
-

Information Technology Research (Human
-
Computer Interaction). This is a
large cross
-
NSF pro
gram that includes collaborative and
multilingual/conversational systems.

ATR (Japan)

Spoken language translation research lab, Human information science lab.

CSTAR

(www.c
-
star.org
) "Consortium for Speech Translation Advanced Research"

An international non
-
funded consortium aiming to promote spoken language
translation research through regular workshops and joint demonstrations. The
partners come from France, Italy, Germany, China,

USA, South
-
Korea and Japan.

W3C


(
www.w3.org/Voice
,
www.w3.org/WAI

and
www.w3.org/Mobile/CCPP
) T
he
world wide web consortium is working to expand access to the Web to allow
people to interact via key pads, spoken commands, listening to pre
-
recorded
speech, synthetic speech and music. This will allow any telephone to be used to
access appropriately de
signed Web
-
based services.

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






8

VoiceXML Forum


(
www.voicexml.org
) The VoiceXML Forum is an industry organization founded
by AT&T, IBM, Lucent and Motorola, and chartered with establis
hing and
promoting the Voice Extensible Markup Language (VoiceXML), a new
specification essential to making Internet content and information accessible via
voice and phone.

ETSI
-

TC HF

(
www.etsi.org/frameset/home.htm?/pressroom/Previous/2001/HF_symposium.htm
)
ETSI
-

Technical Committee Human Factors
-

is responsible f
or standards and
guidelines dealing with ease of use and accessibility of telecommunication
equipment and services, including the requirements of persons with disabilities and
elderly people.


5.

What are the challenges?


Most of the system and application de
sign today is technology driven only because the tools to
incorporate user behaviour as a parameter in the product development process are not available. New
enabling technologies will give birth to new types of applications and services (e.g. situation an
d
context awareness, augmented and virtual realities).
The user interface outlines (and limits) the
possibilities and capabilities of future systems.

Due to this strong interrelationship, the research on user interfaces must consider a correct
understandi
ng of the user expectations and behaviour, particularly in relation to the mobile applications
and services that will be made possible by the introduction of new interaction technologies
. Three main
activity areas are therefore envisaged: understanding the

user, enabling technologies, and new
interaction systems and paradigms.

Understanding the user

Exploratory user research has been used successfully over the years to guide the development of new
technology. It is fairly easy to let users judge whether the
y like a product or don’t, if it’s easy to use and
if they feel they have a need for it. It’s much harder to let users express their needs for the future, to
judge technology not yet implemented or visualised. Sometimes it is even hard for the users to exp
lain
their own behaviour and why they do what they do in a certain situation. To this extent a commonly
used method to gather user data is by observing their behaviour in their own natural context. These
methods are mainly used in academic research today.
Within the industry there is a great need to gather
observational data on users’ behaviour as input into product development. The methods of today are
however not easily applicable into product development. They must be fine
-
tuned to be able to meet
with t
he industries needs in order for them to be widely used for data collection.

In many fields of science and engineering, models of physical behaviour are used as important tools in
the analysis and design of new systems and products. Likewise, models of h
uman behaviour exist, but
are considered too crude for use in product and service design. Research should develop improved
models of human behaviour and users

within their domains of activity, so that it might become
possible to introduce user behaviour da
ta into the interface creation process. This would be crucial to
establish cognitively efficient ways for humans to interact with the systems and between themselves
through virtual environments.

It is also useful to consider methods and tools for creating
novel use situations, by constructing and
analysing a number of
reference scenarios

designed to encompass societal, economic and technologic
developments. Techniques of user study can also be applied to explore the requirements and needs of
users in variou
s dimensions (e.g. related to context awareness, assistive technologies).

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






9

Enabling technologies for the intuitive user interface

In the future computers will not only be increasingly distributed and mobile, but information will be
accessible from any fixed

or mobile position. Devices will recognise who we are and obtain
information about us. There will be a mixture of interoperable fixed and mobile devices. A device will
also be more aware of its user and more aware of its own environment. The terminal will

be able to
sense the presence of a user and calculate his/her current situation. Throughout the environment, bio
-
sensing will be used not only for entertainment and medical reasons, but also to enhance person
-
to
-
person and person
-
to
-
device communication

A
ll this new enabling technology will make it possible to create wireless and mobile specific services
that are not possible either in 3G or in fixed networks e.g. situation and context awareness, virtual
assistants, augmented and virtual realities.

This cl
ass of research lies between the pure user research and the pure technology research. It includes
the study of technologies that are closely bound to the interaction with the user (e.g. technologies for
augmented and virtual spaces

or
Body Area Networks
),
and the study of a class of human related
issues that require a solution in order for the technology to be truly useful (e.g. technologies for the
user to
manage the complexity

of future systems,
knowledge management

techniques).

New interaction systems a
nd paradigms

The user experience and the possible uses of the services and capabilities of a computer or
communication system

are
ultimately defined and charted out by the user interface.
It is probably the
true bottleneck of efficient information technolo
gy usage in general and Mobile Internet services in
particular.
It is therefore very likely that the introduction of new interaction will be the trigger of major
disruptive leaps in the use of new technology. Recent advances in wearable computing and intel
ligent
fabrics hint at future directions of ways to realise truly ubiquitous computing.


There are several major trends within the area of User Interaction that have the potential of causing a
disruptive change in the near
-
medium future:



Ubiquitous Communi
cation



Surrounding people with intelligent intuitive and adaptive
interfaces that are embedded in all kinds of objects, wireless connected.



Augmented User Interfaces


portable, deviceless (no mechanical interference with the user) and
extensible (the in
terface can re
-
shape itself for different purposes);



Conversational Interfaces



Using conversation (i.e. speech) effectively and unobtrusively in
solutions for users’ everyday problems;



Multi
-
modal Interaction



Using a combination of several interaction
techniques (or senses) to
interact or communicate.

This includes language and speech, vision, haptic (communicating
through the sense of touch) and other senses, context
-
awareness and self
-
awareness.



Avatars


to improve the user interaction by means of a
consistent interaction metaphor across
different modalities (e.g. for mobile users where the context of use changes all the time)




Virtual Environments

(using Augmented/Mixed/Virtual Reality techniques)


Using a semi
-
transparent visor or head up display,
to enhance the users’ perception of the reality by embedding
objects such as 3D images, videos, text, computer graphics, sound, etc.



Interaction with knowledge and information spaces



Complex multimedia content will require
advanced multimodal interfaces

to support the user in retrieving, visualising and manipulating the
information. In many cases this will also require negotiation steps between the user and the (self
aware) content.

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






10

A common aspect for all these trends is the need to address the fact tha
t while the complexity of
technology used in everyday life has exploded, the human cognitive capabilities (sensory and motor
limits, short and long term memory, the brains processing power etc.) have evolved very slowly. In
order to fully exploit these cap
abilities we need to extend the interaction with more senses (touch,
smell, and taste) and at the same time make better use of the senses used today (hearing and vision) by
exploring peripheral vision and ambient listening. With even a longer perspective w
e need also to
consider inventing “new senses” by combining current senses and extending them with new
technology. In the future
All Senses Communication

would be a way to enhance the communication
with other entities (humans or machines) using a combinati
on of several of our present or future senses.
Augmented User Interfaces, Avatars, Multi
-
modal Interaction, Conversational Interfaces will be
possible ways to implement this concept.

6.

Constituent disciplines and communities.


The field of interface technolo
gies is very broad, inherently interdisciplinary and involves a range of
scientific and engineering disciplines.



Interface design

addresses issues in human centred design, taking into account the user's
preferences and limitations (e.g. for disabled and
elderly users), the limitations of the device and
environmental demands (such as noise and visibility conditions). It involves disciplines such as
human factors, cognitive science, medicine, art and design



Human interface technologies

aims at exploiting an
d extending the human senses to achieve
optimal interfaces to systems and services allowing also "superhuman performance" when needed
(such as seeing in the dark, understanding multiple languages). This involves a range of disciplines
such as speech, langu
age, user dialogue, user modelling and adaptive systems, vision, scene
analysis, image processing, animation, intelligent agents and avatars, smell, touch, affective
computing. VR/AR technology development.



Communications infrastructure and hardware develo
pment

is part of the required
infrastructure for designing advanced user interfaces. It involves work on visual displays, cameras,
microphones/loudspeakers, tactile displays and input devices, haptic feedback devices and other
sensors and actuators as well

as wireless and fixed networks, LAN, WAN, BAN etc.



System integration

is key to realising multimodal interfaces that build upon combinations of
multiple modalities of interaction, interpretation of multiple data streams, sensor fusion etc.


The science
-

technology and business communities related to these disciplines include:

Science and technology related communities

Application related communities



Human factors, cognitive science, medicine
and design



Image processing, computer vision,
computer graphic
s and animation



Pattern classification, machine learning,
adaptive systems



Speech processing, dialogue design and
natural language processing



Information extraction



Workplace technol
ogy developers/architects



Virtual workplace developers, telework
associations



Educational and training actors



Cultural heritage actors, libraries, museums



Media and broadcasting organisations
including interactive media



Edutainment and game industry



Busin
ess organisations

Internal Reflection Group on Interface Technologies

Multisensorial, multilingual interactivity and virtual environments

25/10/13






11



Virtual and augmented reality



Interface hardware development and design




Accessibility and design
-
for
-
all sector
research actors and associations representing
people with special needs.



Consumer goods manufacturers



Car manufacturers



Computer software makers


7.

Interface to other IRGs

Given the central role of n
atural user interfaces in the implementation of the Ambient Intelligence
vision, strong links are expected with most of the themes of the other Internal Reflection Groups. In
particular, important interaction in the following areas is expected:

Communicati
on Infrastructures
, as for the development of enabling technologies such a Body Area
Networks, availability of wireless communication technologies allowing the implementation of ambient
intelligence concepts, etc.

Knowledge Technologies
, especially for int
uitive access and retrieval of audio/visual information,
visualisation of complex information spaces and interaction with knowledge.

Technologies for major societal challenges and for business and work challenges
, in order to provide
the necessary input to

make such applications user
-
friendly and intuitive in use.

Microsystems, micro and nano technologies, and displays
, to ensure that the proper user requirements
and needs are taken into account in the development of the basic components of “wearable” devic
es.

Future research domains at the frontiers of science and technology
, as for the exploitation of advances
in human interaction techniques (e.g. brain waves used to control systems, etc.).

Trust and Confidence
, to ensure that all the relevant security and

integrity concerns are correctly
considered in the implementation of natural and deviceless interfaces.



…*…