User Interface Agents

jabgoldfishAI and Robotics

Oct 19, 2013 (4 years and 25 days ago)

107 views

User Interface
Agents



Roope Raisamo (
rr@cs.uta.fi
)

Department of Computer Sciences

University of Tampere


http://www.cs.uta.fi/sat/

User Interface Agents


Schiaffino and Amandi [2004]:

Interface agents are computer programs that
have the ability to
learn a user’s preferences

and working habits, and (the ability) to
provide him/her proactive and reactive
assistance

in order to increase the user’s
productivity.

User Interface Agents


A
user interface agent

guides and helps
the user


Many user interface agents observe the activities
of the user and suggest better ways for carrying
out the same operations


They can also automate a series of operations
based on observing the users (e.g., Eager)


Many user interface agents are based on
learning patterns of user activity.

VIDEOS: two examples of

user interface agents


Allen Cypher:
Eager
(2:08)

Henry Lieberman:
Letizia

(1:36)

Eager


automated macro generator

Allen Cypher, 1991

http://www.acypher.com/Eager/


Observes the activities of the user and tries to
detect repeating sequences of actions. When such
a sequence is detected, Eager offers a possibility
to automate that task.


like an automated macro generator


this kind of functionality is still not a part of
common applications, even if it could be.


Eager


Eager observes
repeating
sequences of
actions


When Eager
finds one, it
jumps on the
screen and
suggests the
next phase


Eager


When all the
phases suggested
by Eager have
been shown and
accepted, the user
can give Eager the
permission to
carry out the
automated task.

Letizia


a browser companion agent



Letizia

observes
the user and tries
to preload
interesting web
pages at the same
time as the user
browses through
the web

[Lieberman, 1997]

http://lieber.www.media.mit.edu/people/lieber/Lieberary/Letizia/Letizia
-
Intro.html

Letizia

Letizia

Traditional browsing leads
the user into doing a depth
first search of the Web

Letizia conducts a concurrent
breadth
-
first search rooted from
the user's current position

Another example: [Marais and Bharat, 1997]

Some user interface agents


Letizia
,


Farewell Clippy
,


Eager
,


beebot.com

(MS Agent demos)


KTH
: Olga and
August


REA

(Real Estate Agent)


Hal's Legacy
,
archive

and
2001
.


Philips iCat





Questions?

The appearance of agents


The appearance of an agent is a very important
feature when a user tries to find out what a certain
agent can do.


It is a bad mistake to use such an appearance that
makes the user believe an agent to be more
intelligent than it really is.


The appearance must not be disturbing.

Computer
-
generated talking heads
and moving bodies


one of the most demanding forms of agent
presentation


a human head suggests the agent to be rather
intelligent


a talking head is probably the most natural way to
present an agent in a conversational user interface.

FaceWorks

Drawn or animated characters


the apperance of the agent has a great effect on the
expectations of the user


a paper clip vs. a dog vs. Merlin the Sorceror


Continuously animated, slowly changing or static
presentation


VIDEO: REA
--

an example of a
conversational real estate agent


[Cassel
et al
., 1999] Embodiment in
Conversational Interfaces: Rea.
CHI’99
Video Proceedings
, 1999. (2:08)

Textual presentation


Textual feedback of the actions of an agent


Text input should normally be avoided if it is
not a part of the main task of the agent.


Chatterbots


e.g.,
Julia

that is a user in a MUD (multi user
dungeon) world. It can also answer to questions
concerning this world.

http://lcs.www.media.mit.edu/people/foner/Yenta/
julia.html


so called NPCs (non
-
person characters) in
multiplayer role
-
playing computer games.

Auditory presentation


An agent can also be presented only by voice
or sound, the auditory channel:


ambient sound


beeps, signals


melodies, music


recorded speech


synthetic speech


More on this next week.

Haptic presentation


In addition to auditory channel, or to replace
it, an agent can present information through
haptic feedback


Haptic simulation modalities


force and position


tactile


vibration


thermal


electrical

Haptic feedback devices

Inexpensive devices:



The most common haptic devices are
still the different force
-
feedback
controllers used in computer games, for
example force
-
feedback joysticks and
wheels.


In 1999 Immersion Corporation’s
force
feedback mouse

was introduced as
Logitech Wingman Force Feedback
Gaming Mouse


In 2000 Immersion Corporation’s
tactile
feedback mouse

was introduced as
Logitech iFeel Tactile Feedback Mouse

No (direct) presentation at all


An agent helps the user by carrying out different
supporting actions


e.g., prefetching needed information from the web,
automatic hard disk management, …


An indirectly controlled background agent


The question: How to implement indirect control?


Multisensory input: the agent is observing a system, an
environment, or the user.


Related to ubiquitous (intelligent) environments


More on this next week.


Related user interface concepts:


Conversational User Interfaces


Multimodal User Interfaces

Conversational User Interfaces


Why conversation?


a natural way of communication


learnt at quite a young age


to fix the problems of direct manipulation interfaces


Conversation augments, not necessarily replaces a
traditional user interface


the failure of Microsoft Bob


Microsoft Office Assistant

Microsoft Office Assistant


Microsoft Office assistant tries to
help in the use of Microsoft
Office software with a varying
rate of success.


The user can choose the
appearance of the agent


unfortunately, this has no effect on
the capabilities of the agent


A paper clip is most likely a
better presentation for the
present assistants than Merlin
the sorceror.

Multimodal User Interfaces



Multimodal interfaces

make use of
several input and/or feedback modalities to
interact with the user.


Multimodal User Interfaces


An agent makes use of multimodality when
observing the user, e.g.:


speech recognition


reacts on speech commands, or observes the user
without requiring actual commands


machine vision, pattern recognition:


recognizing facial gestures


recognizing gaze direction


recognizing gestures

Multimodal User Interfaces


a specific problem in multimodal interaction is to
combine the simultaneous inputs.


this requires a certain amount of domain knowledge and
”intelligence”


this way every multimodal user interface is at least in some
way a user interface agent that tries to find out what the
user wants based on the available information


A high
-
level architecture for

multimodal user interfaces

Adapted from
[Maybury and
Wahlster, 1998]

Put



That



There

[Bolt, 1980]

Combining inputs

[Nigay and Coutaz, 1993]

[Nigay and Coutaz, 1995]

Combining inputs

Agents fit well in handling multimodal interaction


there can be specific agents for each input and feedback
channel


the raw input from the lower
-
level input agents is then
processed and combined with others by the higher
-
level
agents working in a higher abstraction level


there can be any necessary amount of agent levels in a
given system


finally, the root agent has all the available information from
different input devices and sensors, and acts based on this
information

Example: DEC Smart Kiosk


Smart Kiosk was a research project at
Compaq
-
Digital (now HP) Cambridge
Research Laboratory in which easy
-
to
-
use information kiosks were built to be
used by all people


Combined new technology:


machine vision, pattern recognition


speech synthesis (DECtalk)


speech recognition


animated talking head (DECface)

[Christian and Avery, 1998]

Example: DEC Smart Kiosk


Vision

DECface

Netscape Navigator

Active vision
zone

Touchscreen

Example: DEC Smart Kiosk


Example: DEC Smart Kiosk

36

Roles of Agents

Agent

Agent

Agent

observes

observes

gives feedback

gives feedback

Fully automatic,
active
observation

collaborative and

passive agent

user
groups

Both active and
passive
gathering of
information

results

results

results

User

User

User

Roles of Agents

System



Network

conversation

Conversational,
anthropomorphic
agents

Agent

Background
maintaining
assistant

observes/

adjusts

results

User

User

Agent

Both active and
passive gathering
of information,
collaborative

User

user
groups

observes

gives feedback

talking head




Questions?

References

[Bolt,

1980
]

Richard

A
.

Bolt,

Put
-
that
-
there
.

SIGGRAPH


80

Conference

Proceedings
,

ACM

Press,

1980
,

262
-
270
.

[Christian

and

Avery,

1998
]

Andrew

D
.

Christian

and

Brian

L
.

Avery,

Digital

Smart

Kiosk

project
.

Human

Factors

in

Computing

Systems,

CHI


98

Conference

Proceedings
,

ACM

Press,

1998
,

155
-
162
.

[Lieberman,

1997
]

Henry

Lieberman,

Autonomous

interface

agents
.

Human

Factors

in

Computing

Systems,

CHI


97

Conference

Prodeedings
,

ACM

Press,

1997
,

67
-
74
.

[Maybury

and

Wahlster,

1998
]

Mark

T
.

Maybury

and

Wolfgang

Wahlster

(Eds
.
),

Readings

in

Intelligent

User

Interfaces
.

Morgan

Kaufmann

Publishers,

1998
.

[Marais

and

Bharat,

1997
]

Supporting

cooperative

and

personal

surfing

with

a

desktop

assistant
.

Proceedings

of

UIST


97
,

ACM

Symposium

on

User

Interface

Software

and

Technology
,

ACM

Press,

1997
,

129
-
138
.

[Nigay

and

Coutaz,

1993
]

Laurence

Nigay

and

Joëlle

Coutaz,

A

design

space

for

multimodal

systems
:

concurrent

processing

and

data

fusion
.

Human

Factors

in

Computing

Systems,

INTERCHI


93

Conference

Proceedings
,

ACM

Press,

1993
,

172
-
178
.

[Nigay

and

Coutaz,

1995
]

A

generic

platform

for

addressing

the

multimodal

challenge
.

Human

Factors

in

Computing

Systems,

CHI


95

Conference

Prodeedings
,

ACM

Press,

1995
,

98
-
105
.

[Schiaffino

and

Amandi,

2004
]

Silvia

Schiaffino

and

Analía

Amandi,

User



interface

agent

interaction
:

personalization

issues
.

International

Journal

on

Human
-
Computer

Studies

60
,

Elsevier

Science,

2004
,

129
-
148
.