User Control over User Adaptation: A Case Study

tripastroturfΤεχνίτη Νοημοσύνη και Ρομποτική

7 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

60 εμφανίσεις

User Control over User Adaptation: A Case Study

Xiaoyan Peng and Daniel L. Silver

Acadia University,
Jodrey School of Computer Science,

Wolfville, NS, Canada B4P 2R6

danny.silver@acadiau.ca

Abstract.

The
A theory of user expectation of system interaction
is introduced
in the context of User Adapted Interfaces. The usability of an intelligent email
cl
i
ent that learns to filter spam emails is tested under three variants of adapt
a-
tion: no user modeling, user modeling with fixed (optimal) spam cut
-
offs, and
u
ser modeling with user adjustable spam cut
-
offs. The results supported our
hypoth
e
sis that user control over adaptation is preferred because the user can
maintain the system’s interaction state within a region of user expectation. This
remains true even w
hen performance of the system (accuracy of spam filtering)
degrades because of errors in user control (adjustment of spam cut
-
offs).

1 Introduction

Research into User Adapted Interfaces (UAI) brings together concepts from Human
-
Computer Interaction (HCI)

and User Modeling (UM) to improve the usability and
performance of software systems.
Controllability

is one

of the major usability issues
for UAI technology. Some researchers advocate maximum user control over all a
s-
pects of system adaptation, others sug
gest that maximum control is not always be the
best approach as it can lead to distraction and inefficiency [1].
There has been much
discussion among researchers about controllability trade
-
offs. However, as Jameson
[2] argues, there is a deficiency of sys
tematically gathered evidence about what users
themselves think about adaptation and controllability.

Our research investigates UAI technology in the context of an intelligent email cl
i-
ent. This paper focuses on user control over a learned User Model tha
t is able to
pr
e
dict the priority of incoming e
-
mail messages. We theorize that as long as the state
of system interaction is within the current region of user expectation, the user will be
satisfied with adaptation. If the system’s interaction falls out
side of this region of
expectation then user satisfaction will degrade. To prevent dissatisfaction the user
must be given
control over aspects of adaptation that limit changes in intera
c
tion state.

An adaptive intelligent email client is our application
of choice for employing UAI
because it offers a lot of functionality that can be personalized, example data is readily
available and knowledgeable test subjects are easily found [3]. In this study we focus
on predicting the priority of incoming email mess
ages based on a learned user model.
Consequently, the predicted priorities can be used to filter out low priority and uns
o-
li
c
ited “spam” email. There has been some excellent work in this area using N
a
ïve
Bayes, Bayesian networks, artificial neural netwo
rks, and

k
-
nearest neighbour met
h-
ods [
4]. Most of this research has focused on the performance of the learning alg
o-
rithms, with little attention given to usability of the system from the user’s pe
r
spective.
The lack of a user’s point of view has led to a

significant barrier against acceptance of
UAI technology. For example, some email
filters automatically file spam into a spam
folder. This can be problematic as users have varying tolerance for placement of l
e
gi
t-
imate email into a spam folder [5].

Althou
gh UAI has great potential, much research is needed. Perhaps most i
m-
po
r
tantly, UAI can frustrate good HCI design because the interface may be perceived
as a moving target that at times does not meet the expectations of the user [6]. The
fo
l
lowing section

presents a theoretical model of the relationship between the expect
a-
tions of a user and the changing state of a UAI.












Fig. 1.

Adaptation viewed as movement through an HCI state space

2 User
Interaction
Expectation and Adaptation

Consider a s
pace of HCI states, as shown in Figure 1, where interaction states are
topologically organized such that similar states are proximal to each other. System
adaptation can be described as a trajectory, P, through the space. Each point

along P
represents the

system
’s state of interaction with the
user

at a particular time. A user
has a region of interaction expectation, R, that preferably is centered on the sy
s
tems
current state of interaction, s, or at least contains the s. The size of R, |R|, is the nu
m-
be
r of interaction states within R. If |R| = 1, then no variation from s will be to
l
erated
by the user;
this
user
is

very conservative in terms of adaptation. If |R| = n then there
are n states within R that will
be
acceptable to the user;
this

user
is

mo
re accep
t
ing of
adaptation. Ideally, as the system inte
r
face adapts, the user shifts her R so as to centre
it once again on the new s. This transition is not always in co
n
cert. If the system
adapts too quickly then the user is left behind at R2. If the

system adapts too slowly
then the user may assume an inte
r
action state too far in advance of the current s, at R3.
In either case the user will not be satisfied with the system and task perfor
m
ance will
suffer. The worst case is when the user’s expectati
on region is R’, a region of intera
c-
tion space through which adaptation will never pass; the user is co
n
tinually dissati
s-
fied.
To be successful, a UAI must provide the user with control over adaptation. We
adv
o
cate that user control should be exercised ove
r the deplo
y
ment of user models

rath
er than their development. Model deployment requires min
i
mal knowledge o
f the
UM subsystem. In the case of an email client, a user model for se
t
ting incoming e
-
mail priority can be automatically developed using informa
tion retrieval and machine
lear
n
ing methods [4]. Control over automatic spam filtering can then be provided by
allo
w
ing the user to adjust cut
-
off values that dete
r
mine when the predicted priority of
a message is at the level of legit
i
mate or spam.

We h
ave created an intelligent email client using this approach and developed an
i
n
tuitive user interface for controlling adaptation.
There are two priority cut
-
off va
l-
ues; one is the suspect cut
-
off and the other is the spam cut
-
off. E
mail with a priority
va
lue lower than the spam cut
-
off will be placed in the Spam folder. Email with a
priority value equal to or higher than the suspect cut
-
off will be filed into the Inbox
folder. Email with a priority equal to or higher than the spam cut
-
off and lower than
th
e su
s
pect cut
-
off will be put in a Suspect folder. Provided the user model is acc
u-
rate, the approach will direct the most important legitimate email to the Inbox folder.
The Su
s
pect folder can be cleaned up periodically, sorting legitimate and spam email
.
Notably, it is this process that provides data for improving the user model. Using a
simple GUI slider, a new or conservative user can select cut
-
off values that curtail the
UAI’s automated classification of legitimate and spam messages (thus reducing

risk).
A more experienced user can establish cut
-
offs that give the UAI greater freedom to
classify email messages (maintaining risk as the user model improves over time). In
this way, adaptation of the systems interaction state can be kept within the us
er’s cu
r-
rent region of interaction expectation.

3

Empirical Study

The objective of this experiment is to demonstrate that user control over appropriate
aspects a UAI can improve the usability of the application and user satisfaction.

3
.1
Materials and
Methods

The study scenario is as follows: Each subject is working as a secretary for a profe
s-
sor. She or he must classify the incoming email (initially received in either the Inbox,
Suspect or Spam folder) by moving the messages into one of six relevant fo
lders i
n-
cluding the Spam folder. Twenty eight subjects were selected from the university
campus (ages 18
-
38). The performance of the email UAI is recorded in terms of false
positives (FP
-

legitimate emails placed in the Spam folder) or false negatives (F
N
-

spam email placed in the Inbox folder) and overall error (FP+FN).

Three variants of the system were tried by each subject and compared as per [2].
Variant N employs a UAI based on “no user model”. The subjects
had
to manually sift
through the email m
essages for legitimate and spam email messages. Variant F deve
l-
ops a user model but uses “fixed cut
-
off values” to determine the priority required for
spam and legitimate emails. The fixed cut
-
off values are set to optimal values as d
e-
termined by prelimin
ary trials. No adjustment from these values would make signif
i-
cant improvement in UAI performance. Variant A for “adjustable cut
-
off values”
develops a user model and allows the subjects to
adjust the spam and suspect cut
-
off
values as desired. This giv
es the subjects control over the UAI.

A within
-
subject experimental design was selected because the subjects were e
x-
pected to vary considerably in their use of the system and tolerance to adaptation.
Each subject was provided the same working environment.

Each subject learned the
experimental procedure from an instruction file without prompting by a researcher.
Each subject used all 3 system variants in one of two possible orders; the first order
being N, F and A and the second being N, A, and F. A diff
erent subset of emails was
used for each variant to prevent subjects from memorizing the content of messages.

The data used in the experiment was collected from a professor at Acadia over a 5
month timeframe in 2003 [4]. A different subset of 200 emails wa
s used for each of
the 3 variants of the system. 100 emails from a subset were sent to the email client
and the subject was asked to manually classify them into their respective folders. A
message was given a priority of 0 if placed in the Spam folder an
d a priority of 1 ot
h-
erwise. This acted as training data for developing a user model for predicting message
priority. A final 100 emails from the subset were sent to the email client for automatic
priorit
i
zation and classification.

Each subject was survey
ed following their trial of the 3 system variants [2]. The
survey asked the subjects for their opinions on 5 questions: (1) Did you find the sy
s-
tem easier to user after the user model was developed; (2) Did you prefer being able to
adjust the spam and susp
ect cut
-
off values; (3) Do you think the cut
-
off adjustment
increased the accuracy of email classification; (4) Would you like to have user mode
l-
ing on you current email client; and (5) Would you like to have user model with cut
-
off adjustment on you curre
nt email client? The subjects were asked to respond using
a five
-
level scale of agreement as shown in Figure 2 and to explain their reasoning in a
comment area. In addition, each subject indicated if they had any significant problems
using the system and

whether they preferred false positives or false negatives when
filtering email. The FP and FN statistics were tracked for each subject and used to
determine the performance of the UM on incoming email.

3
.1 Results & Discussion

The results from the post
-
trial surveys show that 92.86% preferred the UAI variant of
the system after the user model was developed, with 82.14% preferring the adjustable
spam and suspect cut
-
offs over the fixed cut
-
offs. 78.57% felt that the cut
-
off adjus
t-
ment increased the accur
acy of email classifications with more subjects preferring cut
-
off adjustment on their current email clients (78.57%) than user modeling with fixed
cut
-
offs (71.43%). These statistics indicate that user control contributes to user sati
s-
faction. In contra
st, the system variant with adjustable cut
-
offs had higher mean mi
s-
classifications (25.64%) than the fixed cut
-
off variant (20.04%). The difference b
e-
tween misclassifications caused by variant A and F is significant (p
-
value = 0.01).


Despite the fact that

significantly more misclassifications were made by variant A,
67.9% of the subjects (19/28) preferred variant A over F. Subject 14
was

an extreme
case of where a user
preferred control even though it reduced

system performance.
He generated 5 misclassifi
cations under variant F and 49 misclassifications under
variant A. His comment “this allows me to set up values to better match my profiles”
shows a strong desire to remain in control. Other typical responses for those who
preferred adjustable cut
-
offs we
re: “It helps me to control how the emails will be
separated”, “It is good to add user’s point view to the system”, and “I like the feeling
of control”.

The majority of subjects liked adaptation as long as they felt in control. Of the su
b-
jects who liked

cut
-
off adjustment, 95.65% preferred FN over FP meaning they most
dislike finding legitimate email messages in the Spam folder. Adjustment of the cut
-
offs allows the users to err on the side of FN classifications even if this reduces the
overall performan
ce of the UAI. Of the subjects who responded “do not know” or
“disagree” to cut
-
off adjustment, 80% are less sensitive to FP (legitimate emails cla
s-
sified as spam). The fixed default cut
-
off values worked well for that purpose, the
subjects recognized thi
s and preferred it.

4

Conclusions and Future Work

This paper has investigated the relationship between user control over a UAI and user
satisfaction and system performance. We presented a theoretical model that suggests
users will be satisfied with a UA
I provided the interaction state of the system is mai
n-
tained within the current region of user expectation. If the system’s interaction falls
outside of this region of expectation then user satisfaction will degrade. One approach
to preventing this from
happening is to give the user control over aspects of adaptation
that limit changes in interaction state. Specifically, in the case of the email client, the
user controls the cut
-
off at which emails are considered legitimate or spam. The r
e-
sults of an empi
rical study using 28 subjects demonstrated that user satisfaction is
improved with control over adaptation even if this means reducing system perfo
r-
m
ance (higher misclassification rates). We are currently working on a related pro
b-
lem of automat
i
cally cla
ssifying emails to one of several category folders.

References

1. Kay, J.:
Learner control, Proceedings of User Modeling & User
-
Adapted Interaction

(
2001
)

2.
Jameson A., and Schwarzkopf, E.
:

Pros and Cons of Controllability: An Empirical Study
,
Adaptive Hy
permedia and Adaptive Web
-
Based Systems: Proceedings of AH

(
2002
)

3. Crawford, E., Kay, J., and McCreath E.
:

An Intelligent Interface for Sorting Electronic Mail,
IUI’02, San Francisco, California, USA

(
2002
) 13
-
16

4. Fu, C.
:

User Modelling for an Adaptive

System: an Intelligen
t Email Client
, Master Th
e
sis,
Jodrey School of Computer Science, Acadia University, Wolfville, Canada

(
2003
)

5. Crawford, E., Kay, J., and McCreath, E.
:

Automatic Induction of Rules for Email Classific
a-
tion
, Proceedings of 6
th

Austra
lian Document Computing Symposium

(
2001
)

6. Cranor, L., F.
:
Designing a Privacy Preference Specification Interface: A Case Study, Pr
o-
ceedings of the Workshop on HCI and Security Systems
, CHI2003, Florida (2003)