The Massive User Modelling System (MUMS)

hedgebornabaloneSoftware and s/w Development

Dec 2, 2013 (3 years and 11 months ago)

158 views



The Massive User Modelling System (MUMS)

Christopher Brooks
1
, Mike Winter
1
, Jim Greer
2
, and Gordon McCalla
2

1

Department of Computer Science, University of Saskatchewan,

57 Campus Drive, Saskatoon, SK, S7N 5A9, Canada

{cab938, mfw127}@mail.usask.ca

2
Depa
rtment of Computer Science, University of Saskatchewan,

57 Campus Drive, Saskatoon, SK, S7N 5A9, Canada

{greer, mccalla}@cs.usask.ca

Abstract.

Effective distributed user modelling in intelligent tutoring systems
requires the integration of both pedagogica
l and domain applications.

This int
e-
gration is difficult, and often requires rebuilding applications for the specific e
-
learning environment that has been deployed.

This paper puts forth both an a
r-
chitecture and an implementation prototype for achieving th
is integration.

It f
o-
cuses on providing platform and language neutral access to services,
without
commitment to any pa
r
ticular ontology.

1. Introduction

A recent trend within intelligent tutoring systems and related educational technologies
research is to
move away from monolithic tutors that deal with individual learners,
and instead favour “adaptive learning communities” that provide a related variety of
collaborative learning services for multiple learners
[9]
.

An urgent cha
l
lenge facing

this new breed of tutoring systems is the need for precise and timely coordin
a
tion that
facilitates effective adaptation in all constituent comp
o
nents.

In addition to supporting
collaboration between the native parts of a tutoring system, an effective int
er
-
component communic
a
tion system is required to provide the ability to know of and
react to learner actions in external applications.

For exa
m
ple, consider the kinds of
errors a student encounters when trying to solve a Java programming problem.

If the
er
rors are syntactical, a tutor may find it useful to intervene directly within the deve
l-
opment environment that student is using.

If the errors are related to the higher level
course concepts, the tutor may instead find it useful to dynamically assemble and

deliver external resources (learning objects) to the student.

Finally, if an appr
o
priate
solution can not be found that helps the student to r
e
solve their errors, the tutor may
find it useful to refer the user to a domain expert or peer who has had succes
s at sim
i-
lar tasks.

To provide this level of adaptation, the tutor must be able to form a coherent model
of students as they work with different domain applications.

The tutor must be able to
co
l
lect, understand, and respond to user modelling “events” in b
oth real time and on
an a
r
chival basis.

These needs can be partially addressed by integrating intelligent
tutoring system functionality within larger web
-
based e
-
learning systems including
2

Christopher Brooks1,

Mike Winter1, Jim Greer2, and Gordon McCalla2


learning ma
n
agement systems such as WebCT
[28]

and Blackboard
[3]

or e
-
learning
portals like uPortal
[26]
.

These appl
i
cations provide an array of functionality meant to
directly support learning activities including social communication, learner manag
e-
ment, and content delivery functions.

An inherent problem wit
h these e
-
learning sy
s-
tems is that they are often unable to capture interaction b
e
tween a learner and other
applications the learner may be using to complete their learning task.

While a pote
n-
tial solution to this problem is to int
e
grate all possible exter
nal applications that may
be used by the student within an e
-
learning system, this task is difficult at best due to
proprietary API’s and e
-
learning sy
s
tem homogeneity.

In
[27]

we proposed a method of integrating various e
-
learning applications using

a
multi
-
agent architecture, where each application was represented by an agent that
negot
i
ated with other agents to provide information about learners using the system.

A learner using the system was then able to see portions of this information by inte
r-
a
cting with a personal agent, who represented the tutor of the system. In this system,
the tutor’s sole job was to match learners with one another based on learner prefe
r-
ences and competencies.

This system was useful at a conceptual level, but su
f
fered
from

the drawbacks of being difficult to implement and hard to scale
-
up.

The integr
a-
tion of agent features (in particular reasoning and negotiation) within every applic
a-
tion instance required high computational power forcing the user into a more centra
l-
ized co
mputing environment.

To further provide the performance and reliability r
e-
quired, agents had to be carefully crafted using a proprietary protocol for communic
a-
tion.

This hindered both agent interoperability and system extensibi
l
ity.

This paper presents a f
ramework and prototype specifically aimed at supporting the
process of collecting and disseminating user information to software components
inte
r
ested in forming user models.

This framework uses both semantic web and web
service technologies to encourage i
nteroperability and extensibility at both the sema
n-
tic and the syntactic levels.

The organization of this paper is as follows: Section 2
describes the framework at a conceptual level.

Section 3 follows with an outline of the
environment we are using to pro
totype the system, with a particular emphasis on the
integration of our modelling framework with the legacy e
-
learning applications we are
trying to support.

Section 4 contrasts our work with similar work in the semantic web
community.

Finally, Section 5 c
oncludes with a look at future goals.

2. The MUMS Framework

We present the Massive User Modelling System (MUMS) framework, which is i
n-
spired by traditional event systems such as CORBA
[20]

and JINI
[25]
.

The principle
artefact with
in MUMS is the modelling opinion being expressed.

We adopt the defin
i-
tion of an opinion as a
temporally grounded codification of a fact about a set of users
from the pe
r
spective of a given event producer.

Opinions are passed between three
independent entit
ies in the fram
e
work:


1.

Evidence Producers: observe user interaction with an application and publish opi
n-
ions about the user.

These opinions can range from direct observations of the inte
r-
action that has taken place, to beliefs about the user’s knowledge, d
esires, and i
n-
The Massive User Modelling System (MUMS)

3


tentions.

While the opinions created can be of any size, the focus is on creating
brief contextua
l
ized statements about a user, as opposed to fully modelling the u
s-
er.

2.

Modellers: are interested in acting on opinions about the user, usually by

reasoning
over these to create a user model.

The modeller then i
n
teracts with the user (or the
other aspects of the system, such as learning materials) to provide adaptation.

Modellers may be inte
r
ested in modelling more than one user, and may receive
opi
nions from more than one producer.

Further, modellers may be situated and pe
r-
form purpose
-
based user modelling by restricting the set of opinions they are inte
r-
ested in receiving.

3.

Broker: acts as an intermediary between producers and modellers.

The broker
r
e-
ceives opinions from producers and routes them to interested modellers.

Modellers
communicate with the broker using e
i
ther a publish/subscribe model or a qu
e-
ry/response model.

While the broker is a logically centralized component, di
f
ferent
MUMS implemen
t
a
tions may find it useful to distribute and specialize the services
being provided for sca
l
ability reasons.


While the definition of an opinion centers
on

human users, it does not restrict the
producer from describing other entities and relationships of i
nterest.

For instance, an
evidence pr
o
ducer embedded within an integrated software development environment
might not just express information about the particular compile
-
time errors a student
receives, but may also include the context of the student’s his
tory for this progra
m-
ming session, as well as some indication of how the tutor should provide treatment for
the problem.

The definition also allows for evidence producers to have disagreeing
opinions about users, and for the opinion of a producer can chang
e over time.

This three
-
entity system purposefully supports the notion of active learner mode
l-
ling
[17]
.

In the active learning modelling philosophy, the focus is on creating a
learner model situated for a given purpose, as opposed to crea
t
ing a comple
te model
of the learner.

This form of modelling tends to be less intensive than traditional user
modelling techniques, and focuses on the just
-
in
-
time creation and delivery of models
instead of the storage and retrieval of models.

The MUMS architecture su
p
ports this
by providing both a stream
-
based publish/subscribe and an archival query/response
method of obtaining opinions from a broker.

Both of these modes of event delivery
require that modellers provide a semantic query for the opi
n
ions they are interes
ted in,
as opposed to the more traditional event system n
o
tions of channel subscription and
producer subscription.

This approach decouples the produ
c
ers of information from the
consumers of information, and leads to a more easily adaptable system where new

producers and modellers can be added in an as
-
needed fashion.

The stream
-
based
method of retrieving opinions allows modellers to pr
o
vide just
-
in
-
time reasoning,
while the archival method allows for more resource
-
intensive user modelling to occur.

All opin
ions transferred within the MUMS system include a tim
e
stamp indicating
when they were generated, allowing modellers to build up more complete or historical
user models using the asynchronous querying capabilities pr
o
vided by the broker
.

By applying the ada
ptor pattern

[
8]

to the system, a fourth entity of interest can be
d
e
rived, namely the filter.


4

Christopher Brooks1,

Mike Winter1, Jim Greer2, and Gordon McCalla2


4.

Filters: act as broker, modeller, and producer of opinions.

By registering for and
re
a
soning over opinions from producers, a filter can create higher level opinions.

This of
floads the amount of work done by a modeller to form a user model, but
maintains the more flexible decentralized environment.

Filters can be chained t
o-
gether to provide any amount of value
-
added reasoning that is desired.

Finally, fi
l-
ters can be specialize
d within a particular instance of the MUMS framework by
providing d
o
main specific rules that govern the registration of, processing of, and
creation of opi
n
ions.


Interactions between the entities are shown in Fig
.

1.

Some set of evidence produ
c
ers
publis
h opinions based on observations with the user to a given broker.

The broker
routes these opinions to interested parties (in this case, both a filter and the modeller
towards the top of the diagram).

The filter reasons over the opinions, forms derivative
s
tatements
, and publishes these new opinions back to the broker

and any modellers
registered with the filter
.

Lastly, modellers inte
r
ested in retrieving archival statements
about the user can do so by querying any entity which stores these opinions (in this

example
, the
second mode
l
ler
queries the
broker instead of registering for real time
opinion notification
).


Fig.
1
.

A logical view of the MUMS architecture

The benefits of this architecture are numerous.

First, the removal of reasoning a
nd
neg
o
tiation abilities from the producers of opinions greatly decreases the complexity
when creating new producer types.

Instead of being rebuilt from scratch with user
modelling in mind, existing applications (be they applications explicitly meant to
su
pport the learning process, or domain
-
specific applications) can be easily extended
and added to the system.

Second, the decoupling between the producers and the mo
d-
The Massive User Modelling System (MUMS)

5


ellers serves to increase both the performance and the extensibility of the system.

By
addi
ng more physical brokers to store and route messages, a greater number of pr
o-
ducers or modellers can be supported.

This allows for a truly distributed system,
where modelling is done on different physical machines throughout the network.

Third, the semanti
c querying and decoupling between modellers and producers allows
for the dynamic addition of arbitrary numbers of both types of application to the
MUMS system.

Once these entities have joined in the system, their participation can
increase the expressivene
ss of the user models created, without requiring modific
a-
tions to existing producers and modellers.

Finally, the logical centralization the br
o
ker
allows for the setting of administration policies, such as privacy rules and the maint
e-
nance of data integrit
y, through the addition of filters.

All of these benefits address key challenges for adaptive learning systems.

These
sy
s
tems must allow for the integration of both existing domain applications as well as
lear
n
ing management specific applications.

This int
egration must be able to take place
with a minimal amount of effort to accommodate the various stakeholders within an
institution (e.g. administrators, researchers, instructional designers), and must be able
to be centrally managed to provide for privacy o
f user data.

Last, the system must be
able to scale not just to the size of a single classroom, but to the needs of a whole
department or institution.

3. Implementation Prototype

The MUMS architecture is currently being prototyped within the distributed e
-
learning environment in the Department of Computer Science at the University of
Saskatchewan.

This environment has been created over a number of years and i
n-
volves applications from a variety of different research projects.

While initially aimed
at garneri
ng research data, these applications are all currently deployed in a support
fashion within some of our computer science courses.

There are four main applic
a-
tions:


1.

A content delivery system, which deploys IMS content packaging
[15]

formatted
lear
n
ing objects to students using a web browser.

2.

A web based forum discussion system built around the notions of peer help (I
-
Help
Public Discussions
[11]
).

3.

A
n instant messaging and chat application

(I
-
Help Instant Me
s
senger).

4.

A quizzing system which deploys IMS QTILite
[14]

formatted quizzes and rec
ords
evalu
a
tions of students


We are currently in the process of adding new applications to this list.

These applic
a-
tions include development environments, e
-
learning portals, and web browsers.

Each
of these systems
contribute

to and benefit from models of

the user and hence require a
flexible user modelling environment.

To accommodate this, we have impl
e
mented the
MUMS architecture with three clear goals in mind: scalability, interoper
a
bility and
extensibility.

The technical solutions we are using to achie
ve these goals will be a
d-
dressed in turn.

6

Christopher Brooks1,

Mike Winter1, Jim Greer2, and Gordon McCalla2


3.1 Interoperability

With the goal of distributing the system to as many domain specific applications as is
necessary, interoperability is a key concern.

To this end, all opinion publishing from
pr
o
ducers is done u
sing our implementation of the Web Services Events (WS
-
Events)
[5]

infrastructure specification.

This infrastru
c
ture defines a set of data types and rules
for pass
ing events using web services.

These rules include administr
a
tive information
about the producer or modeller (e.g. contact information, quality of service, etc), a
payload that contains the semantics of the opi
n
ion, and information on managing
advertising
and subscriptions.

Using this infrastru
c
ture helps to protect entities from
future changes in the way opinion distribution is handled.

Further, modellers can
either subscribe to events using WS
-
Events (pu
b
lish/subscribe), or can query the
broker directly u
sing standard web service technol
o
gies (query/response).

This allows
for both the real
-
time delivery of new modelling information, as well as the ability to
access archived i
n
formation from the past in a manner independent of platform and
programming la
n
gu
age.

We enhance semantic interoperability by expressing the payload of each event u
s-
ing the Resource Description Framework (RDF)
[16]
.

This language provides a na
t
u-
rally exte
n
sible and ontology
-
neutral method for describing modeling information in a
format that is easily computer readable.

It has become the lingua franca of the sema
n-
tic web, and a number of toolkits (notably, Jena
[13]

and Drive
[24]
) have arisen to
make RDF graph manip
u
lation easier.

When registeri
ng for events, modellers provide
pa
t
terns to match using the RDF Data Query Language (RDQL)
[23]
.

Finally, design time interoperability is achieved by maintaining
a separate ontology
d
a
tabase which authors can inspect when creating new system components.

This
encourages the reuse of previously deployed ontologies, while maintaining the flex
i-
bility of opinion routing independent of ontology.

3.2 Extensibility

Besides

the natural extensibility afforded by the use of the RDF as a payload fo
r
mat,
the MUMS architecture provides for distributed reasoning through the use of filters.

In ge
n
eral, a filter is a component that masquerades as any combination of producer,
modelle
r, or broker of events.

There are at least two specialized instances of a filter:


1.

Reasoners: register or query for events with the goal of being able to produce hig
h-
er level derivative events.

For instance, one might create a reasoner to listen for
events

related to message sending from the web
-
based discussion and instant me
s-
senger producers, and then create new opinions which indicate the changing social
structure amongst peers in the class.

2.

Blockers: are placed between producers and modellers with the g
oal of modifying
or restricting events that are published.

Privacy filters are an example of a blocker.

These filters can anonymize events or require that a modeller provide special a
u-
thentication privileges when subscribing.


The Massive User Modelling System (MUMS)

7


While the system components i
n our current implementation follow a clear separation
between those that are producers and consumers of information, we expect most f
u-
ture components will aim at value adding the network by reasoning over data sources
before producing opinions. Thus we i
magine that the majority of the network will be
made up of reasoner filters chained together with a few blockers to implement admi
n-
istrative policies.

3.3 Scalability

Early lessons learned from testing the implementation prototype indicated that there
are
two main factors involved in slowing down the propagation of opinions:


1.

Message serialization: The deserialization of SOAP messages into native data
types is an expensive process.

This process is especially important to the broker,
which shares a many
-
to
-
o
ne relationship with producers.

2.

Subscription evaluation: Evaluating RDF models against a RDQL query is a time
co
n
suming operation.

This operation grows with the complexity of the models, the
complexity of the query, and the number of queries (number of mod
eller registr
a-
tions) that a broker has.


To counteract this, the MUMS architecture can be extended to include the notion of
d
o
main brokers.

A domain broker is a broker that is ontology aware, and can provide
e
n
hanced quality of service because of this awar
eness.

This quality of service usually
comes in the form of more efficient model storage, and thus faster query res
o
lution.

Further, brokers are free to provide alternative transport mechanisms which may lead
to faster data transfers (e.g. a binary protoco
l which compresses RDF messages could
be used for mobile clients with error
-
prone connections, while a UDP protocol d
e-
scribing RDF using N
-
Triples
[10]

could be used

to provide for the more real
-
time
delivery of events).

The use of domain brokers can be combined with re
a
soners and
blockers to meet the performance, management, and expressability r
e
quirements of
the system.

Finally, the architectural notion of a broker
as a centralized entity is a logical n
o-
tion only.

Physically we distribute the load of the broker amongst a small cluster of
machines
connected to a single data store to maintain inte
g
rity.

An overview of the prototype, including the technologies in use,
is presented in
Fig
.

2.

Evidence producers are written in a variety of languages, including a Java
producer for the course delivery system, a C# producer for the public discussion sy
s-
tem and a C++ pr
o
ducer (in the works) for the Mozilla web browser.

The br
oker is
realized through a cluster of Tomcat web application servers running an Apache Axis
application which manage subscriptions and semantic routing.

This application uses a
PostreSQL database to store both subscription and archival information. Subscri
ptions
are stored as a tuple indicating the RDQL pattern that should be matched, and the
URL at which the modeller can be contacted.

At this moment there is one Java
-
based
modeller which graphically displays aggregate student information for instructors
fr
om the I
-
Help public forums.

Besides a description of student posting frequency,
this modeller displays statistics for a whole f
o
rum, as well as a graphical picture of
8

Christopher Brooks1,

Mike Winter1, Jim Greer2, and Gordon McCalla2


student interaction.

In addition there are two other applications under development
incl
uding a pedagogical content planner and a peer help matchmaker.


Fig.
2
.

Prototype implementation of the MUMS architecture

4. Related Work

While inspired by the needs for distributed intelligent tutoring systems, we see this
work overlappi
ng three distinct fields of computer science; distributed computing, the
semantic web, and learner modelling.

Related research in each of these fields will be
addressed in turn.

The distributed systems field is a mature field that has provided a catalyst f
or much
of our work.

Both general and specific kinds of event systems are described throug
h-
out the literature, and a number of mature specifications, such as the Java RMI and
the CORBA, exist.

Unlike MUMS, these event systems require the consumers of
event
s (modellers) to subscribe to events (opinions) based on the expected event pr
o-
ducer or the channel (su
b
ject) the events will arrive on.

This increases the coupling
between entities in the system, requiring that either the consumer is aware of a given
prod
ucer, or that they share a strict messaging ontology.

In
[4]
, Carzaniga et al. d
e-
scribe a model for content
-
based addres
s
ing and routing at the network level.

W
e
build upon this model by applying similar principles in the application layer, allowing
the modellers of opi
n
ions to register for those opinions which match some semantic
pattern.

This allows for the ad hoc cre
a
tion and removal of both evidence producers

and modellers within the system.

While the semantic web as a research area has been growing quickly for a number
of years, the focus of this area has been on creating formalisms for knowledge ma
n-
The Massive User Modelling System (MUMS)

9


agement representation.

The general approach with sharing da
ta over the semantic
web is to co
n
sider it just “an extension of the current web”
[2]
, and to follow a qu
e-
ry/response comm
u
nication model.

Thus, a fair amount of

work has been done in
conjunction with database research to produce efficient mechanisms for sto
r
ing (e.g.
[22]

[12]
) and querying data (e.g.
[23]
), b
ut new methods for transmi
t
ting this data
have largely been unexplored.

For instance, the HP Joseki project
[1]

and the Nokia
URI Query Agent Model
[19]
provide methods for publishing, updating, and retrie
v-
ing RDF data models using HTTP.

This approach is useful for large centralized mo
d-
els where data transfer uses more r
esources than data querying; however, it provides
poor support for the real
-
time delivery of mode
l
ing information.

Further, it supports
the notion of a single model per user which is formed through consensus between
producers, as opposed to the more lightw
eight situated user modeling su
g
gested by
active modeling researchers.

We instead provide a method which completely deco
u-
p
les producers from one another, and offload the work in forming user modellers to
the consumers of opinions.

The work done by Nejdl et

al. in
[18]

and Dolog and Nejdl in
[7]

and
[6]

marries
the idea of the semantic web with learner modelling.

In these works the authors d
e-
scribe a network of learning materials set up in a peer
-
to
-
peer fashion.

Resources are
described in RDF using both gene
ral pedagogical metadata (in partic
u
lar the IEEE
Learning Object Metadata specification) and learner specific metadata (such as the
IMS LIPS or PAPI).

The network is searchable by end
-
users through the use of pe
r-
sonal learning assistants who can query peer
s in the network for learning resource
metadata, then filter the results based on a user model.

While this architecture distri
b-
utes the responsibility for user mo
d
eling, it also limits communication to the qu
e-
ry/response model.

Thus, pe
r
sonal learning agen
ts must continually query data
sources to discover new inform
a
tion about the student they are modelling.

In addition,
by arranging data sources in a peer network the sy
s
tem loses its ability to effectively
centrally control these sources.

For instance, an
institution would need to control all
of the peers in the network to provide for data integrity or pr
i
vacy over the data being
shared.

5. Conclusions and Future Work

As cited by Picard et al., the ITS working group of 1995 described tutoring systems
as:



…hand
-
crafted, monolithic, standalone applications.

They are time
-
consuming
and costly to design, implement, and deploy.

Each development teams must red
e-
velop all of the component functionalities needed.

Because these components are
so hard and costly to b
uild, few tutors of realistic depth and breadth ever get built,
and even fewer ever get tested on real students.”
[21]


Despite research invested into providing agent based architectures for tutoring sy
s-
tems, tutors remain largely centralized in deploy
ment.

These tutors are generally
domain specific, and are unable to easily interface with the various legacy applic
a-
10

Christopher Brooks1,

Mike Winter1, Jim Greer2, and Gordon McCalla2


tions that students may be using to augment their learning.

When such interfacing is
available, it comes with a high cost to designers, as i
ntegration requires both a shared
ontology to describe what the student has done, as well as considerable low level
software integration work. MUMS provides an alternative architecture where produ
c-
ers can be readily associated with legacy applications and
where modellers and re
a-
soners can readily produce useful learner mode
l
ling information.

This paper has presented both a framework and a prototype to support the just
-
in
-
time production and delivery of user modelling information.

It provides a general
archi
tecture for e
-
learning applications to share user data, as well as details on a sp
e-
cific implement
a
tion for this architecture, which builds on technologies being used
within the web services and semantic web communities.

It provides an approach to
student
modelling that is pla
t
form, language, and ontology independent.

Further, this
approach allows for both the just
-
in
-
time delivery of modelling information, as well
as the archival and retrieval of past modelling opinions.

Our immediate future work involves
further integration of domain specific applic
a-
tions within this framework.

We will use this new domain specific information to
provide for more accurate resource suggestions to the learner, including both the
acquisition of lear
n
ing objects from learning o
bject repositories as well as expertise
location through peer matchmaking.

Tangential to this, we are interested in pursuing
the use of user defined filters through personal learning agents.

These agents can act
as a “front
-
end” for the learner to have inp
ut over the control and dissemination rights
of their learner information.

Finally, we are examining the issue of design time i
n-
te
r
operability through ontology sharing using the Web Ontology Language (OWL).


Acknowledgements.

We would like to thank the rev
iewers for their valuable reco
m-
me
n
dations.
This work has been conducted with support from a grant funded by the
Natural Science and Engineering Research Council of Canada (NSERC) for the
Learning Object Repositories Network (LO
R
NET).

References

1.

Joseki
-

The Jena RDF Server. Available online at http://www.joseki.org/. Last accessed
March 22, 2004.

2.

Berners
-
Lee, T., Hendler, J., and Lassila, O., The Semantic Web
Scientific American
,
May, 2001. Scientific American, Inc.

3.

Blackboard Inc. blackboard. Av
ailable online at http://www.blackboard.com/. Last a
c-
cessed March 22, 2004.

4.

Carzanigay, A., Rosenblumz, D. S., and Wolfy, A. L. Content
-
Based Addressing and
Routing: A General Model and its Application. In
Technical Report CU
-
CS
-
902
-
00

.

5.

Catania, N.,

et al. Web Services Events (WS
-
Events) Version 2.0. Available online at
http://devresource.hp.com/drc/specifications/wsmf/WS
-
Events.pdf. Last accessed March
22, 2004.

6.

Dolog, P. and Nejdl, W. Challenges and Benefits of the Semantic Web for User Mode
l-
li
ng. In
Workshop on Adaptive Hypermedia and Adaptive Web
-
Based Systems 2003, Held
at WWW 2003
.

7.

Dolog, P. and Nejdl, W. Personalisation in Elena: How to cope with personalisation in
distributed eLearning Networks. In
International Conference on Worldwide
Coherent
The Massive User Modelling System (MUMS)

11


Workforce, Satisfied Users
-

New Services For Scientific Information
. Oldenburg, Ge
r
m
a-
ny.

8.

Gamma, E., Helm, R., Johnson, R., and Vlissides, J. (eds)
Design Patterns
, 1st edition.
Addison
-
Wesley, 1995.

9.

Gaudioso, E. and Boticario, J. G. Towards

Web
-
based Adaptive Learning Communities.
In
Artificial Intelligence in Education 2003
.

10.

Grant, J. and Beckett, D. RDF Test Cases. Available online at
http://www.w3.org/TR/2004/REC
-
rdf
-
testcases
-
20040210/. Last accessed March 22,
2004.

11.

Greer, J., Mc
Calla, G., Vassileva, J., Deters, R., Bull, S., and Kettel, L. Lessons Learned
in Deploying a Multi
-
Agent Learning Support System: The I
-
Help Experience. In
Artif
i-
cial Intelligence in Education 2001
. San Antonio, TX, USA.

12.

Harris, S. and Gibbins, N. 3s
tore: Efficient Bulk RDF Storage. In
Workshop on Semantic
Web Storage and Retrieval 2003
. Vrije Universiteit, Amsterdam, Netherlands.

13.

Hewlett
-
Packard Development Company, L.P. Jena 2
-

A Semantic Web Framework.
Available online at http://www.hpl.hp.com
/semweb/jena.htm. Last accessed March 22,
2004.

14.

IMS Global Learning Consortium Inc. IMS Question & Test Interoperability Lite Specif
i-
cation, Version 1.2. 2002.

15.

IMS Global Learning Consortium Inc. IMS Content Packaging Specification version
1.1.3. 2
003.

16.

Klyne, G. and Carroll, J. J. Resource Description Framework (RDF): Concepts and A
b-
stract Syntax. Available online at http://www.w3.org/TR/2004/REC
-
rdf
-
concepts
-
20040210/. Last accessed March 22, 2004 .

17.

McCalla, G., Vassileva, J., Greer, J., an
d Bull, S. Active Learner Modelling.

18.

Nejdl , W., Wolpers, M., Siberski, W., Schmitz, C., Schlosser, M., Brunkhorst, I., and
Lser, A. Super
-
peer
-
based routing and clustering strategies for rdf
-
based peer
-
to
-
peer ne
t-
works. In
12th International World Wid
e Web Conference
. Budapest, Hungary.

19.

Nokia. URIQA: The Nokia URI Query Agent Model. Available online at
http://sw.nokia.com/uriqa/URIQA.html. Last accessed March 22, 2004.

20.

Object Management Group. Common Object Request Broker Architecture (CO
R-
BA/II
OP).

21.

Picard, R. W., Kort, B., and Reilly, R. Affective Learning Companion Project Summary:
Exploring the Role of Emotion in Propelling the SMET Learning Process. Available
online at http://affect.media.mit.edu/AC_research/lc/nsf1.html.

22.

Reggiori,
A., van Gulik, D.
-
W., and Bjelogrlic, Z. Indexing and retrieving Semantic Web
resources: the RDFStore model. In
Workshop on Semantic Web Storage and Retrieval
2003
. Vrije Universiteit, Amsterdam, Netherlands.

23.

Seaborne, Andy. RDQL
-

A Query Language for

RDF: W3C Member Submission.

24.

Singh, R. Drive
-

An RDF Parser for the .NET Platform. Available online at
http://www.driverdf.org/. Last accessed March 22, 2004.

25.

Sun Microsystems, Inc. Jini Technology Core Platform Specification.

26.

uPortal. uPort
al by JA
-
SIG. Available online at http://uportal.org/. Last accessed March
22, 2004.

27.

Vassileva, J., McCalla, G., and Greer, J. Multi
-
Agent Multi
-
User Modeling in I
-
Help.
User Modeling and User
-
Adapted Interaction: Special Issue on User Modelling and In
te
l-
ligent Agents
, 13(1):1
-
31, 2002

28.

WebCT. WebCT.com. Available online at http://www.webct.com/. Last accessed March
22, 2004.




12

Christopher Brooks1,

Mike Winter1,
Jim Greer2, and Gordon McCalla2