Presentation Slides - School of Computing

boorishadamantAI and Robotics

Oct 29, 2013 (4 years and 10 days ago)

60 views

School of something

FACULTY OF OTHER

School of Computing

FACULTY OF ENGINEERING

Evaluation

Kleanthous Styliani

www.comp.leeds.ac.uk/stellak

Overview



Common Evaluation Approaches


Before Planning an Evaluation


Evaluation for this PhD

School of Computing

FACULTY OF ENGINEERING

Evaluation approaches for Intelligent Systems



Formative & Summative Evaluation




Layered Evaluation (Specific to Adaptive Systems)




Simulations




Control Groups

School of Computing

FACULTY OF ENGINEERING


Simulations:
preferred when you need large amounts of data or data
is too expensive to collect or when people have to be involved and there
is no available sample. (e.g. P2P Communities, Social Networks)



Control Groups:
allow to different samples to use a system with and
without the intelligent functionality and measure which group did best or
according to what they are evaluating. (Comtella)

(
negative:
the non
-
intelligent version cannot be optimal in any way if the
system built with intelligent functionality at the beginning )



School of Computing

FACULTY OF ENGINEERING

Formative:

the system should be evaluated for its usability and
effectiveness in the early stages.

Summative:

the effectiveness of the system is determined in real
environments after development or completion of a major stage.


Layered approach:


Layer 1


Interaction Assessment Evaluation


e.g. Are the user’s characteristics being successfully detected by
the system and maintained in the user model?


Layer 2


Adaptation Decision Making Evaluation


e.g. Are the adaptation decisions valid and meaningful for
selected assessment results?

School of Computing

FACULTY OF ENGINEERING

School of Computing

FACULTY OF ENGINEERING

Before planning the evaluation:


What are you evaluating?



What are your research questions/hypothesis?



How will you test the above with the evaluation?



Plan the evaluation

What are you evaluating?



Extraction of community model



Pattern detections



Evolution algorithms



Advantages of interventions


School of Computing

FACULTY OF ENGINEERING

What are your research questions?


Formative Evaluation


Do the CM algorithms work as are intended to work?



Do the pattern detection algorithms extract the correct
patterns?



Do the evolution algorithms pick the changes happen through
time?


School of Computing

FACULTY OF ENGINEERING


Summative Evaluation

Structured specifically for this framework with specific questions to be
answered


Suitability of interventions:


How do members evaluate the interventions?


Benefits for users:


Do the users find the interventions helpful in


Identifying people relevant to them?


Identifying resources relevant to them?


Become aware of who is working on what?


Identify potential collaborators?


School of Computing

FACULTY OF ENGINEERING


Benefits for the community:


Newcomers


How quickly are they integrated in the VC?


Oldtimers


How active they are?


Transactive Memory


Do they know who to ask or where to find resources for topic A?


Do they know what other members know in the VC?


To whom is your knowledge important?



School of Computing

FACULTY OF ENGINEERING


Benefits for the community:


Shared Mental Models


What others are doing in this VC?


What is the purpose of this VC?


Cognitive Centrality


Who shares the most valuable resources in this community?



Have the centrality shifted more effectively between members?


Any peripheral members became cognitively central?

School of Computing

FACULTY OF ENGINEERING

Where to evaluate?


BSCW Data of Semantic Web VC (
Formative
)


AWESOME Simulated Data (
Formative
) & (prove
generality of approach)


Active VC to evaluate the whole framework & focusing
on the intelligent interventions (
Summative
)



Qualitative (questionnaires)



Quantitative (statistics)


School of Computing

FACULTY OF ENGINEERING

Why am I telling you these things?


BSCW VC for our group


I need your contribution to complete the major part
of evaluation


Summative Evaluation



Share some resources with the others


Download resources that might interest you


Try to follow the guidelines given

School of Computing

FACULTY OF ENGINEERING


Thank you!

School of Computing

FACULTY OF ENGINEERING



Mark, M. & Greer, J. (1993). Evaluation Methodologies for Intelligent Tutoring
Systems.
Journal of Artificial Intelligence and Education
, 4 (2/3), pp. 129
-
153



Karagiannidis, C. and D.G. Sampson.
Layered Evaluation of Adaptive
Applications and Services
. in

International Conference on Adaptive Hypermedia
and Adaptive Web
-
Based Systems
. 2000 Springer
-
Verlag.



Millan, E. and J. Perez
-
De
-
La
-
Cruz,
A Bayesian Diagnostic Algorithm for
Student Modeling and its Evaluation.

User Modeling and User
-
Adapted
Interaction, 2002. 12(2): p. 281
-
330.



Shlomo, B., et al.
Evaluating User Model Effectiveness by Simulation
. in
Workshop on Personalized Access on Cultural Heritage at 11th international
conference on user modeling
. 2007.



Cheng R., Vassileva J. (2006) Design and Evaluation of an Adaptive Incentive
Mechanism for Sustained Educational Online Communities.
User Modelling and
User
-
Adapted Interaction
, 16 (2/3), 321
-
348. (special issue on User Modelling
Supporting Collaboration and Online Communities).

School of Computing

FACULTY OF ENGINEERING