Cristina
Conati
Department of Computer Science
University of British
Columbia
Beyond Problem

solving:
Student

adaptive Interactive Simulations
for Math and Science
Overview
Motivations
Challenges of devising student

adaptive simulations
Two examples of how we target these challenges
–
ACE: interactive simulation for mathematical functions
–
CSP Applet:
interactive simulation for AI algorithm
Conclusions and Future work
Intelligent Tutoring Systems (ITS)
Create computer

based tools that support
individual
learners
By
autonomously
and
intelligently
adapting to their specific
needs
Student
Model
Tutor
Domain
Model
Adaptive
Interventions
ITS Achievements
In the last 20 years, there have been many successful
initiatives in devising Intelligent Tutoring Systems
(Woolf 2009, Building Intelligent Interactive Tutors, Morgan Kaufman)
Mainly ITS that provide individualized support to problem
solving through tutor

lead interaction (
coached problem
solving
)
–
Well defined problem solutions => guidance on problem solving
steps
–
Clear definition of correctness => basis for feedback
Beyond Coached Problem Solving
Coached problem solving is a very important component of
learning
Other forms of instruction, however, can help learners
acquire the target skills and abilities
–
At different stages of the learning process
–
For learners with specific needs and preferences
Our Goal
: Extend ITS to other learning activities that
support student initiative and engagement:
–
Interactive Simulations
–
Educational Games
Overview
Motivations
Challenges of devising student

adaptive simulations
Two examples of how we target these challenges
–
ACE: interactive simulation for mathematical functions
–
CSP Applet:
interactive simulation for AI algorithm
Conclusions and Future work
Challenges
Activities more open

ended and less well

defined than
pure problem solving
–
No clear definition of correct/successful behavior
Different user states to be captured (meta

cognitive,
affective) in order to provide good tutorial interventions
–
difficult to assess unobtrusively from interaction events
How to model what the student is doing?
How to provide feedback that fosters learning while
maintaining student initiative and engagement?
Our
Approach
Student models based on formal methods for
probabilistic reasoning and machine learning
Increase
information available to student model through
innovative input
devices:
–
e.g
.
eye

tracking and physiological
sensors
Iterative model design and evaluation
Overview
Motivations
Challenges of devising student

adaptive simulations
Two examples of how we target these challenges
–
ACE: interactive simulation for mathematical functions
–
CSP Applet:
interactive simulation for AI algorithm
Conclusions and Future work
ACE: Adaptive Coach for Exploration
Activities organized into units to explore mathematical
functions (e.g. input/ouput, equation/plot)
Probabilistic student model that captures student
exploratory behavior and other relevant traits
Tutoring agent that generates tailored suggestions to
improve student exploration/learning when necessary
(Bunt, Conati, Hugget, Muldner, AIED 2001)
Adaptive Coach for Exploration
EDM 2010
11
12
Adaptive Coach for Exploration
13
Adaptive Coach for Exploration
Before you leave
this exercise, why don’t you
try scaling the function by a large negative
value?
Think about how this will affect
the plot
ACE Student Model
(Bunt and
Conati
2002)
Knowledge
Individual
Exploration Cases
Exploration
of Exercises
Exploration
Categories
Exploration
of Units
Iterative
process of design and evaluation
Probabilistic model of how individual exploration actions
influence exploration and understanding of exercises and concepts
e.g. (in Plot unit)
•
positive/negative slope
•
positive/negative intercept
•
large/small, positive/negative exponents…
Modeling Student Exploration
Our first attempt (Bunt and
Conati
, 2002)
Learning
Student Model
Number and Coverage
of Exploratory Actions, e.g.
•
Positive/negative Y

Intercept
•
Odd/Even, Positive Negative Exponent....
Interface Actions
Preliminary Evaluation
Quasi

experimental design with 13 participants using
ACE
(Bunt and
Conati
2002)
–
The more exercises were effectively explored according to
the student model, the more the students improved
–
The more hints students followed, the more they learned
Because the model only considers coverage of student
actions, it can overestimate student exploration
Need to consider whether the student is reasoning about
the effects of his/her actions
–
Self

explanation
meta

cognitive skill:
Revised User Model
(Bunt,
Muldner
and Conati, ITS2004;
Merten
and Conati, Knowledge Based Systems 2007)
Learning
Student Model
Number and coverage of student actions
Self

explanation of
action outcomes
Time between actions
Gaze Shifts in Plot Unit
Interface Actions
Input from eye

tracker
Results on Accuracy
We evaluated the complete model against
–
The original model with no self

explanation
–
A model that uses only time in between actions as evidence of self

explanation
50
60
70
80
Accuracy on SE
Accuracy on
Learning
No SE
SE (Time)
SE (Time + Gaze)
What’s Next (1)
Test adaptive interventions to trigger self

explanation
(
Conati
2011)
Discussion
ACE work provided evidence that
•
It is possible to track more “open ended” students’
behaviors
than structured problem solving
•
eye

tracking can support the process
However, hand

coding the relevant behaviors, as we did
for ACE (
knowledge

based approach
)
•
is time consuming
•
likely to miss other, less intuitive patterns of interaction
related to learning (or lack thereof)
Alternative Approach
(
Amershi
and
Conati
2009,
Kardan
and
Conati
2011)
Behavior Discovery Via Data Mining
Association
Rules
Mining
Clustering
Actions Logs
Other Data
Fe
atu
re
Ve
cto
rs
Vector of Interaction
Features

Frequency Of Actions

Latency Between
Actions
……………
Extract
rules describing
distinguishing patterns in
each cluster
Groups
together students
that have similar
interaction behaviors
Interpret in terms of
learning
•
Experts
•
Performance
Measure(s)
Overview
Motivations
Challenges of devising student

adaptive simulations
Two examples of how we target these challenges
–
ACE: interactive simulation for mathematical functions
–
CSP Applet:
interactive simulation for AI algorithm
Conclusions and Future work
Tested with AI Space CSP applet
AISpace
(
Amershi
et al., 2007)
–
set
of
applets implementing interactive simulations of
common
Artificial Intelligence
algorithms
–
Used regularly in our AI courses
–
Google “
AISpace
” if you want to try it out
Applet for Constraint Satisfaction problems (CSP),
visualizes the working of the AC3 algorithm
27
AISpace
CSP Applet
Direct Arc
Clicking
User Study
(
Kardan
and
Conati
2011)
65 subjects
–
Read intro material on the AC

3 algorithm
–
Pre test
–
Use CSP applet on two problems
–
Post test
13,078 actions
More than 17
hours of interaction
Dataset
Features:
–
frequencies of use for each action
–
pause duration between actions (Mean and SD)
–
7 actions
㈱e慴ures
Performance measure for validation
–
Learning Gain from pretest to posttest
Feature
vectors
Clustering
Behavior Discovery
Rule Mining
Found 2 clusters
Statistically significant
difference in Learning
Gains (LG)
–
High Learners (HL) and
Low Learners (LL) clusters
3
2
Feature
vectors
Clustering
Behavior Discovery
Rule Mining
Clustering
Usefulness:
Sample Rules
HL members:
Use
Direct Arc Click
action very frequently (
R
1
).
HL
cluster
:
R
1
:
Direct
Arc
Click
frequency
=
Highest
(Conf
=
100
%
,
Class
Cov
=
100
%
)
LL
cluster
:
R2
:
Direct Arc Click Pause
Avg
= Lowest (Conf =100%, Class
Cov
= 100%)
R3
:
Direct Arc Click frequency
= Lowest (Conf = 93%, Class
Cov
=93.5%)
3
3
LL members:
Use
Direct Arc Click
sparsely (
R3
)
Leave l
ittle
time between a
Direct Arc Click
and the next action
(
R2
)
Feature
vectors
Clustering
Behavior Discovery
Rule Mining
Great, but what do we do with this?
We can use the learned clusters and rules to classify a new
student based on her behaviors
Use detected behaviours for adaptive support
–
Promoting the behaviours conducive of learning
–
Discouraging/preventing detrimental behaviours
3
4
The User Modeling Framework
3
5
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features
If
user is
a LL
and
uses
Direct Arc Click
very
infrequently
(
R
3
)
Then
prompt this
action
If
user is
a LL
and
pauses
very briefly after a
Direct Arc Click
(
R2
)
Then
take action to slow
her down
Classifier Evaluation
Leave

one

out Cross Validation on dataset of 64 users
For each user
u
in dataset
1.
Remove user
u
2.
do
Behaviour
Discovery on the remaining 63
3.
for each of
u
’s
actions:
»
Calculate the feature vector
u
v
»
Classify
u
v
»
Compare with
u
’s
original label
Accuracy as a function of observed actions
Discussion
User modeling framework for open

ended and
unstructured interactions
–
Relevant behaviours are discovered via data mining
techniques instead being hand

crafted
Very encouraging results with CSP applet
–
Detected clusters represent groups with different
learning gains
–
Online classifier:
good accuracy soon enough to generate
adaptive interventions
–
These interventions can be derived from the generated
rules
Current Work
Applying the discovered rules to generate the adaptive
version of the CSP applet
Adding eye

tracking input to the dataset
Conclusions
Research on devising student

adaptive didactic support
for exploratory activities beyond problem solving
Interactive simulations
Challenges in modeling interactions with no clear
structure or definition of correctness
Student modeling approaches based on probabilistic
techniques and unsupervised machine learning
very promising results
Shown how eye

tracking can help!
We are also exploring it in relation to assessing engagement and
attention in educational games (Muir and
Conati
2011)
Comments 0
Log in to post a comment