Modeling Student Knowledge

ocelotgiantAI and Robotics

Nov 7, 2013 (3 years and 9 months ago)

72 views

Modeling Student Knowledge

Using Bayesian Networks to Predict Student Performance

By Zach Pardos, Neil Heffernan,
Brigham Anderson and Cristina Heffernan

To evaluate the predictive performance of
various fine
-
grained student skill models
in the ASSISTment tutoring system using
Bayesian networks.

Goal



ASSISTment is a web
-
based assessment system for
8
th
-
10
th

grade math that tutors students on items they
get wrong. There are 1,443 items in the system.



The system is freely available at www.assistment.org



Question responses from 600 students using the
system during the 2004
-
2005 school year were used.


Each student completed around 260 items each.

The Skill Models

The skill models were created for use in the online
tutoring system called ASSISTment, founded at WPI.
They consist of skill names and associations (or
tagging) of those skill names with math questions on
the system. Models with 1, 5, 39 and 106 skills were
evaluated to represent varying degrees of concept
generality. The skill models’ ability to predict
performance of students on the system as well as on a
standardized state test was evaluated.


The five skill models used:



WPI
-
106:

106 skill names were drafted and tagged
to items in the tutoring system and to the questions on
the state test by our subject matter expert, Cristina.



WPI
-
5

and
WPI
-
39:

5 and 39 skill names drafted
by the Massachusetts Department of Education.



WPI
-
1:

Represents unidimensional assessment.

Background on ASSISTment

Predicting student responses within the
ASSISTment tutoring system


The ASSISTment fine
-
grained skill models excel at
assessment of student skills (see Ming Feng’s poster
for a Mixed
-
Effects approach comparison)


Accurate prediction means teachers can know when
their students have attained certain competencies.

1.

Skill probabilities are inferred from a student’s responses to
questions on the system

Bayesian Belief Network

Student Test Score Prediction Process

This work has been accepted for publication at the 2007 User Modeling Conference in Corfu, Greece.

Sponsors

Collaborators

Online Data Prediction Error
0
5
10
15
20
25
30
WPI-1
WPI-5
WPI-39
WPI-106
Skill Models
Average Error %
MCAS Test Prediction Error
0
5
10
15
20
25
30
WPI-1
WPI-5
WPI-39
WPI-106
Skill Models
Average Error %
Result:

The finer
-
grained the model, the better prediction
accuracy. The finest
-
grained WPI
-
106 performed
the best with an average of only 5.5% error in
prediction of student answers within the system.

Result:

The finest
-
grained model, the WPI
-
106, came in 2
nd

to the WPI
-
39 which may have performed better
than the 106 because 50% of its skills are sampled
on the MCAS Test vs. only 25% of the WPI
-
106’s.

Predicting student state test scores

Conclusions



A Bayesian Network is a probabilistic machine
learning method. It is well suited for making predictions
about unobserved variables by incorporating prior
probabilities with new evidence.

Bayesian Networks



Arrows represent associations of skills with question items. They also represent
conditional dependence in the Bayesian Belief Network.


Probability of Guess is set to 10% (tutor questions are fill in the blank)


Probability of getting the item wrong even if the student knows it is set to 5%


2. Inferred skill probabilities from above are used to predict the
probability the student will answer each test question correctly


Probabilities are summed to generate total test score.


Probability of Guess is set to 25% (MCAS questions are multiple choice)


Probability of getting the item wrong even if the student knows it is set to 5%