Using Data-Driven Discovery Techniques for the Design and Improvement of Educational Systems

odecrackAI and Robotics

Oct 29, 2013 (4 years and 15 days ago)

133 views

Using Data
-
Driven Discovery Techniques for the Design
and Improvement of Educational Systems



John Stamper

Pittsburgh Science of Learning Center

Human
-
Computer Interaction Institute

Carnegie Mellon University



4/8/2013

The Classroom of the Future

Which picture represents the
“Classroom of the Future”?

2

3

The Classroom of the Future

The answer is both!

Depends of how much money you have...



… but maybe not what you think…


4

The Classroom of the Future

Rich vs. Poor


Poor kids will be forced to rely on “cheap” technology


Rich kids will have access to “expensive” teachers


We are seeing this today!


Waldorf school in Silicon Valley


no technology


NGLC Wave III Grants


MOOCs


Growth of adaptive technology companies


Online instruction


… and more…

5

What does this mean?

My view is that we cannot stop this, I believe we
must accept that economics will force this route.


We should focus on improving learning technology


New ways to improve teacher
-
student access


Add more adaptive features to learning software


Adaptive learning, at scale, using data!

6

Educational Data Mining


“Educational Data Mining is an emerging
discipline, concerned with developing
methods for exploring the unique types of
data that come from educational settings, and
using those methods to better understand
students, and the settings which they learn
in.”


www.educationaldatamining.org

Types of EDM methods

(Baker &
Yacef
, 2009)


Prediction


Classification


Regression


Density estimation


Clustering


Relationship mining


Association rule mining


Correlation mining


Sequential pattern mining


Causal data mining


Distillation of data for human judgment


Discovery with models

7

Emerging Communities


Society for Learning Analytics Research


First conference: LAK2011



International Educational Data Mining Society


First conference: EDM2008


Publishing JEDM since 2009



Plus an emerging number of great people
working in this area who are (not yet) closely
affiliated with either community

Emerging Communities


Joint goal of exploring the “big data” now
available on learners and learning



To promote


New scientific discoveries & to advance learning
sciences


Better assessment of learners along multiple
dimensions


Social, cognitive, emotional, meta
-
cognitive, etc.


Individual, group, institutional, etc.


Better real
-
time support for learners

EDM Methods to discuss


Prediction


understand what the student
knows


Discovery with models


improve
understanding of the structure of knowledge

10

LearnLab

Pittsburgh Science of Learning Center (PSLC)


Created to bridge the
Chasm

between science &
practice


Low success rate (<10%) of randomized field trials


LearnLab

= a socio
-
technical bridge between lab
psychology & schools


E
-
science of learning & education


Social processes for research
-
practice engagement


Purpose
:
Leverage cognitive theory and computational
modeling to identify the conditions that cause robust student
learning

11

Chemistry Virtual Lab

Algebra

Cognitive Tutor

Ed tech + wide use =

Research
in practice


=

LearnLab
: Data
-
driven improvement
infrastructure


2004
-
14, ~$50 million


Tech enhanced courses,
assessment, & research


School cooperation


In vivo
experiments

+

English

Grammar Tutor

Educational Games

Interaction data is

surprisingly revealing


Accurate assessment
during learning


Detect student

work ethic,
engagement



Discover better
models of what

is hard to learn

R = .82

Online interactions
=> state tests

Learning Curve
Analysis

Flat

curve
=
> improvement opportunity


Central Repository


Secure place to store & access research data


Supports various kinds of research


Primary analysis of study data


Exploratory analysis of course data


Secondary analysis of any data set



Analysis & Reporting Tools


Focus on student
-
tutor interaction data


Data Export


Tab delimited tables you can open with your favorite
spreadsheet program or statistical package


Web services for direct access


DataShop

14

14

Repository


Allows for full data management


Controlled access for collaboration


File attachments


Paper attachments


Great for secondary analyses


How big is DataShop?


15

How big is DataShop?

Domain

Files

Papers

Datasets

Student Actions

Students

Student Hours

Language

64

11

78

6,237,523

6,499

6,877

Math

222

53

189

75,754,530

37,218

173,175

Science

92

19

93

13,849,756

16,939

45,465

Other

18

12

50

8,604,016

13,018

31,111

Total

396

95

410

104,445,825

73,674

256,630

As of April 2013

16

What kinds of data?


By domain based on studies from the Learn Labs



Data from intelligent tutors



Data from online instruction



Data from games



The data is fine grained at a transaction level!



17

Web Application


Explore data through the DataShop tools


Where is DataShop?


http://pslcdatashop.org


Linked from DataShop homepage and learnlab.org


http://pslcdatashop.web.cmu.edu/about/


http://learnlab.org/technologies/datashop/index.php


Getting to DataShop

19

19


KC
: Knowledge component


also known as a skill/concept/fact


a piece of information that can be used to
accomplish tasks


tagged at the step level


KC Model:


also known as a cognitive model or skill model


a mapping between problem steps and knowledge
components


DataShop Terminology

20

Getting the KC Model Right!

The KC model drives instruction in adaptive
learning


Problem and topic sequence


Instructional messages


Tracking student knowledge

21

What makes a good KC Model?


A correct expert model is one that is consistent with
student behavior.


Predicts task difficulty


Predicts transfer between instruction and test


The model should fit the data!

22

Good KC Model => Good Learning
Curve


An empirical basis for determining when a
cognitive model is good


Accurate predictions of student task
performance & learning transfer


Repeated practice on tasks involving the same skill
should reduce the error rate on those tasks

=> A declining learning curve should emerge

23

A Good Learning Curve

24

How do we make KC Models?

25

Traditionally CTA has been used

But Cognitive Task Analysis has some issues…


Extremely human driven


It is highly subjective


Leading to differing results from different analysts


And these human discovered models are usually
wrong!

26

If Human centered CTA is not the
answer

How should these models be designed?


They shouldn’t!


The models should be discovered
not

designed!

27

Solution


We have lots of log data from tutors and other systems











We can harness this data to validate and improve
existing student models


28

29

Human
-
Machine Student Model Discovery

DataShop provides easy interface to add and modify
KC models and ranks the models using AFM


29

Human
-
Machine Student Model
Discovery

3 strategies for discovering improvements to the
student model



Smooth learning curves



No apparent learning



Problems with unexpected error rates


30

A good cognitive model
produces a learning
curve

Without decomposition, using
just a single “Geometry” skill,

Is this the correct or “best”
cognitive model?

no smooth learning curve.

a smooth learning curve.

But with decomposition,

12 skills for area,

(Rise in error rate because
poorer students get
assigned more problems)

Inspect curves for individual
knowledge components (KCs)

Some do not =>

Opportunity to
improve model!

Many curves show a
reasonable decline

32

No apparent Learning

33

Problems with Unexpected Error Rates

34

Inspect problems to hypothesize new KC labels


Here scaffolding is originally absent, but other problems
have fixed scaffolding


They start with columns for square & area

These strategies suggest an
improvement


Hypothesized there were additional skills involved
in some of the compose by addition problems


A new student model (better BIC value) suggests
the splitting the skill.

36

Redesign based on Discovered Model

Our discovery suggested changes needed to be
made to the tutor


Resequencing



put problems requiring fewer
skills first


Knowledge Tracing


adding new skills


Creating new tasks


new problems


Changing instructional messages, feedback or
hints


37

Study : Current tutor is control


Current fielded tutor only uses
scaffolded

problems

Study: Treatment


Scaffolded
, given areas, plan
-
only, &
unscaffolded


Isolate practice

on problem

decomposition

Study Results


Much more efficient & better learning on
targeted decomposition skills

Post
-
test % correct by item type

0.7
0.75
0.8
0.85
0.9
0.95
1
Control: Original
tutor
Treatment: Model-
based redesign
Composition
Area
Instructional time (minutes) by step type

0
10
20
30
Control: Original tutor
Treatment: Model-
based redesign
Area and other steps
Composition steps
Design

Deploy

Data

Discover

Translational Research Feedback Loop

Can a data
-
driven process be
automated & brought to scale?

Yes!



Combine Cognitive Science, Psychometrics,
Machine Learning …


Collect a rich body of data


Develop new model discovery algorithms,
visualizations, & on
-
line collaboration
support

42

DataShop’s

“leaderboard” ranks discovered cognitive models

100s of datasets coming from
ed

tech in math, science, & language

Some models are
machine

generated
(based on
human
-
generated learning factors)

Some models are
human

generated

43

Metrics for model prediction


AIC & BIC penalize for more parameters,

fast & consistent



10 fold cross validation


Minimize root mean squared error (RMSE) on
unseen data


44

Automated search for better models

Learning Factors
Analysis (LFA)


(Cen, Koedinger, & Junker, 2006)




Method for discovering & evaluating cognitive models


Finds model “Q matrix” that best predicts student learning
data


Inputs


Data: Student success on tasks over time


Factors hypothesized to explain learning


Outputs


Rank order of most predictive Q matrix


Parameter estimates for each


Simple search process example:
modifying Q matrix
by input factor
to
get new Q’ matrix


Produces new Q matrix


Two new
KCs

(Sub
-
Pos & Sub
-
Neg
) replace old KC (Sub)


Redo opportunity counts



Q matrix factor Sub
split
by factor
Neg
-
result

LFA: Best First Search Process

Cen
, H., Koedinger, K., Junker, B. (2006).

Learning Factors Analysis:
A general method for cognitive model evaluation and improvement.
8th
International Conference on Intelligent Tutoring Systems
.


Search algorithm guided by a
heuristic
:

AIC


Start

with

single
skill

cog
model (Q matrix)


Scientist “
crowd”sourcing
:

Feature input comes “for free”

Scientist generated models

48

Union of all hypothesized KCs in

human generated models

Validating Learning Factors Analysis



Discovers better cognitive models in 11 of 11
datasets …

Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement.
In
Proceedings of the Fifth International Conference on Educational Data Mining
.
[Conference best paper.]

Data from a variety of educational
technologies & domains

50

Numberline

Game

Statistics Online Course

English Article Tutor

Algebra Cognitive Tutor

Applying LFA across domains

11 of 11 improved

models

9 of 11 equal

or greater learning

Variety of domains

& technologies

Can we go even bigger?

52


Competitions?

KDD Cup Competition


Knowledge Discovery and Data Mining (KDD) is the most
prestigious conference in the data mining and machine
learning fields



KDD Cup is the premier data mining challenge



2010 KDD Cup called “Educational Data Mining Challenge”



Ran from April 2010 through June 2010

54

KDD Cup Competition

Competition goal is to predict student responses given tutor data
provided by Carnegie Learning



Dataset

Students

Steps

File size

Algebra I 2008
-
2009

3,310

9,426,966

3 GB

Bridge to Algebra 2008
-
2009

6,043

20,768,884

5.43 GB

55

KDD Cup Competition



655 registered participants




130 participants who submitted predictions




3,400 submissions


KDD Cup Competition


Advances in prediction, cognitive modeling, new methods
applied to EDM



Spawned a number of workshops and papers



The datasets are now in the “wild” and showing up in non
KDD conferences



New competitions to continue momentum

57

Marigames.org


Two stage competition with $100,000 in
prizes


$50,000 Game Development


$50,000 Educational Data Mining


Goal is to go beyond individual datasets


This requires common data formats


58

Take
aways


The amount of data coming from educational
technology is growing exponentially



Huge potential for EDM to improve educational
systems




Optimal instructional design requires discoveries
(The student is not like me)



These methods require common forms of data for
analysis (standards!)

59

Opportunities


New Learning Science and Engineering
professional masters degree at Carnegie
Mellon University



New concentration in Learning Analytics, MA
in Cognitive Studies in Education at Teachers
College, Columbia University



Other programs in the works

60

Thank you

Special Thanks to:

Ken Koedinger, Director
LearnLab


Ryan Baker, President IEDMS

Steve Ritter, Carnegie Learning


61

http://pslcdatashop.org

Questions?

john@stamper.org

http://dev.stamper.org

62