Cognitive Tutors: Bringing advanced cognitive research to the ...

siennaredwoodAI and Robotics

Feb 23, 2014 (3 years and 7 months ago)

81 views

1

LearnLab
: Bridging the Gap
Between Learning Science and
Educational Practice

Ken Koedinger

Human
-
Computer Interaction & Psychology, CMU

PI & CMU Director of
LearnLab

2

Real World Impact of

Cognitive Science


Algebra Cognitive Tutor


Based on

ACT
-
R theory

& cognitive models
of
student learning


Used in

3000
schools

600,000
students


Spin
-
off:

Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

3

Personalized instruction

Challenging questions

… individualization

Progress…

Authentic problems

Feedback
within
complex solutions

Cognitive Tutors:
Interactive Support for
Learning by Doing

4

Success ingredients


AI technology


Cognitive Task Analysis


Principles of instruction &

experimental methods


Fast development &

use
-
driven iteration

Cognitive Task Analysis:

What is hard for
Algebra students?




Story Problem


As a waiter, Ted gets $6 per hour. One night he made $66 in
tips and earned a total of $81.90. How many hours did Ted
work?


Word Problem


Starting with some number, if I multiply it by 6 and then add
66, I get 81.90. What number did I start with?


Equation


x * 6 + 66 = 81.90

6

0

10

20

30

40

50

60

70

80

90

100

Elementary

Teachers

Middle

School

Teachers

High School

Teachers

% Correctly ranking equations as
hardest

Nathan & Koedinger (2000). An
investigation of teachers’ beliefs of
students’ algebra development.
Cognition and Instruction.

Expert Blind Spot!

Koedinger & Nathan (2004). The real story
behind story problems: Effects of
representations on quantitative reasoning.
The Journal of the Learning Sciences
.

Data contradicts common beliefs
of researchers and teachers

7

Cognitive Tutor Algebra
course

yields significantly better
learning

Course

includes text,
tutor, teacher
professional
development


~11
of
14
full
-
year
controlled studies
demonstrate

significantly better

student learning

Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

8

Success? Yes

Done? No!

Why not?


Student achievement still not ideal


Field study results are imperfect


Many design decisions with no research
base



Use deployed
technology to collect
data, make discoveries,

& continually
improve

9

PSLC Vision


Why?

Chasm

between science &
ed

practice



Purpose
:
I
dentify the conditions
that cause robust student
learning


Educational technology as instrument


Science
-
practice collaboration structure



Core Funding:

2004
-
2014


10

What we
know about
our own
learning

What we do
not

know

You can’t design for what you don’t know!

Do you know what you know?

11

Chemistry Virtual Lab

Algebra

Cognitive Tutor

Ed tech + wide use = “Basic research
at scale



=

Transforming
Education
R&D


Fundamentally transform


Applied research
in education


Generation
of

practice
-
relevant learning theory

+

English

Grammar Tutor

Educational Games

Ed Tech => Data => Better learning

LearnLab Thrusts

LearnLab Course
Committees

13

How you can benefit from
LearnLab


Research


General principles to improve learning


Methods


Cognitive task analysis, in vivo studies


Technology tools


People


Masters students & projects

14

What instructional
strategies work best?


More assistance vs. more challenge


Basics vs. understanding


Education wars in reading, math, science…

Koedinger &
Aleven

(2007). Exploring the assistance dilemma
in experiments with Cognitive Tutors.
Ed Psych Review.


Research on many dimensions


Massed vs.
distributed

(
Pashler
)


Study vs.
test

(
Roediger
)


Examples

vs. problem solving (
Sweller,Renkl
)


Direct instruction

vs. discovery learning (
Klahr
)


Re
-
explain vs.
ask for explanation

(Chi,
Renkl
)


Immediate

vs.
delayed

(Anderson vs.
Bjork
)


Concrete

vs.
abstract

(
Pavio

vs. Kaminski)





15

Knowledge
-
Learning
-
Instruction
(KLI) Framework:
What conditions
cause robust learning

LearnLab

research thrusts
address KLI elements


Cognitive
Factors



Charles
Perfetti
,

David
Klahr


Metacognition

&
Motivation


Vincent
Aleven
, Tim
Nokes
-
Malach



Social Communication



Lauren
Resnick
,

Carolyn Rose


Computational Modeling &
Data Mining



Geoff Gordon
,

Ken Koedinger

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the science
-
practice
chasm to enhance robust student learning.
Cognitive Science
.

16

Results of ~200
in vivo
experiments =>

Optimal instruction depends on knowledge goals


17

Cognitive Task Analysis
using
DataShop’s

learning curve tools

Without decomposition, using
just a single “Geometry” KC,

Upshot
:

Can automate analysis
& produce better student models

But with decomposition,
12 KCs for area concepts,

a smoother learning curve.

no smooth learning curve.

18

How you can benefit from
LearnLab


Research


General principles to improve learning


Methods


Cognitive task analysis, in vivo studies


Technologies


Tutor authoring


Language processing


Educational Data Mining


People: Masters students & projects

19

Questions?

20

Question for you

What do you need in a learning
science professional?

21


22

Extra slides


23

3(2x
-

5) = 9

6x
-

15 = 9

2x
-

5 = 3

6x
-

5 = 9

Cognitive Tutor
Technology


Cognitive Model
: A system that can solve problems in
the various ways students can



If goal is solve a(bx+c) = d

Then rewrite as abx + ac = d

If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

If goal is solve a(bx+c) = d

Then rewrite as bx+c = d/a


Model Tracing
: Follows student through their individual
approach to a problem
-
> context
-
sensitive instruction

24

3(2x
-

5) = 9

6x
-

15 = 9

2x
-

5 = 3

6x
-

5 = 9

Cognitive Tutor
Technology


Cognitive Model
: A system that can solve problems in
the various ways students can



If goal is solve a(bx+c) = d

Then rewrite as abx + ac = d

If goal is solve a(bx+c) = d

Then rewrite as abx + c = d


Model Tracing
: Follows student through their individual
approach to a problem
-
> context
-
sensitive instruction

Hint message: “Distribute
a


across the parentheses.”

Bug message: “You need to

multiply c by a also.”


Knowledge Tracing
: Assesses student's knowledge
growth
-
> individualized activity selection and pacing

Known? = 85% chance

Known? = 45%

25

Cognitive Task Analysis
Improves Instruction


Studies: Traditional instruction vs. CTA
-
based


Med school catheter insertion
(
Velmahos

et al., 2004)


Radar system troubleshooting
(
Schaafstal

et al., 2000)



Spreadsheet use
(Merrill, 2002)



Lee (2004) meta
-
analysis: 1.7 effect size!


26

Learning Curves


27

Inspect curves for individual
knowledge components (KCs)

Some do not =>

Opportunity to
improve model!

Many curves show a
reasonable decline

28

DataShop’s

“leaderboard” ranks alternative models

100s of datasets from
ed

tech in math, science, & language

Best model finds 18 components of knowledge
(
KCs
) that best predict transfer

28

Data from a variety of educational
technologies & domains

29

Numberline

Game

Statistics Online Course

English Article Tutor

Algebra Cognitive Tutor

Model discovery across domains

30

11 of 11 improved

models

Variety of domains

& technologies

Koedinger, McLaughlin, &
Stamper (2012). Automated
student model improvement.
In
Proceedings of

Educational
Data Mining
. [Conference best
paper.]

31

Data reveals
students’
achievement

& motivations

We have used it to


Predict future state test scores as well
or better than the tests themselves


Assess dispositions like work ethic


Assess motivation & engagement


Assess & improve learning skills like
help seeking



32

LearnLab courses at
K12 & College Sites


6
+

cyber
-
enabled

courses:
Chemistry, Physics,
Algebra, Geometry,
Chinese, English


Data collection


Students do home/lab work
on tutors,
vlab
, OLI, …


Log data, questionnaires,
tests


DataShop

Researchers

Schools

Learn
Lab

Chemistry virtual lab

Physics intelligent tutor

REAP
vocabulary
tutor

33

Lab
experiment

In Vivo
Experiment

Design
Research

Randomzd
Field Trial

Setting

Lab

School

School

School

Control condition

Yes

Yes

No

Yes

Focus on principle
vs. on solution

(Change N things)

Scientific
Principle

Scientific

Principle

Instr.
Solution

Instr.
Solution


Cost/Duration

$/Short

$$/Medium

$$
/Long

$$$$/Long

Bridging methodology:

in vivo

experiments

34

Knowledge Components


Definition: An
acquired

unit of cognitive
function or structure that can be
inferred

from
performance on
a set of related tasks


Includes:


skills, concepts, schemas, metacognitive strategies,
malleable habits of mind, thinking & learning skills


May also include:


malleable motivational beliefs & dispositions


Does not include:


fixed cognitive architecture,

transient states of cognition or affect


Components of
“intellectual plasticity”

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the science
-
practice chasm to enhance robust student learning.
Cognitive Science
.

35

General knowledge components,
sense
-
making, motivation, social
intelligence

Possible domain
-
general
KCs


Metacognitive

strategy


Novice KC: If I’m studying an example, try to remember
each step


Desired KC: If I’m studying an example, try to explain how
each step follows from the previous


Motivational belief


Novice: I am no good at math


Desired: I can get better at math by studying & practicing


Social communicative strategy


Novice: If an authority makes a claim, it is true


Desired: If considering a claim, look for evidence for &
against it


36

What is Robust Learning?


Achieved through:


Conceptual understanding & sense
-
making
skills


Refinement of initial understanding


Development of procedural fluency with
basic skills



Measured by:


Transfer to novel tasks


Retention over the long term, and/or


Acceleration of future learning



37

KLI

summary


Learning occurs in
components (
KCs
)


KCs

vary in kind/
cmplxty


Require different kinds of
learning
mechanisms



Optimal instructional
choices are dependent

on KC complexity


Intelligence does not improve generically

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the
science
-
practice chasm to enhance robust student learning.
Cognitive Science
.

38

Conclusions


Learning & education are complex
systems



Lots of work for learning science!



Use
ed

tech for “basic research at scale”

=> Bridge science
-
practice chasm