# Cognitive Tutors: Bringing advanced cognitive research to the ...

IA et Robotique

23 févr. 2014 (il y a 8 années et 9 jours)

353 vue(s)

1

LearnLab
: Bridging the Gap
Between Learning Science and
Educational Practice

Ken Koedinger

Human
-
Computer Interaction & Psychology, CMU

PI & CMU Director of
LearnLab

2

Real World Impact of

Cognitive Science

Algebra Cognitive Tutor

Based on

ACT
-
R theory

& cognitive models
of
student learning

Used in

3000
schools

600,000
students

Spin
-
off:

Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

3

Personalized instruction

Challenging questions

… individualization

Progress…

Authentic problems

Feedback
within
complex solutions

Cognitive Tutors:
Interactive Support for
Learning by Doing

4

Success ingredients

AI technology

Principles of instruction &

experimental methods

Fast development &

use
-
driven iteration

What is hard for
Algebra students?

Story Problem

As a waiter, Ted gets \$6 per hour. One night he made \$66 in
tips and earned a total of \$81.90. How many hours did Ted
work?

Word Problem

Starting with some number, if I multiply it by 6 and then add
66, I get 81.90. What number did I start with?

Equation

x * 6 + 66 = 81.90

6

0

10

20

30

40

50

60

70

80

90

100

Elementary

Teachers

Middle

School

Teachers

High School

Teachers

% Correctly ranking equations as
hardest

Nathan & Koedinger (2000). An
investigation of teachers’ beliefs of
students’ algebra development.
Cognition and Instruction.

Expert Blind Spot!

Koedinger & Nathan (2004). The real story
behind story problems: Effects of
representations on quantitative reasoning.
The Journal of the Learning Sciences
.

Data contradicts common beliefs
of researchers and teachers

7

Cognitive Tutor Algebra
course

yields significantly better
learning

Course

includes text,
tutor, teacher
professional
development

~11
of
14
full
-
year
controlled studies
demonstrate

significantly better

student learning

Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

8

Success? Yes

Done? No!

Why not?

Student achievement still not ideal

Field study results are imperfect

Many design decisions with no research
base

Use deployed
technology to collect
data, make discoveries,

& continually
improve

9

PSLC Vision

Why?

Chasm

between science &
ed

practice

Purpose
:
I
dentify the conditions
that cause robust student
learning

Educational technology as instrument

Science
-
practice collaboration structure

Core Funding:

2004
-
2014

10

What we
our own
learning

What we do
not

know

You can’t design for what you don’t know!

Do you know what you know?

11

Chemistry Virtual Lab

Algebra

Cognitive Tutor

Ed tech + wide use = “Basic research
at scale

=

Transforming
Education
R&D

Fundamentally transform

Applied research
in education

Generation
of

practice
-
relevant learning theory

+

English

Grammar Tutor

Educational Games

Ed Tech => Data => Better learning

LearnLab Thrusts

LearnLab Course
Committees

13

How you can benefit from
LearnLab

Research

General principles to improve learning

Methods

Cognitive task analysis, in vivo studies

Technology tools

People

Masters students & projects

14

What instructional
strategies work best?

More assistance vs. more challenge

Basics vs. understanding

Education wars in reading, math, science…

Koedinger &
Aleven

(2007). Exploring the assistance dilemma
in experiments with Cognitive Tutors.
Ed Psych Review.

Research on many dimensions

Massed vs.
distributed

(
Pashler
)

Study vs.
test

(
Roediger
)

Examples

vs. problem solving (
Sweller,Renkl
)

Direct instruction

vs. discovery learning (
Klahr
)

Re
-
explain vs.

(Chi,
Renkl
)

Immediate

vs.
delayed

(Anderson vs.
Bjork
)

Concrete

vs.
abstract

(
Pavio

vs. Kaminski)

15

Knowledge
-
Learning
-
Instruction
(KLI) Framework:
What conditions
cause robust learning

LearnLab

research thrusts

Cognitive
Factors

Charles
Perfetti
,

David
Klahr

Metacognition

&
Motivation

Vincent
Aleven
, Tim
Nokes
-
Malach

Social Communication

Lauren
Resnick
,

Carolyn Rose

Computational Modeling &
Data Mining

Geoff Gordon
,

Ken Koedinger

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the science
-
practice
chasm to enhance robust student learning.
Cognitive Science
.

16

Results of ~200
in vivo
experiments =>

Optimal instruction depends on knowledge goals

17

using
DataShop’s

learning curve tools

Without decomposition, using
just a single “Geometry” KC,

Upshot
:

Can automate analysis
& produce better student models

But with decomposition,
12 KCs for area concepts,

a smoother learning curve.

no smooth learning curve.

18

How you can benefit from
LearnLab

Research

General principles to improve learning

Methods

Cognitive task analysis, in vivo studies

Technologies

Tutor authoring

Language processing

Educational Data Mining

People: Masters students & projects

19

Questions?

20

Question for you

What do you need in a learning
science professional?

21

22

Extra slides

23

3(2x
-

5) = 9

6x
-

15 = 9

2x
-

5 = 3

6x
-

5 = 9

Cognitive Tutor
Technology

Cognitive Model
: A system that can solve problems in
the various ways students can

If goal is solve a(bx+c) = d

Then rewrite as abx + ac = d

If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

If goal is solve a(bx+c) = d

Then rewrite as bx+c = d/a

Model Tracing
: Follows student through their individual
approach to a problem
-
> context
-
sensitive instruction

24

3(2x
-

5) = 9

6x
-

15 = 9

2x
-

5 = 3

6x
-

5 = 9

Cognitive Tutor
Technology

Cognitive Model
: A system that can solve problems in
the various ways students can

If goal is solve a(bx+c) = d

Then rewrite as abx + ac = d

If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

Model Tracing
: Follows student through their individual
approach to a problem
-
> context
-
sensitive instruction

Hint message: “Distribute
a

across the parentheses.”

Bug message: “You need to

multiply c by a also.”

Knowledge Tracing
: Assesses student's knowledge
growth
-
> individualized activity selection and pacing

Known? = 85% chance

Known? = 45%

25

Improves Instruction

Studies: Traditional instruction vs. CTA
-
based

Med school catheter insertion
(
Velmahos

et al., 2004)

(
Schaafstal

et al., 2000)

(Merrill, 2002)

Lee (2004) meta
-
analysis: 1.7 effect size!

26

Learning Curves

27

Inspect curves for individual
knowledge components (KCs)

Some do not =>

Opportunity to
improve model!

Many curves show a
reasonable decline

28

DataShop’s

“leaderboard” ranks alternative models

100s of datasets from
ed

tech in math, science, & language

Best model finds 18 components of knowledge
(
KCs
) that best predict transfer

28

Data from a variety of educational
technologies & domains

29

Numberline

Game

Statistics Online Course

English Article Tutor

Algebra Cognitive Tutor

Model discovery across domains

30

11 of 11 improved

models

Variety of domains

& technologies

Koedinger, McLaughlin, &
Stamper (2012). Automated
student model improvement.
In
Proceedings of

Educational
Data Mining
. [Conference best
paper.]

31

Data reveals
students’
achievement

& motivations

We have used it to

Predict future state test scores as well
or better than the tests themselves

Assess dispositions like work ethic

Assess motivation & engagement

Assess & improve learning skills like
help seeking

32

LearnLab courses at
K12 & College Sites

6
+

cyber
-
enabled

courses:
Chemistry, Physics,
Algebra, Geometry,
Chinese, English

Data collection

Students do home/lab work
on tutors,
vlab
, OLI, …

Log data, questionnaires,
tests

DataShop

Researchers

Schools

Learn
Lab

Chemistry virtual lab

Physics intelligent tutor

REAP
vocabulary
tutor

33

Lab
experiment

In Vivo
Experiment

Design
Research

Randomzd
Field Trial

Setting

Lab

School

School

School

Control condition

Yes

Yes

No

Yes

Focus on principle
vs. on solution

(Change N things)

Scientific
Principle

Scientific

Principle

Instr.
Solution

Instr.
Solution

Cost/Duration

\$/Short

\$\$/Medium

\$\$
/Long

\$\$\$\$/Long

Bridging methodology:

in vivo

experiments

34

Knowledge Components

Definition: An
acquired

unit of cognitive
function or structure that can be
inferred

from
performance on
a set of related tasks

Includes:

skills, concepts, schemas, metacognitive strategies,
malleable habits of mind, thinking & learning skills

May also include:

malleable motivational beliefs & dispositions

Does not include:

fixed cognitive architecture,

transient states of cognition or affect

Components of
“intellectual plasticity”

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the science
-
practice chasm to enhance robust student learning.
Cognitive Science
.

35

General knowledge components,
sense
-
making, motivation, social
intelligence

Possible domain
-
general
KCs

Metacognitive

strategy

Novice KC: If I’m studying an example, try to remember
each step

Desired KC: If I’m studying an example, try to explain how
each step follows from the previous

Motivational belief

Novice: I am no good at math

Desired: I can get better at math by studying & practicing

Social communicative strategy

Novice: If an authority makes a claim, it is true

Desired: If considering a claim, look for evidence for &
against it

36

What is Robust Learning?

Achieved through:

Conceptual understanding & sense
-
making
skills

Refinement of initial understanding

Development of procedural fluency with
basic skills

Measured by:

Transfer to novel tasks

Retention over the long term, and/or

Acceleration of future learning

37

KLI

summary

Learning occurs in
components (
KCs
)

KCs

vary in kind/
cmplxty

Require different kinds of
learning
mechanisms

Optimal instructional
choices are dependent

on KC complexity

Intelligence does not improve generically

Koedinger et al. (2012). The Knowledge
-
Learning
-
Instruction (KLI) framework: Bridging the
science
-
practice chasm to enhance robust student learning.
Cognitive Science
.

38

Conclusions

Learning & education are complex
systems

Lots of work for learning science!

Use
ed

tech for “basic research at scale”

=> Bridge science
-
practice chasm