Automated Assessment for Adaptive Learning of Complex Tasks

hartebeestgrassΤεχνίτη Νοημοσύνη και Ρομποτική

7 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

61 εμφανίσεις

Automated Assessment for

Adaptive Learning of Complex Tasks

Alan
Koenig

Markus
Iseli

Allen Munro

CCT

Characteristics of Cognitively Complex Tasks


Often require multiple, non
-
trivial steps to complete


Performance can be highly variable


Often include interdependent task features


Often well suited to representation in games & simulations


Multiple scenarios


Low cost

Examples
:

Tactical Planning Simulation

Damage Control Simulation



Low
risk



But… assessment
can be difficult

Anti
-
Submarine Warfare

Anti
-
submarine warfare

(
ASW
)
is a branch
of naval
warfare that uses surface warships, aircraft, or other
submarines to find, track and deter, damage, or destroy
enemy submarines.



Carried out by teams of people within and across
ships


Requires synthesis of large quantities of information in
short periods of time


Heavy emphasis on situation awareness and decision
making skills (especially among TAO’s)



Assessment Methodology Overview

ASW Ontology

Construction

Experts

Rules /

Doctrine

Procedures /

Processes

Facts /

Observations

DOMAIN

Ontology Excerpt

Bayesian Network Development

Ontology

Nodes

Directionality

of Links

Observables /

SME Input

Strength of

Relationships

Characterizing the
nature
of relationships is essential

Observable Actions

Latent Variables

Sonar

Planning

Passive

Sonar Planning

Active

Sonar
Planning

U
s
eShipPassiveSonar

DropPassiveSonobuoys

DropActiveSonobuoys

Use
DippingSonar

UseShipActiveSonar

ASW Skilled

Torpedo
CounterMeasures

MEU

Protection

Sub Hunting

Plan for
Opposition

Intell on
Opposition

Opposition
Estimation

ShowMissionInfo

ShowBriefing

Est
LL
SubSpee
d

Est
LL
TorpedoRang
e

0.5

Sonar

Planning

TDA Avoid

Maneuvers

Passive

Sonar Use

Active

Sonar Skill

AlterCourseTo

AvoidFast

AlterCourseTo

AvoidSlow

UseShipPassiveSonar

DropPassiveSonobuoy
s

DropActiveSonobuoys

Use
DippingSonar

UseShipActiveSonar

TDA
Avoidance

DontEverEnterTDA

LimLines

Estimation

Est
Datum
SubSpeed

Est
Datum
TorpedoRang
e

Datum
Estimation

Torpedo

Deception

Torpedo

Escape

NixieDropped

MICDropped

TorpedoEvasion

TorpedoCounter

Bayesian

Network for TDA Avoidance

ASW Skilled

Torpedo
CounterMeasures

MEU

Protection

Sub Hunting

Plan for
Opposition

Intell on
Opposition

Opposition
Estimation

0.5

TDA Avoid

Maneuvers

Drop
ActiveSonobuoys

Use
DippingSonar

Use
ShipActiv
e
Sonar

TDA
Avoidance

LimLines

Estimation

Datum
Estimation

Torpedo

Deception

Torpedo

Escape

Bayesian

Network for TDA Avoidance

ShowMissionInfo

ShowBriefing

Est
LL
SubSpee
d

Est
LL
TorpedoRang
e

Sonar

Planning

Passive

Sonar Use

Active

Sonar Skill

AlterCourseTo

AvoidFast

AlterCourseTo

AvoidSlow

Use
ShipPassiv
e
Sonar

Drop
PassiveSonobuoy
s

DontEverEnterTDA

Est
Datum
SubSpeed

Est
Datum
TorpedoRang
e

NixieDropped

MICDropped

TorpedoEvasion

TorpedoCounter

Conditions for TDA Avoidance to be considered by Bayesian network

Observable Actions

Latent Variables

The Advantage of Bayesian Networks


Sonar

Planning

Passive

Sonar Planning

Active

Sonar
Planning

U
s
eShipPassiveSonar

DropPassiveSonobuoys

DropActiveSonobuoys

Use
DippingSonar

UseShipActiveSonar

Probability

of mastery of the latent variables is
inferred
from the scored observable actions

P
(
ASP
):
Probability of mastery of concept
Active Sonar
Planning

P
(
UDS | ASP
):
Conditional probability. Probability of mastery of
concept
UseDippingSonar
, given information about mastery of
concept
Active Sonar Planning

Example
: Determining probabilities

P
(
UDS
):
Probability of mastery of concept
UseDippingSonar

P
(ASP)

x
P
(
UDS

|
ASP
)

P
(UDS)

P
(ASP |
UDS
)
=

Real
-
Time Formative Assessment


Provide performance feedback


Provide practice / resources


Add / change tasks


Add / change affordances

FORMATIVE
ASSESSMENT

Simulation

(Sandbox)

ADAPTIVE OPTIONS

Observable

Actions & Events

CAA

(with Bayesian Network)

Check the Briefing

Good!

1
st

PROBLEM: Chinese Kilos Threat Problem

00:00:00 .k.Announce. ShowBriefing 0.95

The fact that we’re up

against Kilos implies a

silent sub speed of 6 Kt.

and a torpedo range of

12,000 yds.

Failed to Correct Estimates

Bad!

1
st

PROBLEM: Chinese Kilos Threat Problem

07:26:32 .k.Announce. EstSubSpeed 0.2

07:26:32 .k.Announce. EstTorpedoRange 0.2

The estimated sub speed
was left at 3 Kt. So the

limit lines were not

accurate, relative to the

briefing.

Scored at end, because these
values were never changed.

Failed to Correct Estimates

Bad!

1
st

PROBLEM: Chinese Kilos Threat Problem

07:26:32 .k.Announce. DontEverEnterTDA 0.1

Uh
-
oh! The MEU entered
the TDZ.

17

Problem 1

18

Problem 1

19

Problem 1

20

Problem 1

21

Problem 1

Check the Briefing

Good!

2
nd

PROBLEM: Threat West of Gibraltar

This time the mission

brief gives specific
estimates of hostile sub
speed and torpedo
range.

00:00:00 .k.Announce. ShowBriefing 0.95

Datum Estimates

2
nd

PROBLEM: Threat West of Gibraltar

The datum estimates for

sub speed and torpedo

range don’t match

the briefing.

Datum Estimates Corrected

Good!

2
nd

PROBLEM: Threat West of Gibraltar

Set to the correct values.

01:14:03 .k.Announce. EstSubSpeed 0.95

01:14:03 .k.Announce. EstTorpedoRange 0.95

Scored at end.

Check Limit Lines

2
nd

PROBLEM: Threat West of Gibraltar

Perhaps not a wide enough
berth…

Plot a New Course

2
nd

PROBLEM: Threat West of Gibraltar

A safer course?

Check Limit Lines Again

2
nd

PROBLEM: Threat West of Gibraltar

Looks safer

Successful Avoidance

2
nd

PROBLEM: Threat West of Gibraltar

Success!

01:14:03 .k.Announce. DontEverEnterTDA 0.95

29

Problem 2

30

Problem 2

31

Problem 2

32

Problem 2

33

0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1
2
3
4
5
Problem 1
Problem 2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2
3
4
5
Problem 1
Problem 2
Plan for Opposition

TDA Avoidance

Sub speed: Wrong estimate

Entered TDA

Probability of Mastery

Probability of Mastery

time

time

ASW Skilled

Torpedo
CounterM.

MEU

Protection

Sub Hunting

Plan for
Opposition

Intelligence on
Opposition

Opposition
Estimation

GetMissionInfo

GetBriefing

EstSubSpeed

EstTorpedoRange

Sonar

Planning

TDA Avoid

Maneuvers

Sense

Quietly

Sense

Loudly

ManeuverFast

ManeuverSlow

ShipsPassivSonar

PassiveSonobuoys

ActiveSonobuoys

DippingSonar

ShipsActivSonar

TDA
Avoidance

B20: score 0.95

D20: score 0.6

F20: score 0.1

B16: score 0.95

D16: score 0.7

F16: score 0.05

B5: score 0.9

D5: score 0.7

B6: score 0.95

D6: score 0.7

B11: score 0.95

D11: score 0.8

F11: score 0.2

B12: score 0.95

D12: score 0.8

F12: score 0.2

B36: score 0.9

E36: score 0.3

F36: score 0.2

B40: score 0.95

C40: score 0.4

E40: score 0.3

C44: score 0.9

D44: score 0.6

F44: score 0.05

B28: score 0.95

D28: score 0.6

B32: score 0.95

E32: score 0.3

DontEnterTDAIfPos

B24: score 0.95

C24: score 0.9

F24: score 0.1

TDA avoidance

Bayesian network sub
-
net

Initial Evaluation Feedback

CAA Debugging Output

Instructional Feedback

High Level Assessments

First Successful Problem

Into the Next Problem

Adaptive Problem Selection

A new problem is advised.

Loading the Problem

The advised problem is selected

Initial Actions in the New Problem

Preparing to Find the Sub

Launching a helicopter

Detecting the Sub

Sonobuoys

detect sub

Recording of Performances & Assessments

Simulation

(Sandbox)

Log Files

(Actions and Events)

These performance records

can be replayed.

Report Generation

(CAA Monitor)

Assessment History

(Per Student)

Report

CAA

Questions / Comments?

CCT