Corpora for Plan Recognition

ghostslimIA et Robotique

23 févr. 2014 (il y a 3 années et 1 mois)

66 vue(s)

Thinking about Evaluation and
Corpora for Plan Recognition

Nate Blaylock

Florida Institute for Human and Machine Cognition (IHMC)

Ocala, Florida

blaylock@ihmc.us

Plan Recognition Evaluation


Extrinsic (Tom Dietterich’s comment)


Online


prediction after each observation


Precision/recall


ability to predict “don’t know”


Offline


predict right answer for the session


convergence

More Evaluation


How early in session do we get it?


convergence point

(Lesh


“work saved”)


Partial results (often enough)


Lower subgoals in HTN plan


More abstract (subsuming) goals


Schema only or only some parameters


N
-
best prediction

Example: Results on Monroe

Level
prec.
recall
param%
conv.
conv. pt.
conv param%
0
82.5%
56.4%
24.0%
90.8%
5.6/10.3
49.8%
1
81.3%
52.8%
23.5%
67.6%
3.1/6.1
26.5%
2
85.4%
44.3%
22.5%
45.8%
3.4/4.7
38.5%
3
72.9%
41.7%
82.4%
41.2%
3.0/3.5
90.6%
4
73.6%
50.0%
99.9%
61.8%
3.7/3.7
100.0%
5
58.8%
45.7%
100.0%
6.2%
4.2/4.2
100.0%
6
69.3%
69.3%
100.0%
0.0%
N/A
N/A
7
95.2%
95.2%
100.0%
N/A
N/A
N/A
8
100.0%
100.0%
100.0%
N/A
N/A
N/A
Total
79.0%
50.4%
44.1%
61.7%
3.9/6.8
46.4%
1 Best

2 Best

Total
91.3%
68.7%
47.2%
92.5%
3.6/6.1
43.7%
Plan Corpora: Types


Unlabeled
: sequence of actions taken, e.g.,


Unix commands
(Davidson and Hirsh’98)


also GPS data
(e.g., Patterson et al. 2003)


Goal
-
labeled
: actions + top
-
level goal(s), e.g.,


MUD domain
(Albrecht et al. ’98)


Unix/Linux
(Lesh ’98, Blaylock and Allen 2004)


Linux Plan Corpus available online

Plan Corpora: Types (2)


Plan
-
labeled
: actions + hierarchical plan


Monroe Plan Corpus (Blaylock and Allen 2005)


available online


(future?) Problem
-
solving labeled


Action failure, replanning, goal abandonment, ...

Creating Plan Corpora
(from humans)

Human annotation of everything,
OR


Action sequence
: record observations directly


Top
-
level goal(s):



idea 1: environment where goal achievement
observable (e.g., MUD)


idea 2: controlled environment where goal is known a
priori (e.g., Unix/Linux)


Plan
-
labeled
:


annotate with existing plan recognizer (Bauer ’96)


May not apply to all domains

Generating Artificial Corpora

(Blaylock and Allen, 2005)


Randomized AI planner (SHOP2)


Model domain for planner (HTN)


For each desired plan session


stochastically generate goal(s)


stochastically generate start state


find plan using planner

Using the Method:

The Monroe Corpus


Emergency planning domain


10 top
-
level goal schemas


46 methods (recipes)


30 operators (subgoals/actions)


Average depth to action: 3.8


5000 plan sessions generated in less than 10
minutes


plan
-
labeled corpus


Download at



http://cs.rochester.edu/research/speech/monroe
-
plan/

Future Directions


Problem
-
solving labeled corpus


Similar method to Monroe


Build stochastic agent to do problem solving in
domain with plan monitoring, replanning, goal
abandonment, etc.


Label steps where PS behavior happened


cf. (Rosario, Oliver, and Pentland, 1999)