Knowledge Engineering for

lettuceescargatoireAI and Robotics

Nov 7, 2013 (4 years and 1 day ago)

80 views

1

Knowledge Engineering for
Bayesian Networks

Ann Nicholson




School of Computer Science

and Software Engineering

Monash University


2

Overview


Representing uncertainty


Introduction to Bayesian Networks

»
Syntax, semantics, examples


The knowledge engineering process


Case Study: Intelligent Tutoring


Summary of other BN research


Open research questions

3

Sources of Uncertainty


Ignorance


Inexact observations


Non
-
determinism


AI representations

»
Probability theory

»
Dempster
-
Shafer

»
Fuzzy logic

4

Probability theory for
representing uncertainty


Assigns a numerical degree of belief between
0 and 1 to facts

»
e.g. “it will rain today” is T/F.

»
P(“it will rain today”) = 0.2 prior probability
(unconditional)


Posterior probability (conditional)

»
P(“it wil rain today” | “rain is forecast”) = 0.8


Bayes’ Rule:
P(H|E) =
P(E|H) x P(H)


P(E)

5

Bayesian networks


Directed acyclic graphs


Nodes: random variables,

»
R: “it is raining”, discrete values T/F

»
T: temperature, cts
or

discrete variable

»
C: colour, discrete values {red,blue,green}


Arcs indicate dependencies (can have causal
interpretation)

6

Bayesian networks


Conditional Probability Distribution (CPD)


Associated with each variable


probability of each state given parent states

“Jane has the flu”

“Jane has a

high temp”

“Thermometer

temp reading”

X

Flu

Y

Te

Q

Th

Models causal relationship

Models possible sensor error

P(Flu=T) = 0.05

P(Te=High|Flu=T) = 0.4

P(Te=High|Flu=F) = 0.01

P(Th=High|Te=H) = 0.95

P(Th=High|Te=L) = 0.1

7

BN inference


Evidence: observation of specific state


Task: compute the posterior probabilities for
query

node(s) given
evidence
.

Th

Y

Flu

Te

Diagnostic


inference

Th

Flu

Y

Te

Causal


inference

Intercausal


inference

Te

Flu

TB

Flu

Mixed


inference

Th

Flu

Te

8

BN software


Commerical packages: Netica, Hugin,
Analytica (all with demo versions)


Free software: Smile, Genie, JavaBayes, …

http://HTTP.CS.Berkeley.EDU/~murphyk/Bayes/bnsoft.
html


Example running Netica software

9

Decision networks


Extension to basic BN for decision making

»
Decision nodes

»
Utility nodes


EU(Action) =



p(o|Action,E) U(o)


o

»
choose action with highest expect utility




Example


10

Elicitation from experts


Variables

»
important variables? values/states?


Structure

»
causal relationships?

»
dependencies/independencies?


Parameters (probabilities)

»
quantify relationships and interactions?


Preferences (utilities)

11

Expert Elicitation Process


These stages are done iteratively


Stops when further expert input is no longer
cost effective


Process is difficult and time consuming.


Current BN tools

»
inference engine

»
GUI


Next generation of BN tools?

BN

EXPERT

BN TOOLS

Domain


EXPERT

12

Knowledge discovery


There is much interest in automated methods
for learning BNS from data

»
parameters, structure (causal discovery)


Computationally complex problem, so current
methods have practical limitations

»
e.g. limit number of states, require variable
ordering constraints, do not specify all arc
directions


Evaluation methods

13

The knowledge engineering process

1. Building the BN

»
variables, structure, parameters, preferences

»
combination of expert elicitation and knowledge discovery

2. Validation/Evaluation

»
case
-
based, sensitivity analysis, accuracy testing

3. Field Testing

»
alpha/beta testing, acceptance testing

4. Industrial Use

»
collection of statistics

5. Refinement

»
Updating procedures, regression testing

14

Case Study: Intelligent tutoring


Tutoring domain: primary and secondary school
students’ misconceptions about decimals


Based on Decimal Comparison Test (DCT)

»
student asked to choose the larger of pairs of decimals

»
different types of pairs reveal different misconceptions


ITS System involves computer games involving
decimals


This research also looks at a combination of expert
elicitation and automated methods (
UAI2001
)

15

Expert classification of Decimal
Comparison Test (DCT) results

Item Type
expert
class
1
0.4
0.35
2
5.736
5.62
3
4.7
4.08
4
0.452
0.45
5
0.4
0.3
6
0.42
0.35
ATE
H
H
H
H
H
H
AMO
H
H
H
L
H
H
MIS
L
L
L
L
L
L
AU
H
H
-
-
-
-
LWH
L
H
L
H
H
H
LZE
L
H
H
H
H
H
LRV
L
H
L
H
H
L
LU
L
H
-
-
-
-
SDF
H
L
H
L
H
H
SRN
H
L
H
L
L
L
SU
H
L
-
-
-
-
UN
-
-
-
-
-
-
“apparent


expert”

“longer is


larger”

“shorter is


larger”

H = high (all correct or only one wrong)

L = low (all wrong or only one correct)

16

The ITS architecture

Adaptive

Bayesian

Network

Decimal
comparison

test
(optional)

Inputs

Computer Games

Generic BN
model of student


Information about
student e.g. age
(optional)


Hidden

number

Flying

photographer

Decimaliens

….

Number
between

Student

Item

Answer

Item

Answer

Classroom

diagnostic test

results (optional)

Classroom

Teaching

Activities

Report

on student

Answer

Item type

New
game


Diagnose
misconception


Predict outcomes


Identify most
useful information

Sequencing

tactics



Select next item
type


Decide to present
help


Decide change to
new game


Identify when
expertise gained

Teacher

System

Controller

Module


Answers


Help

Feedback

Help

17

Expert Elicitation


Variables

»
two classification nodes: fine and coarse (mut. ex.)

»
item types: (i) H/M/L (ii) 0
-
N


Structure

»
arcs from classification to item type

»
item types independent given classification


Parameters

»
careless mistake (3 different values)

»
expert ignorance:
-

in table (uniform distribution)

18

Expert Elicited BN

19

Evaluation process


Case
-
based evaluation

»
experts checked individual cases

»
sometimes, if prior was low, ‘true’ classification did
not have highest posterior (but usually had biggest
change in ratio)


Adaptiveness evaluation

»
priors changes after each set of evidence


Comparison evaluation

»
Differences in classification between BN and
expert rule

»
Differences in predictions between different BNs

20

Comparison evaluation


Development of measure: same classification,
desirable and undesirable re
-
classification


Use item type predictions


Investigation of effect of item type granularity
and probability of careless mistake

21

Comparison: expert BN vs rule


lwh

lze

lrv

lu

sdf

srn

su

ate

amo

mis

au

un

lwh

386

0

0

0

0

0

0

0

0

0

0

0

lze

0

98

0

0

0

0

0

0

0

0

0

0

lrv

10

0

0

0

0

0

0

0

0

0

0

0

lu

6

9

0

54

0

0

0

0

0

0

0

6

sdf

0

0

0

0

83

0

4

0

0

0

0

0

srn

0

0

0

0

0

159

0

0

0

0

0

0

su

0

0

0

0

2

22

40

3

0

0

0

2

ate

0

0

0

0

0

0

0

1050

0

0

0

0

amo

0

0

0

0

0

0

0

0

79

0

0

0

mis

0

0

0

0

0

0

0

0

0

6

0

0

au

9

0

0

0

0

0

0

63

8

0

0

1

un

43

6

0

15

35

14

11

119

26

2

0

66










































Undesirable

Desirable

Same

22

Results


0
-
N

H/M/L


A

S

L

UN

A

S

L

UN

0.22

86.95%

12.56%

0.49%


87.61%

11.98%

0.441%


A

1207

0

9

0

1213

0

0

3

S

3

312

0

0

4

310

0

1

L

0

0

569

0

0

0

557

2

UN

157

71

78

31

150

82

60

45

0.11

88.02

11.12%

0.86%


87.28%

10.71%

2.01%


A

1206

0

9

1

1184

0

23

9

S

3

310

0

2

4

310

0

1

L

0

0

563

6

7

0

557

5

UN

147

60

64

66

139

73

49

76

0.03

89.29%

9.48%

1.23%


91.63%

5.25%

3.12%


A

1202

0

8

6

1173

0

0

43

S

3

308

0

4

0

304

0

11

L

0

0

560

9

0

0

547

22

UN

102

49

80

106

83

9

36

309



Undes.

Desir.

Same

varying

prob. of

careless

mistake

varying granularity of item type: 0
-
N and H/M/L

23

Investigation by Automated
methods


Classification (using SNOB program, based
on MML)


Parameters


Structure (using CaMML)

24

Results

Method
Type
values
Match
Desir.
Change
Undes.
Change
0.22
77.88
20.39
1.72
0.11
82.93
15.63
1.44
0-N
0.03
84.37
11.86
3.78
0.22
80.47
18.71
0.82
0.11
83.91
13.66
2.42
Expert
H/M/L
0.03
90.40
6.48
3.12
79.81
17.60
2.49
72.06
16.00
11.94
SNOB
24 DCT
0-N
H/M/L
72.51
17.03
10.46
Avg
95.97
2.36
1.66
EBN
learned
0-N
H/M/L
Avg
97.63
1.61
0.75
Avg
86.51
5.08
8.41
CaMML
contr.
0-N
H/M/L
Avg
83.48
8.12
8.34
Avg
85.15
5.87
7.92
CaMML
uncontr.
0-N
H/M/L
Avg
92.63
4.61
2.76
25

Another Case Study: Seabreeze
prediction


2000 Honours project, joint with Bureau of
Meteorology (
with Russell Kennett and Kevin Korb
,
PAKDD’2001 paper, TR)


BN network built based on existing simple expert rule


Several years data available for Sydney seabreezes


CaMML (
Wallace and Korb, 1999
) and Tetrad
-
II (
Spirtes et
al. 1993)

programs used to learn BNs from data



Comparative analysis showed automated methods
gave improved predictions.

26

Other BN
-
related projects


DBNS for discrete monitoring
(PhD, 1992)


Approximate BN inference algorithms based on a
mutual information measure for relevance (
with Nathalie
Jitnah, ICONIP97, ECSQARU97, PRICAI98,AI99)


Plan recognition: DBNs for predicting users actions
and goals in an adventure game (
with David Albrecht,

Ingrid Zukerman,

UM97,UMUAI1999,PRICAI2000
)


Bayesian Poker (
with Kevin Korb,

UAI’99, honours students
)

27

Other BN
-
related projects (cont.)


DBNs for ambulation monitoring and fall diagnosis
(
with biomedical engineering,

PRICAI’96
)


Autonomous aircraft monitoring and replanning (
with

Ph.D. student

Tim Wilkin,

PRICAI2000
)


Ecological risk assessment (
2003 honours project with
Water Studies Centre)


Writing a textbook! (
with Kevin Korb
)

Bayesian Artificial Intelligence

28

Open Research Questions


Methodology for combining expert elicitation and
automated methods

»
expert knowledge used to guide search

»
automated methods provide alternatives to be presented to
experts


Evaluation measures and methods

»
may be domain depended


Improved tools to support elicitation

»
e.g. visualisation of d
-
separation


Industry adoption of BN technology