# Knowledge Engineering for

AI and Robotics

Nov 7, 2013 (5 years and 5 months ago)

104 views

1

Knowledge Engineering for
Bayesian Networks

Ann Nicholson

School of Computer Science

and Software Engineering

Monash University

2

Overview

Representing uncertainty

Introduction to Bayesian Networks

»
Syntax, semantics, examples

The knowledge engineering process

Case Study: Intelligent Tutoring

Summary of other BN research

Open research questions

3

Sources of Uncertainty

Ignorance

Inexact observations

Non
-
determinism

AI representations

»
Probability theory

»
Dempster
-
Shafer

»
Fuzzy logic

4

Probability theory for
representing uncertainty

Assigns a numerical degree of belief between
0 and 1 to facts

»
e.g. “it will rain today” is T/F.

»
P(“it will rain today”) = 0.2 prior probability
(unconditional)

Posterior probability (conditional)

»
P(“it wil rain today” | “rain is forecast”) = 0.8

Bayes’ Rule:
P(H|E) =
P(E|H) x P(H)

P(E)

5

Bayesian networks

Directed acyclic graphs

Nodes: random variables,

»
R: “it is raining”, discrete values T/F

»
T: temperature, cts
or

discrete variable

»
C: colour, discrete values {red,blue,green}

Arcs indicate dependencies (can have causal
interpretation)

6

Bayesian networks

Conditional Probability Distribution (CPD)

Associated with each variable

probability of each state given parent states

“Jane has the flu”

“Jane has a

high temp”

“Thermometer

X

Flu

Y

Te

Q

Th

Models causal relationship

Models possible sensor error

P(Flu=T) = 0.05

P(Te=High|Flu=T) = 0.4

P(Te=High|Flu=F) = 0.01

P(Th=High|Te=H) = 0.95

P(Th=High|Te=L) = 0.1

7

BN inference

Evidence: observation of specific state

Task: compute the posterior probabilities for
query

node(s) given
evidence
.

Th

Y

Flu

Te

Diagnostic

inference

Th

Flu

Y

Te

Causal

inference

Intercausal

inference

Te

Flu

TB

Flu

Mixed

inference

Th

Flu

Te

8

BN software

Commerical packages: Netica, Hugin,
Analytica (all with demo versions)

Free software: Smile, Genie, JavaBayes, …

http://HTTP.CS.Berkeley.EDU/~murphyk/Bayes/bnsoft.
html

Example running Netica software

9

Decision networks

Extension to basic BN for decision making

»
Decision nodes

»
Utility nodes

EU(Action) =

p(o|Action,E) U(o)

o

»
choose action with highest expect utility

Example

10

Elicitation from experts

Variables

»
important variables? values/states?

Structure

»
causal relationships?

»
dependencies/independencies?

Parameters (probabilities)

»
quantify relationships and interactions?

Preferences (utilities)

11

Expert Elicitation Process

These stages are done iteratively

Stops when further expert input is no longer
cost effective

Process is difficult and time consuming.

Current BN tools

»
inference engine

»
GUI

Next generation of BN tools?

BN

EXPERT

BN TOOLS

Domain

EXPERT

12

Knowledge discovery

There is much interest in automated methods
for learning BNS from data

»
parameters, structure (causal discovery)

Computationally complex problem, so current
methods have practical limitations

»
e.g. limit number of states, require variable
ordering constraints, do not specify all arc
directions

Evaluation methods

13

The knowledge engineering process

1. Building the BN

»
variables, structure, parameters, preferences

»
combination of expert elicitation and knowledge discovery

2. Validation/Evaluation

»
case
-
based, sensitivity analysis, accuracy testing

3. Field Testing

»
alpha/beta testing, acceptance testing

4. Industrial Use

»
collection of statistics

5. Refinement

»
Updating procedures, regression testing

14

Case Study: Intelligent tutoring

Tutoring domain: primary and secondary school

Based on Decimal Comparison Test (DCT)

»
student asked to choose the larger of pairs of decimals

»
different types of pairs reveal different misconceptions

ITS System involves computer games involving
decimals

This research also looks at a combination of expert
elicitation and automated methods (
UAI2001
)

15

Expert classification of Decimal
Comparison Test (DCT) results

Item Type
expert
class
1
0.4
0.35
2
5.736
5.62
3
4.7
4.08
4
0.452
0.45
5
0.4
0.3
6
0.42
0.35
ATE
H
H
H
H
H
H
AMO
H
H
H
L
H
H
MIS
L
L
L
L
L
L
AU
H
H
-
-
-
-
LWH
L
H
L
H
H
H
LZE
L
H
H
H
H
H
LRV
L
H
L
H
H
L
LU
L
H
-
-
-
-
SDF
H
L
H
L
H
H
SRN
H
L
H
L
L
L
SU
H
L
-
-
-
-
UN
-
-
-
-
-
-
“apparent

expert”

“longer is

larger”

“shorter is

larger”

H = high (all correct or only one wrong)

L = low (all wrong or only one correct)

16

The ITS architecture

Bayesian

Network

Decimal
comparison

test
(optional)

Inputs

Computer Games

Generic BN
model of student

student e.g. age
(optional)

Hidden

number

Flying

photographer

Decimaliens

….

Number
between

Student

Item

Item

Classroom

diagnostic test

results (optional)

Classroom

Teaching

Activities

Report

on student

Item type

New
game

Diagnose
misconception

Predict outcomes

Identify most
useful information

Sequencing

tactics

Select next item
type

Decide to present
help

Decide change to
new game

Identify when
expertise gained

Teacher

System

Controller

Module

Help

Feedback

Help

17

Expert Elicitation

Variables

»
two classification nodes: fine and coarse (mut. ex.)

»
item types: (i) H/M/L (ii) 0
-
N

Structure

»
arcs from classification to item type

»
item types independent given classification

Parameters

»
careless mistake (3 different values)

»
expert ignorance:
-

in table (uniform distribution)

18

Expert Elicited BN

19

Evaluation process

Case
-
based evaluation

»
experts checked individual cases

»
sometimes, if prior was low, ‘true’ classification did
not have highest posterior (but usually had biggest
change in ratio)

»
priors changes after each set of evidence

Comparison evaluation

»
Differences in classification between BN and
expert rule

»
Differences in predictions between different BNs

20

Comparison evaluation

Development of measure: same classification,
desirable and undesirable re
-
classification

Use item type predictions

Investigation of effect of item type granularity
and probability of careless mistake

21

Comparison: expert BN vs rule

lwh

lze

lrv

lu

sdf

srn

su

ate

amo

mis

au

un

lwh

386

0

0

0

0

0

0

0

0

0

0

0

lze

0

98

0

0

0

0

0

0

0

0

0

0

lrv

10

0

0

0

0

0

0

0

0

0

0

0

lu

6

9

0

54

0

0

0

0

0

0

0

6

sdf

0

0

0

0

83

0

4

0

0

0

0

0

srn

0

0

0

0

0

159

0

0

0

0

0

0

su

0

0

0

0

2

22

40

3

0

0

0

2

ate

0

0

0

0

0

0

0

1050

0

0

0

0

amo

0

0

0

0

0

0

0

0

79

0

0

0

mis

0

0

0

0

0

0

0

0

0

6

0

0

au

9

0

0

0

0

0

0

63

8

0

0

1

un

43

6

0

15

35

14

11

119

26

2

0

66

Undesirable

Desirable

Same

22

Results

0
-
N

H/M/L

A

S

L

UN

A

S

L

UN

0.22

86.95%

12.56%

0.49%

87.61%

11.98%

0.441%

A

1207

0

9

0

1213

0

0

3

S

3

312

0

0

4

310

0

1

L

0

0

569

0

0

0

557

2

UN

157

71

78

31

150

82

60

45

0.11

88.02

11.12%

0.86%

87.28%

10.71%

2.01%

A

1206

0

9

1

1184

0

23

9

S

3

310

0

2

4

310

0

1

L

0

0

563

6

7

0

557

5

UN

147

60

64

66

139

73

49

76

0.03

89.29%

9.48%

1.23%

91.63%

5.25%

3.12%

A

1202

0

8

6

1173

0

0

43

S

3

308

0

4

0

304

0

11

L

0

0

560

9

0

0

547

22

UN

102

49

80

106

83

9

36

309

Undes.

Desir.

Same

varying

prob. of

careless

mistake

varying granularity of item type: 0
-
N and H/M/L

23

Investigation by Automated
methods

Classification (using SNOB program, based
on MML)

Parameters

Structure (using CaMML)

24

Results

Method
Type
values
Match
Desir.
Change
Undes.
Change
0.22
77.88
20.39
1.72
0.11
82.93
15.63
1.44
0-N
0.03
84.37
11.86
3.78
0.22
80.47
18.71
0.82
0.11
83.91
13.66
2.42
Expert
H/M/L
0.03
90.40
6.48
3.12
79.81
17.60
2.49
72.06
16.00
11.94
SNOB
24 DCT
0-N
H/M/L
72.51
17.03
10.46
Avg
95.97
2.36
1.66
EBN
learned
0-N
H/M/L
Avg
97.63
1.61
0.75
Avg
86.51
5.08
8.41
CaMML
contr.
0-N
H/M/L
Avg
83.48
8.12
8.34
Avg
85.15
5.87
7.92
CaMML
uncontr.
0-N
H/M/L
Avg
92.63
4.61
2.76
25

Another Case Study: Seabreeze
prediction

2000 Honours project, joint with Bureau of
Meteorology (
with Russell Kennett and Kevin Korb
,
PAKDD’2001 paper, TR)

BN network built based on existing simple expert rule

Several years data available for Sydney seabreezes

CaMML (
Wallace and Korb, 1999
-
II (
Spirtes et
al. 1993)

programs used to learn BNs from data

Comparative analysis showed automated methods
gave improved predictions.

26

Other BN
-
related projects

DBNS for discrete monitoring
(PhD, 1992)

Approximate BN inference algorithms based on a
mutual information measure for relevance (
with Nathalie
Jitnah, ICONIP97, ECSQARU97, PRICAI98,AI99)

Plan recognition: DBNs for predicting users actions
and goals in an adventure game (
with David Albrecht,

Ingrid Zukerman,

UM97,UMUAI1999,PRICAI2000
)

Bayesian Poker (
with Kevin Korb,

UAI’99, honours students
)

27

Other BN
-
related projects (cont.)

DBNs for ambulation monitoring and fall diagnosis
(
with biomedical engineering,

PRICAI’96
)

Autonomous aircraft monitoring and replanning (
with

Ph.D. student

Tim Wilkin,

PRICAI2000
)

Ecological risk assessment (
2003 honours project with
Water Studies Centre)

Writing a textbook! (
with Kevin Korb
)

Bayesian Artificial Intelligence

28

Open Research Questions

Methodology for combining expert elicitation and
automated methods

»
expert knowledge used to guide search

»
automated methods provide alternatives to be presented to
experts

Evaluation measures and methods

»
may be domain depended

Improved tools to support elicitation

»
e.g. visualisation of d
-
separation