Assessing Instability in the Information Age: Managing Overwhelming Information with Simple Models

fancyfantasicΤεχνίτη Νοημοσύνη και Ρομποτική

7 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

149 εμφανίσεις



Assessing Instability in the

Information Age: Managing Overwhelming Information

with Simple Models

Daniel T. Maxwell, Ph.D.


Evidence Based Research

maxwell@ebrinc.com

Presented

to:

ISMOR

August, 2004

Abstract


Pred
icting and avoiding violent conflict
which result from governmental
instability is a long standing goal of analysts supporting modern diplomacy. Historically,
completing this analysis has required labor intensive searches for relevant data,
significant am
ounts of expert judgment

to interpret the data, allowing only
infrequent
updates to
very aggregate

models. The advent of the information age has caused an
explosion in the number of sources an analyst must
,

can
,

and should search for relevant
data;

conseq
ue
ntly forcing a reduction

the amount of time available for analysis and for
collaboration with other experts regarding effective response strategies. This paper
outlines an approach that automates the search and initial information screening process
usin
g
available information technology and a simple Bayes
ian

Network. The result is
that analyst labor can be
quickly
focused on the most relevant information with
significantly less

labor. Additionally, this research points to other simple Operations
Resear
ch techniques which can be applied to large quantities of structured data for the
purpose of increasing efficiency, and consequently the effectiveness of
the
policy
analys
i
s

process
. And, finally the paper presents a set of
open research questions which
a
re critical to addressing large quantities of data with models and other automated
techniques.

Background


There are many ongoing and historical efforts aimed at predicting and ultimately
preventing violent conflict and instability of all sorts. (DFID, 20
03 &
Scarborough, et.
al, 1998) Most of these efforts combine some level of social science modeling and
manual information collection with collaboration among concerned policy professionals
in an attempt to provide early appropriate intervention

with the
goal of preventin
g
a
conflict.


The arrival of the information age as well as a multi
-
polar world is challenging
these traditional techniques in two key

ways. First, the numbers of potential areas of
instability have increased significantly both geographi
cally and culturally. Second, the
amount of information potentially available for analysts to consider in conducting their
research
has exploded with the growth of the internet. Instead of episodic discussions of
issues in journals, occasional news, and
site visits as primary sources, the internet has a
seemingly infinite supply of information through news services, organizational web sites,
and
chat rooms

to identify just a few. The challenge for analysts has become to keep up


with the deluge of informa
tion and to focus on the right areas and iss
ues at the right time.
Overcoming
this challenge is necessary if analysts are to meet the needs
of the leaders
they serve as they attempt to meet the policy challenges of the twenty first century.

Overview

The r
esearch describ
ed herein is an early attempt to

meet the challenges described
above. Similar to early operations research, this
effort
is the result of a multi
-
disciplinary
effort. The areas of expertise supporting this effort consist of political scienc
e, providing
insight into a set of variables that have historically been useful in identifying, predicting,
and understanding political instability. Expertise from commercial competitive
intelligence provided insight into relevant open source information,

and conceptual ideas
on how to harvest that information. Informati
on technology provided the automated
power

necessary to identify, organize and assemble the information processing
infrastructure. And, operations research provided modeling experti
se to
implement the
prototype
Bayesian Network

which is supporting research and experimentation
.

In general, pieces of this approach are in use supporting competitive intelligence

(Shaker
,

2003
) and

ongoing
policy analysis.

And, visionaries into better dec
ision

making
and policy support have been indicating for some time that such automated and
potentially distributed systems are both possible and desirable. (Alberts & Hayes, 2003)

This research begins to explore the art of the possible with respect to effecti
vely
employing computers and relatively simple models to review
,
organize
,

and prioritize
large bodies of data. This allows analysts to increase the amount of time spent analyzing
the most relevant information
and collaborating

with other analysts and sta
keholders
,
thereby increasing the effectiveness and efficiency of their efforts.

The entire process is best described sequentially. The next section

describes how
the information infrastructure collects, organizes, and structures the data for use by the
m
odel. That is followed by a discussion of the baseline
Bayes net
model and its
intellectual
origins in prior political science modeling.

The next set of sections describe
how the model might behave over time and
provides some ideas for specific applicati
ons.
The paper concludes with a summary of conclusions we can already draw and perhaps
most importantly thoughts on future research that might aid in the maturation of these
concepts.



Information Collection and Processing


There are three key steps to
information collection and processing. The

process begins
,
as highlighted in Figure 1,

with the collection of unstructured open source information
.
This information is then structured into event records whose attri
butes are formally
defined in a supporting

ontology.
Records are then processed through a linguistic
processing engine which is seeking to identify
instances of specific types of events which
are relevant evidence for the Bayes network model. These events are then uniquely
identified and stored i
n a data base
. This provides a very highly resolved data stream for
use
upon demand

by th
e Bayes net, or other modeling and analysis tools which might
consume that type of data.






Documents

Open Source

All Source

DB

Pars
ing

Linguistic

Processing

Extract /

Transform

KB

Rule

Development

H
H
i
i
g
g
h
h


R
R
e
e
s
s
o
o
l
l
u
u
t
t
i
i
o
o
n
n


D
D
a
a
t
t
a
a


S
S
t
t
r
r
e
e
a
a
m
m


DB



Figure 1: Information Collection and Processing Architecture

The curren
t project
has

a data set of over 10,000 articles

and grows daily
. From
these 10,000 articles a set of approximately 75 rules have identified
approximately 250
events
spanning a two year period which are believed to be

relevant evidence for the
model. Thi
s is accomplished over the entire data set in a
matter of a few hours on a
single high end windows machine. This level of processing speed allows for new da
ta to
be processed daily in a ma
tter of minutes, and for potentially useful new rules to be
applie
d against the entire data set in a matter of hours.



Systematic collection and analysis of this magnitude is usually cost prohibitive because
of the
number of labor hours required.

Accomplishing this level of functionality does
require some investm
ent in

software.

For instance, at EBR we use primarily
Intellisonar™ to perform the collection and initial processing and AeroText™ to perform
the text structuring.
Additionally, effective implementation of this capability requires a
significant effort to ensure there ar
e reliable and efficient proce
sses in place supporting
the au
tomation.
Fore those interested in more detail,
Shaker and Richardson

(2004)
provides a more complete

description of this process
.

B
aseline
Bayes Net M
odel


Figure 2 below depicts a ten node Bay
esian network

which evaluates the
significance of reported events. Of the ten nodes in the current baseline model, there are
five hypothesis nodes and five evidence nodes informing the hypothesis nodes.
Specifically, the hypotheses relate to stability (o
r instability) due to: individual public
dissent, group public dissent, demonstrations, turmoil, and political violence.
These
variables were selected though interaction with domain experts in political science


currently on staff at EBR, an
d are grounded
in earlier model
research

on instability
(Scarborough,

1999).


For those unfamiliar with
Bayesian Network
s, they are

essentially graph based
representation
s

of probability tree
s
. This visualization, supported by
efficient
computational algorithms allow
s for the representation and analysis of extremely large
probability (as well as decision) models. The interested reader is referred to Maxwell
,
1998 for a more complete description of Bayesia
n Networks and their
key characteristics.


Figure 2
: Baseline
Bayes Net Model


The

model

s hypothesis nodes have two states, stable and unstable. And the
evidence nodes have two states favorable and unfavorable. While this is a very coarse
model, for this initial research we are first
attempting

to demonstrate an a
bility to
automatically move in the correct direction in response to evidence.

Once proper
direction of movement can be confirmed on
e

can work with domain experts to increase
the resolution of the model.


The a priori prob
abilities in this initial model

a
re set to fifty percent for each state.
This is done for simplicity in
explanation
and
for early testing. In point of fact, because
the model is designed to process a data stream over time, the evidence nodes will settle in
to the observed frequency of e
vidence over time.
The enduring numbers in the model are
the conditional probability distributions which are contained in the hypothesis nodes.
They effectively provide a
n embedded

weighting

function

over the evidence
, and will
affect the movement of the

observed probability (stable, unstable) over time.


It is this
set
of
distribution
s

which should be the focus of discussion
between the OR Analyst doing
model development and the Political Scientist with Domain expertise.

A significant
literature exists
on how these models can be effectively and efficiently elicited,
constructed, and executed. (Howard, 1990;
Heckerman, 1990;
Maxwell, 1994)





Instability Assessment Process


We have now describ
ed all of the components of this approach. Figure 3
illustrates

how data flows through the automated system and into the Bayesian inference
model.
As illustrated, the first step in the process is to harvest documents from all
sources. Currently, the process uses
Quigo
’s Intellisonar
™ to access
a combination
publicly available and subscription sources to generate input records
. There appear to be
no significant technical obstacles to incorporating other sources of data.

The next step
in the process identifies events of interest us
ing a linguistic processing engine; currently
Aerotext™.

The process is initiated with “training” of the system. Domain experts, with
skills similar to the historical readers and “coders” in prior efforts,
develop rules which
allow the software to identi
fy events of interest. In this case we are interested in
populating the five evidence nodes identified above.


Documents

Open Source

All Source

DB

Parsing

Derive

Events

Extract /

Transform

KB

Rule

Development

B
B
a
a
y
y
e
e
s
s
i
i
a
a
n
n


I
I
n
n
f
f
e
e
r
r
e
e
n
n
c
c
e
e


M
M
o
o
d
d
e
e
l
l


DB


Figure 3
: Instability Assessment Process

Figure 4 below illustrates a “find” by the software. The text on the right hand
side of the screen po
ssesses properties which the domain expert has identified as a dissent
event. This causes a set of rules to fire in the software and the information is further
structured, and key details are organized for potential use later
. The model currently only
us
es the “Dissent_event” field of the data. The remaining information has some
important properties for the domain analyst. First, the processing approach is traceable
back to the source data from some interesting result in the Bayes Network. Second, the
data is searchable through other means. One could envision a specialized “google”
search engine
which
allows the analyst to search this specialized data set much more
efficiently than traditional open source search.



This extraction process is then foll
owed by a straightforward software engineering
step in which the key information is organized into a format which the Bayes Network


software will understand. For this prototype, the conversion and transfer process is being
accomplished manually. To date,

we have explored approximately 3 years of data using
quarterly time steps for updating the model. The current rule set in Aerotext
™ identifies
negative events. Favorable evidence reports are generated for each variable which has no
instances of negative evidence in that time period. Over time, recurring favorable reports
increase the expectation of a favorable report in the next ti
me step and have the effect of
increasing the model’s sensitivity to unexpected negative evidence.



Event

classificati
on

Textual data of

interest


Figure 4
: Linguistic Processing Approach

Sample
Model
R
esults

Figure 5 below illustrates the change in model value that results in this model
given
se
t
of evidence reporting
unfavorable information along four of the evidence nodes
.

The likelihood of instability grows from the a priori fifty percent to the 62.6 percent
which
is observed in the red ellipse
. Currently, this
value
should not,

and

in fact
can not
,

be strictly interpreted to be the probability of political instability in the future, given
political violence today. Rather, this value should be though of as an index score which
points to an increasing likelihood of political instability, give
n the evidence that is
passing through the model.

This should stimulate analysts into focused research and
collaboration concerning the source data at the root of the problem.



Currently the model is designed to be very sensitive to unfavorable evidence.
Our
logic for this is we would prefer initially to have analysts responding to “false positive”
model states. This stimulates interaction with the model developers, as well as a more
aggressive review of the linguistic rules which are at the foundation of

event generation.



Changes Probability of

Inst
ability


Figure 5
: Model Result for 1

Time
Step

Figure 6 below is a depiction of what the model’s output might look like to an
analyst over time.
Immediately observable is that probability is being replaced by an
index score. Analysts coul
d observe trends over time on virtually any variable in the
model. In this particular visualization we are looking at simulated results over ten time
steps of the model. This simulation pointed to two possible ways in which one might
alert a domain exper
t. First, as shown at time step two, the index value exceeds a
threshold which can be set by the domain expert. The
second

is a three time step long
increase in the index

score. This would facilitate an alert based on a pattern of increasing
activity, b
ut an absolute level on the index that had not yet achieved a critical value.

There are two important features to this approach. First, traceability is maintained
from the model to the data which caused the model behavior
. So, when an alert occurs
a
do
main analyst can

query the source data for the exact informat
ion which caused the
alert. This allows domain experts to efficiently access and use the material with which
they are most comfortable. It also provides a very useful feedback loop to

modelers

and
rule developers f
o
r

refining

the model and the data stream populating it. The second
important feature is this structure appears to lend itself beautifully to other supporting
simple models is an OR analysts toolkit. For instance, this type of data
stream over time
is ideally suited to techniques found in statistical process control currently supporting
manufactu
ring types of applications.






Probability of Violent Instability
-10
-5
0
5
10
15
1
2
3
4
5
6
7
8
9
10
11
Time
Probability
Alert
Desk
Officer

Figure 6
: A sample “D
ashboard” for a Domain Analyst

Conclusions and Recommended Future Research


The researc
h completed in support of this prototype has
already produced some
interesting findings. First, the automated rules appear

to effectively “trigger”
in response
to
areas of interest

which are meaningful to domain experts. Second,
the approach
overall prov
ides consistent and traceable results for all concerned. The source data is
linked to model behavior for use by both domain experts, and OR oriented model
developers. This increases immediate usability. Additionally,

the Bayesian Network is
provably cor
rect in dealing with probabilistic evidence, and is relatively explained to
domain experts so that one could develop user trust in the techniques.


One large potential benefit of an automated approach would be its ability to deal
with massive data, and ope
rate across a range of topic areas and geographical regions.
Other applications of this approach are already effectively
continuously
collecting and
processing

virtually thousands of sources on a weekly basis in matters of hours using
high end desk top co
mputing to power the software. This is very positive evidence for
believing this concept could be extended to process large quantities of data across many
countries, regions, or perhaps special interest subjects.


While the research has to date has had so
me very promising results, there are
some known limitations which in turn point to some potentially fruitful areas of research.
First, the prototype
is exploring a very limited data set. Currently the model is addressing
one country using readily availab
le data spanning approximately five years time.
Exploration of these concepts would benefit from access to broader sets of sources
spanning longer time horizons, as well as application over a broader sample of countries,


regions, and political genres. It

is anticipated that this type of empirical work will result
in the

development of multiple models, each with strengths and weaknesses. It is the set
of models supported by a rich data stream will provide the “power” of the concept.


Currently there is no

adjudication of information quality. This means that reliable
and unreliable data is passing through the model.
Schum, 1994 provides a good
foundation for thinking about issues of evidence in support of these concepts. Noble,
2004 presents a Bayesian i
mplementation which could be integrated into this

approach in
a relatively straight forward manner.


While all of the components of this approach are relatively mature, and proven,
within their respective disciplines this collectively is an instantiation

o
f an i
mmature

collective

t
heory
. There is much work to do at the seams of our collective disciplines
to
develop sound empirical evidence that confirms or refutes the positive assertions
contained in this paper. Exploring and understanding these seams is

absolutely essential
to effectively employing information
technology to the policy challenges of the twenty
first century.

References

Alberts, D & Hayes R. (2003)
Power to the Edge,

Command and Control R
ese
a
r
ch
Program, Washington D.C.

Cooper, G.F. (1990)

"Th
e Computational Complexity of of

Probabilistic Inference Using
Bayesian Belief Networks"
Artificial Intelligence,
Vol. 42, pp.393
-
405.

DFID, UK MOD

(2003)
, “The Global Conflict Prevention Pool: A Joint UK Government
Approach”
.

Heckerman, D. E. (1990) P
robabilistic Similarity Networks, Ph.D. dissertation, MIS
Department, Stanford University, Stanford, Ca.

Howard, R.A. (1990) "From Influence to Relevance to Knowledge"
,

in

Influence
Diagrams, Belief Nets and Decision Analysis
,
Oliver & Smith (eds.), John Wiley & Sons,
London.

Maxwell, D.

(1994) Composing and Constructing Value Focused Influence Diagrams: A
Specification for Decision Model Formulation, Ph.D. dissertation, SITE, George Mason
University.

Maxwell, D.
(1996)
“Support
ing Decision
-
Makers in Future Conflicts: A Decision
Theoretic Perspective,
Analytical Approaches to the Study of Future Conflict
, Woodcock
& Davis ed., The Canadian Peacekeeping Press.

Maxwell, D. (1998),
“What Every Analyst Should Know About Bayesian Netwo
rks”,
Phalanx ,
Vol. 31, No. 1.

Noble, David F. (2004),

Assessing the Reliability of O
pen Source Information”,
XXX



Scarborough
, G (1999
), “An Expert System for Assessing Vulnerability to Instability”,
Unpublished manuscript.

Scarborough, G., Ulfelder, J &

Kay, B (1998), “Forecasting Political Instability in a
Three Year Period: Technical Report
”, Unpublished Manuscript.

Schum, D. (1994)
The Evidential Foundations of Probabilistic Reasoning,
Northwestern
University Press, Evanston.

Shaker, S. (2003)

Connec
ting the Dots: War Room Team
-
Based Analysis
”,

InterSymp
2
003 Proceedings of the focus symposium on Collaborative Decision
-
Support Systems.


Shaker, S. & Richardson, V. (2004), “Putting the System Back Into Early Warning”,
SCIP, Vol. 7, no.
3, pp. 14
-
17. (ww
w.scip.org)