Proposed studies in evidential reasoning

tripastroturfAI and Robotics

Nov 7, 2013 (3 years and 5 months ago)


Proposed studies in evidential reasoning

Evidence is crucial to many aspects of human cognition. We use it to
update our mental models of the world, and to guide our inferences and
decisions. Psychological studies of judgment and decision reveal a variety

cognitive biases in the gathering, assessment and use of evidence, but with no
unifying framework. In part this is due to the lack of a comprehensive normative
account. Bayesian networks provide such an account

a normative theory of
belief revision a
nd inference, dependency relations, evidence integration, and a
natural link to causal models. The concept of an inference network formalizes the
notion of a mental model, and the graphical representation suggests a
compelling format to aid human inference
, especially in complex situations.

We plan to use Bayesian networks as a normative framework against
which to explore how humans use evidence. This is a necessary step towards a
more unified and systematic model of human reasoning. It will also facilit
ate the
construction of appropriate inference aids where humans deviate from the
normative standard. We do not presume that people’s actual inferential practices
are based on mental ‘Bayesian network’ structures (although this has been
proposed by some, e.
g., Glymour, 2001; Gopnik et al. 2004). Current evidence
suggests that when people represent and reason about uncertainty they adopt
simplifying strategies and heuristics (Gilovich et al., 2002). These will sometimes
approximate sound Bayesian reasoning, b
ut can deviate in systematic ways.

The discovery, integration and use of evidence depend crucially on the
prior beliefs and assumptions of the agent. This interaction will provide an
integrating theme for our psychological experiments. We will examine the

various ways in which people’s prior conceptions (e.g., beliefs, causal models
etc.) and processing mechanisms (e.g., belief revision and information
integration) affect the assimilation of evidence.

1. Evidence gathering

Cognitive biases in how peopl
e search for evidence have been well documented
(but perhaps not well explained).

Confirmation bias.
Many studies suggest that people are prone to a confirmation
bias (or positive test strategy), where they seek only confirmatory evidence for
their favore
d hypotheses (Klayman & Ha, 1987; Wason, 1960, 1968). Some
theorists argue that this is not sub
normative, but that people are engaged in
rational Bayesian inference with certain prior assumptions about the distribution
of events in the environment (McKenz
ie, 2003; Oaksford & Chater, 1994). We
plan to test the plausibility of these claims (by manipulating people’s prior
beliefs and seeing how this affects test strategies), and their applicability to
singular hypothesis testing.

Evidence seeking in decision

Previous research suggests that people flout
basic optimal rules (such as to gather new evidence only if the expected return is
greater than the cost). They collect evidence when they shouldn’t and fail to
collect it when they should (Harvey & Bol
ger, 1999). Further, studies in medical
decision making (Harries, Evans, and Dennis, 1999) show that doctors often
think that they use more types of evidence than they actually do. This implies
that their evidence
seeking behaviour is likely to be unnecess
arily expensive. We
aim to generalize this work to other domains (law, intelligence analysis), and
elaborate the reasons for such evidence
seeking behaviour.

This work will benefit from a clearer picture of exactly what is normative in such
cases (provide
d by other branches of the Evidence project).

2. Evidence integration

This covers a wide range of cognitive activities: the weighting of individual items
of evidence; the integration of multiple items; the affect of prior knowledge or
opinion. Psychologi
cal research has revealed a variety of biases, although here
again there is controversy.

Base rates.

Kahneman & Tversky (1982) demonstrated numerous cases of base
rate neglect, where people under
weighted or ignored base rate information
relevant to the s
olution of a judgment problem. These findings have since been
questioned on both normative and empirical grounds (Koehler, 1996). Factors
that encourage base rate usage include perceived causal relevance (Sloman,
2003), experiential exposure (Lagnado & Sha
nks, 2002), and frequency
representations (Gigerenzer, 2000; Harries & Harvey, 2000). The use of base rate
information is of particular relevance in a Bayesian model because these furnish
the reasoner with priors. We will further investigate what pro
motes base rate
usage, and incorporate these findings into the design of appropriate decision

Conservatism and anchoring.
Another robust finding is conservatism: when
confronted with new evidence people tend to adjust their prior beliefs less than i
warranted (Edwards, 1968). A related tendency is for people to use an accessible
anchor point, and under
adjust their beliefs in relation to this anchor (Tversky &
Kahneman, 1974). Although often run together, these may correspond to distinct

operations (Harvey & Harries, in press). We aim to clarify the nature of
these biases, and uncover the cognitive mechanisms that underlie them.

Sequential belief adjustment and order effects.
Evidence is often encountered in a
sequential fashion. Hogarth
& Einhorn (1992) have developed a general belief
adjustment model that describes certain aspects of human inference very well. It
distinguishes between the process of evaluation (which invokes an additive
model) and estimation (which invokes an averaging m
odel), and between
adjustments that are made as each new item of evidence is received, and those
made after all items are received. The model accounts for various order effects, in
which people’s final judgments depend on the precise order that information

received and integrated. On most Bayesian models this is sub
normative. We will
examine the possibility of order effects in some of the areas addressed in the
Evidence project (e.g., in forensic and criminal investigations), and see what
factors accent
uate or alleviate these effects.

3. Cascaded inference

Inferences are often made on the basis of uncertain evidence, or a previous
uncertain inference. This involves multiple chains of probabilistic inference.
Bayesian networks provide a normative model

here but people appear to adopt
cut strategies. Earlier research (e.g., Steiger, 1972; Gettys, Kelly & Peterson,
1973) suggests that people use ‘As
if’ or ‘Best
guess’ reasoning. That is, they infer
the most probable outcome in the first step, and t
hen condition on this outcome
in the second step, neglecting alternative inferential paths. There is very little
recent research on this, although Lagnado & Shanks (2003) show that people
condition on an initial uncertain categorization in order to make a
inference about a related category, and that this can lead to anomalous
judgments. We plan to re
formulate some of the earlier research questions (by
Schum and colleagues, 1973), and develop a cognitive model of cascaded
inference. We will se
e under what conditions people make inferential errors, and
whether these can be attenuated by ‘inference’ aids. We will also link this work
with questions about order effects.

4. Causal models

Causal knowledge is often used to structure inference proble
ms, and can
influence how beliefs are revised in the light of new evidence. For example, in
studies of jury decision making Pennington and Hastie (1991) found that jurors
construct causal or narrative
based summaries of the presented evidence, and
that the
se representations predicted their verdicts (and confidence in verdicts).
Further, a story
based account was judged more plausible if the order of
evidence receipt matched the real world unfolding of events. A more general
example involves the discounting
(or explaining away) of alternative hypotheses
once a target hypothesis is confirmed (an issue much studied in social
psychology, e.g., Morris & Larrick, 1995). Sloman & Lagnado (2004) show that
the nature and extent of discounting depends on the assumed c
ausal relations
between hypotheses and evidence. Finally, Sloman (2003) has shown that
people’s prior causal models predict whether they give Bayesian responses to the
cab problem. We plan to investigate how people’s causal knowledge affects
their str
ucturing of inference problems, and their integration and use of
evidence. This will be particularly relevant to how people use graphical
inference aids, as the efficacy of these aids depends on appropriate problem

5. Inference aids

As we
ll as providing a framework for normative computations, graphical
representations provide a compelling visual presentation format. They have
already been used in publicly available diagnosis software (e.g., Microsoft
Bayesian Networks). Previous studies h
ave shown that reasoning is facilitated
with appropriate diagrammatic representations (Novick, 2001; Tversky et al.,
2000, Sloman et al., 2003), although no work has been done with Bayesian
networks. Can people use such visual ‘inference aids’ to improve t
heir evidential
reasoning? We will investigate this question, and compare a variety of software
packages (including FLINTS and Maverick (Leary) and any prototypes
developed within the Evidence program).

Experimental paradigms

Ongoing work by other mem
bers of the Evidence project will be used to generate
experimental scenarios and task paradigms. For example, we plan to use
simplified versions of paternity investigations (Dawid), in which participants
make judgments about paternity on the basis of incom
plete evidence from DNA
markers, prior probabilities, etc. We also plan to use modified versions of real
criminal investigation tasks (Leary). Such paradigms provide natural
environments in which to explore various aspects of human inference (e.g., use
multiple sources of evidence; order effects in evidence receipt; evidence
gathering strategies; cascaded inference etc.). They will also facilitate the
comparison of different decision aids.