The Importance of Negative Evidence - Climate-Eval

oppositemincedΔιαχείριση

28 Οκτ 2013 (πριν από 3 χρόνια και 10 μήνες)

66 εμφανίσεις


The Importance of Negative
Evidence


Rob D. van den Berg

May, 2013


Evaluations should enable “learning from
mistakes”


Negative evidence (“what does not work”) could
help us


Evidence movement focuses on positive evidence


If it does not work: no clue why, just stop funding


If it does work: no certainty on why, just increase
funding


The nature of positive and negative evidence


A framework for integrating negative evidence: a
Theory of No Change (Christine
W
o
erlen
)

2


Every day the Turkey gathers data on food, water
and security:


Safe and secure environment on the farm with a fence
to keep wolves and foxes out


Food and water delivered by the farmer every day


Counterfactual: Turkey’s distant cousin lives in the
wild and faces many uncertainties…


Often on the run from predators


Organized hunts


Food and water availability have wild fluctuations


High probability that life is good for a turkey on a
farm

3

4

Food availability

Water availability

Predator threats

5

Food availability

Water availability

Predator threats

Farm turkey

Wild turkey

Cut
-
off point:
head of the
Turkey


Many data points on food and water availability and
predator threats


Positive proof that farm turkeys are better off than wild
turkeys


Farmer cuts off the head of the farm turkey


One event proofs that the “naïve” theory is not correct


Large n provides statistically significant proof for a theory


One n (a black swan event) is more powerful than the
combined might of many n


However large the n is, it will never deliver 100 percent
proof


One n may proof the theory wrong with 100 percent
certainty

6


Causality refers to the relationship between two
events: the cause and the effect, where the second
event is
caused

by the first


Scientific theories predict and explain effects


Early 20
th

century: logical positivism


Logic guides deductions from general theories to set up tests


Empirical data can provide positive proof of theory


Popper: logical positivism cannot escape the induction
problem of Hume


However many data you gather, it will never constitute
positive proof that the theory is right


The proof that is scientifically and logically sound is
negative

proof


Challenge is to
falsify

a theory


7


Logical positivism is no longer in vogue in the
natural
sciences


Testing of medicine is based on logical positivism and
has been adopted as the “gold standard” by the
evidence movement


Naïve positivism has been replaced with nuanced
positivism that poses a null hypothesis that should be
disproved; however, this still delivers “positive” proof
the treatment works


Health, Education and Economics are heavily
influenced; development has followed


Large n, divided in two groups (with/without
intervention) is needed for evidence


8


Explanatory power: zero difference in n


Theory that explains more is accepted


Example: fractal geophysics (chaos theory) versus linear
geophysics


Occam’s razor: zero difference in n


Theory that is simple wins against theory that is complicated


Example: Copernicus versus Ptolemy


Predictive power: one n may suffice


Special theory of relativity was proven through one
observation of gravitational pull on light during a solar eclipse


Falsifying a theory: one n may suffice


One black swan will disproof the theory that all swans are
white (Popper)

9


Data on natural or human phenomena over time


Can establish historical trends


the more n the better


Modeling of large n through macro
-
economic or other
theories


Mathematical approach to “what if” questions


the more n
the better


Natural experimentation; also known as quasi
-
experimental


Large n is welcome but often difficult to find


Randomized controlled trials


Large n is welcome but costly and difficult to control


Systemic reviews


Sifting through large n to find relevant n

10


The term evidence increasingly refers to outcome
of research/studies


Hierarchies of evidence (Campbell collaboration,
Maryland hierarchy) focus on large n only


Evidence based on n=1 or no difference in n is no
longer recognized as such in some of the literature
of the “evidence
-
movement”


Sciences that use n=1 or no difference in n tend to
not be less in policy discussions and provide hardly
any countervailing perspectives


11


Causality in research focuses on
new

subjects


to
proof or disproof causal linkages that are predicted
by theory


Causality in evaluations also tackles
old

subjects
and is not focused on proof or disproof of scientific
theories, but on what works and why


Interventions take place in a mixed environment of
scientific and technical certainties, unproven
theories and scientifically unknown territory


Identification of possible causal linkages takes
place through a theory based approach


12

13


A theory based approach may lead to a theory of
change identifying causal linkages and assumptions
covering these


This may also lead to an identification of what could
possibly prevent these causal linkages from “working”


It may also identify what prevents the intervention as a
whole to move forward


Analogy: a car needs many working components to
function as a car, but take away the wheels and it will
stop moving


Identification of these factors leads to a “theory of no
change”

14


Systemic Reviews go through existing evidence in
research and evaluations from the perspective of a
specific question


Are cash transfers effective in promoting school attendance?


Many studies and evaluations do not address this
question in the exact same way and are thus not
accepted as evidence


Health review: only 50 studies accepted from 49.000


Other forms of meta
-
evaluations do not pose
restrictive questions but pose to explore existing
evidence


All quality evaluations on a subject are accepted; and quality
evidence in a bad evaluation may also be accepted


Theory based approach

15

Over to Christine
Woerlen
!