The Importance of Negative Evidence - Climate-Eval


28 Οκτ 2013 (πριν από 4 χρόνια και 8 μήνες)

89 εμφανίσεις

The Importance of Negative

Rob D. van den Berg

May, 2013

Evaluations should enable “learning from

Negative evidence (“what does not work”) could
help us

Evidence movement focuses on positive evidence

If it does not work: no clue why, just stop funding

If it does work: no certainty on why, just increase

The nature of positive and negative evidence

A framework for integrating negative evidence: a
Theory of No Change (Christine


Every day the Turkey gathers data on food, water
and security:

Safe and secure environment on the farm with a fence
to keep wolves and foxes out

Food and water delivered by the farmer every day

Counterfactual: Turkey’s distant cousin lives in the
wild and faces many uncertainties…

Often on the run from predators

Organized hunts

Food and water availability have wild fluctuations

High probability that life is good for a turkey on a



Food availability

Water availability

Predator threats


Food availability

Water availability

Predator threats

Farm turkey

Wild turkey

off point:
head of the

Many data points on food and water availability and
predator threats

Positive proof that farm turkeys are better off than wild

Farmer cuts off the head of the farm turkey

One event proofs that the “naïve” theory is not correct

Large n provides statistically significant proof for a theory

One n (a black swan event) is more powerful than the
combined might of many n

However large the n is, it will never deliver 100 percent

One n may proof the theory wrong with 100 percent


Causality refers to the relationship between two
events: the cause and the effect, where the second
event is

by the first

Scientific theories predict and explain effects

Early 20

century: logical positivism

Logic guides deductions from general theories to set up tests

Empirical data can provide positive proof of theory

Popper: logical positivism cannot escape the induction
problem of Hume

However many data you gather, it will never constitute
positive proof that the theory is right

The proof that is scientifically and logically sound is


Challenge is to

a theory


Logical positivism is no longer in vogue in the

Testing of medicine is based on logical positivism and
has been adopted as the “gold standard” by the
evidence movement

Naïve positivism has been replaced with nuanced
positivism that poses a null hypothesis that should be
disproved; however, this still delivers “positive” proof
the treatment works

Health, Education and Economics are heavily
influenced; development has followed

Large n, divided in two groups (with/without
intervention) is needed for evidence


Explanatory power: zero difference in n

Theory that explains more is accepted

Example: fractal geophysics (chaos theory) versus linear

Occam’s razor: zero difference in n

Theory that is simple wins against theory that is complicated

Example: Copernicus versus Ptolemy

Predictive power: one n may suffice

Special theory of relativity was proven through one
observation of gravitational pull on light during a solar eclipse

Falsifying a theory: one n may suffice

One black swan will disproof the theory that all swans are
white (Popper)


Data on natural or human phenomena over time

Can establish historical trends

the more n the better

Modeling of large n through macro
economic or other

Mathematical approach to “what if” questions

the more n
the better

Natural experimentation; also known as quasi

Large n is welcome but often difficult to find

Randomized controlled trials

Large n is welcome but costly and difficult to control

Systemic reviews

Sifting through large n to find relevant n


The term evidence increasingly refers to outcome
of research/studies

Hierarchies of evidence (Campbell collaboration,
Maryland hierarchy) focus on large n only

Evidence based on n=1 or no difference in n is no
longer recognized as such in some of the literature
of the “evidence

Sciences that use n=1 or no difference in n tend to
not be less in policy discussions and provide hardly
any countervailing perspectives


Causality in research focuses on


proof or disproof causal linkages that are predicted
by theory

Causality in evaluations also tackles

and is not focused on proof or disproof of scientific
theories, but on what works and why

Interventions take place in a mixed environment of
scientific and technical certainties, unproven
theories and scientifically unknown territory

Identification of possible causal linkages takes
place through a theory based approach



A theory based approach may lead to a theory of
change identifying causal linkages and assumptions
covering these

This may also lead to an identification of what could
possibly prevent these causal linkages from “working”

It may also identify what prevents the intervention as a
whole to move forward

Analogy: a car needs many working components to
function as a car, but take away the wheels and it will
stop moving

Identification of these factors leads to a “theory of no


Systemic Reviews go through existing evidence in
research and evaluations from the perspective of a
specific question

Are cash transfers effective in promoting school attendance?

Many studies and evaluations do not address this
question in the exact same way and are thus not
accepted as evidence

Health review: only 50 studies accepted from 49.000

Other forms of meta
evaluations do not pose
restrictive questions but pose to explore existing

All quality evaluations on a subject are accepted; and quality
evidence in a bad evaluation may also be accepted

Theory based approach


Over to Christine