Power and Pitfalls of Experiments in Development Economics

minorbigarmΑσφάλεια

30 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

62 εμφανίσεις

Presented at the annual meetings of the

Agricultural and Applied Economics Association (AAEA)

July 2010

Denver, Colorado

Power and Pitfalls of Experiments in
Development Economics:

Some Non

牡湤r洠剥晬散瑩潮t


Christopher B. Barrett

Michael R. Carter

Experiments now wildly popular in economics overall.


So what role do they play in development economics?
Subject of considerable dispute currently (Deaton and
Heckman vs.
Banerjee
,
Duflo

and
Imbens
)


We study human beings as agents whose choices,
conditioned by the external environment, result in
behaviors that matter not just to their own well
-
being
but also, due to externalities and general equilibrium
effects, to the aggregate experience of their
communities
.


Overview

Ultimately development economists
interested in welfare and behavioral
outcomes:



y=f(
s,p,b,e,α,ε
) and b= g(
s,p,y,e,θ,φ
)


s = structural variables


p = policy variables


e =
elicitable

behavioral characteristics


α/θ = unobservable characteristics


Estimation challenges:
(
i
) simultaneity of b and y,


(ii) unknown true functional form


(iii) the common non
-
orthogonality

of (
α+ε
)

and the
b,p,s

and e variables of primary interest.


Overview


Two different threads of experimental
work to address these challenges:


1)
Behavioral field experiments to elicit e or to
identify ∂b/∂p cleanly.


2) Randomized controlled trials (or social
experiments) aim to identify ∂y/∂p cleanly.

2. The Power of Experiments:


Behavioral field experiments …


1)
Substantially reduce unobserved heterogeneity and
endogeneity

concerns. Ex: key risk, time and trust
preference parameters. Carter and Castillo (2005)
study of Hurricane Mitch in Honduras.


2) Can replicate realistic choice settings to test key
hypotheses. Ex: BDM auctions to identify
effectiveness of free distribution of insecticide treated
bednets

in Uganda (Hoffmann et al. 2009).


… remain underemployed in development economics.

2. The Power of Experiments:


RCTs… in contrast to BE, these are now dominant in
empirical
dev’t

economics


1)
May resolve econometric problems associated with
program placement and selection effects as well as
the
endogeneity

of key p or s variables.


2) Use random assignment to evaluate pilots and do
more rigorous program evaluations. Huge donor
demand for more evidence of near
-
term impacts.


2. The Pitfalls of Experiments:


A) Internal Validity

The supposed trump claim of experiments, esp. RCTs,
often not as strong as claimed.


1)
Randomization bias

2)
Faux
exogeneity

where true treatment effects are
subjective and
unmeasurable

3)
Ideal designs commonly compromised in field
implementation, esp. by non
-
research partners


2. The Pitfalls of Experiments:


B) External Validity

Unobservable and observable features inevitably vary
at community level and cannot be controlled for in
experimental design because context matters.


1)
Non
-
random implementation partner

2)
Essential heterogeneity


experimental results are
unknown weighted average of heterogeneous
responses … may be no external population that
matches the sample mean from the RCT!


2. The Pitfalls of Experiments:


C) Asking The Right Questions?

Only a few relevant topics are amenable to randomization. Omits
macro, political, GE, negative shocks questions.


1)
Limited ability to study phenomena with lagged effects (e.g.,
early childhood interventions)

2)
RCTs only identify the mean treatment effect. Often want other
characteristics of
dist’n

of effects: conditional effects, proportion
+/
-
, etc.

3)
Crucial distinction between
efficacy

(the study of a treatment’s
capacity to have an effect under controlled conditions) and
effectiveness (
real world impact). Overcorrection for
endogeneity

can render findings consistently and
unbiasedly

irrelevant.



Serious risks of distorting research agendas … economics
reduced to evaluation … often of points obvious to laypeople.

2. The Pitfalls of Experiments:


D) What Opportunity Cost?


1)
Behavioral experiments aim to identify and explain
variation in behaviors. Good.


2)
RCTs compare against the “no intervention” counterfactual.
Infeasible to do multi
-
factorial randomized block design of
high order dimensionality. Leads to distorted
recommendations (example:
deworming
).

2. The Pitfalls of Experiments:


E) Ethical Concerns


Class 1: Predictably violate basic “do no harm” obligation. IRBs
not sufficient. Example: Bertrand et al. (2007 QJE)


Class 2: Ignore responsibility to secure informed consent.
RCTs commonly blind subjects to intervention to avoid
endogenous behavioral responses.


Class 3: Suspend targeting principle, compromising the
expected effectiveness of resources expended in addressing
social ills. Explicitly ignore local information.

1.
Evolution of capital access theory and policy


Monopolistic perspective


public banking & interest rate regulation


Laissez faire

in theory & practice


Imperfect information & the search for collateral substitutes

2.
Evaluation of credit markets & interventions is difficult


Double selection in credit markets heightens concerns over separating
the impact of capital access from the impact of the characteristics of
those with credit


Fundamental Identification problem of (potentially) disequilibrium
market: Observed transactions (loan or no loan) do not allow complete
sorting of observations into correct behavioral regime:










b
i


u
p
i


c
x
i

[
e
i


i
]
i
f
D
i

S
i

u
r
i


u
z
i

[
e
i


i
]
i
f
D
i

S





3. Access to Capital as Central Issue in
Development Economics

3. Access to Capital as Central Issue in
Development Economics

1.
Key empirical questions regarding capital access:


Does the financial market work such that we find households in the non
-
price
rationed regimes or do markets work in an efficient, price
-
rationed manner for all?


If there is non
-
price rationing, is it systematically biased against any particular set of
households (
e.g
., low wealth households) such that the operation of the competitive
economy tends to reinforce initial levels of poverty and inequality?


How costly is non
-
price rationing and how much would household input use and
income increase if liquidity constraints could be relaxed and non
-
price rationing
eliminated?


Are there contractual or institutional innovations that can change the rules of access
to capital, lessen non
-
price rationing and decrease its cost?

2.
Econometric analysis of observational data has long struggled to
answer these questions:


Control for latent characteristics (panel and
Heckmanesque

methods)


Getting the regime sorting (regression heterogeneity) right by getting beyond
naïve approaches

3.
RCT and behavioral experiments seem potentially valuable

indeed success claimed for
RCTs

in realm of credit.





RCTs & Existence of Credit Rationing

1.
Theory suggests that under asymmetric information, interest rate
increases can cause lenders to decline because these price increases
induce adverse selection & moral hazard. Would therefore expect
lenders to self
-
impose interest rate ceilings, resulting in excess
demand at that rate & market clearing via quantity rationing.

2.
If correct, this would imply that higher interest rates should result in
more default and lower profit via either adverse selection or moral
hazard.

3.
In an innovative experiment,
Karlan

&
Zinman

(
Econometrica
, 2010)
worked with a South African paycheck lender to randomize the
interest rate to see if higher rates had this impact


Existing clients invited to borrow at an announced rate


Some were then offered a lower rate when they applied


Find very little evidence that higher rates influence default






RCTs

& Existence of Credit Rationing

1.
At best this strategy would only take us part way towards
understanding the existence and incidence of non
-
price rationing as
it only operated on an existing base of borrowers and would say
nothing about those that already rationed out

2.
While no study can do it all, there is another level at which we may
wonder about the meaning of such a price variation experiment:


In a companion paper,
Karlan

&
Zinman

(
AER

2008) use this data to the
price elasticity of demand for credit


Find a kink in the relationship around the ‘normal’ market price: demand
responds to prices above that price, but does not respond to prices below
that price


One explanation could be that the some of the treated did not find the
announced lower price credible (there is no free lunch!)


From this perspective, we have to ask if we can really randomize things
like prices that have a social meaning and depend on the subject’s
perceptions. Was this a case of faux randomization where the true
treatment received was endogenously determined by the subject’s
education and sophistication?





Randomized Liquidity Injections 1: Santa
Claus Treatment Effects

1.
Normal credit selection processes inexorably make access to capital
correlated with hard to observe borrower characteristics, leading to
identification problems

2.
De Mel,
MacKenzie

& Woodruff (
QJE
, 2009) suggest solving this
problem by simply randomly distributing liquidity injections to small
scale entrepreneurs in Sri Lanka.

3.
While solves one of the identification problems noted above, it does not
deal with the regime switching which has been a central preoccupation of
the empirical literature for 20 years.

4.
Similar to some of the older naïve econometric literature, this approach
only identifies the impact of capital under the assumption that all
businesses were in an excess demand regime

5.
If not true, then estimated treatment effects are an unknown, data
-
weighted average of multiple regression regimes

6.
From a policy perspective, does not tell us what to expect if credit access
were expanded through normal banking, as opposed to ‘Santa Claus’
mechanisms






Randomized Liquidity Injections: Take 2

1.
One seeming way to circumvent this problem of the de Mel et al.
study would be to give liquidity injections only to those who reveal
excess demand by applying for loans

2.
Working with the same paycheck lender mentioned above,
Karlan

and
Zinman

(
Rev of Fin Studies
, 2009) created a design to de
-
ration
some randomly selected applicants whose credit scores deemed them
credit unworthy

3.
However, because this study was creating real debt (unlike Santa
Claus liquidity gifts)
,
several problems ensued:


Loan officers failed to comply with random assignment almost 50% of the time


Experiment exposed borrowers to real harm if they could not repay the debt

a non
-
trivial concern given that the lender’s scoring model in fact predicted repayment
problems for these de
-
rationed agents


Under standard human subjects protection protocols, experiments like this would require
full disclosure (no deception) and procedures to compensate for any harm caused by the
treatment


However, implementing such standard protections would likely invalidate the results of
the study (telling borrowers that their experimentally induced debt would be repaid in
case of default would clearly alter repayment incentives and behavior)


Again see how economic experimentation is very different than, say, drug trials





Purging the Error Term with
Behavioral Experiments

1.
While the
RCTs

just discussed try to break the problematic
correlation between error term and capital access by randomizing the
latter, an alternative approach is use behavioral experiments to make
unobservables

like risk aversion, business acumen and time
preferences observable and thereby purge the error term of its most
problematic part

2.
Binswanger’s early and influential field experiments on risk aversion
(
EJ
, 1980;
AJAE
, 1981) suggest that this can be done (though his own
results were problematic)

3.
A few studies are moving in this direction, but still much to be done
on credible measurement of these kinds of characteristics





Changing the Structural Determinants of
Capital Access

1.
Theory offers a number of insights into the structural conditions
(risk, asymmetric information, etc.) that create non
-
price rationing

2.
An ambitious approach seen in some recent studies is to innovate and
implement new contracts and institutions designed to change these
fundamentals and induce new rules of credit access:


McIntosh, de
Janvry

&
Sadoulet

(
EJ

2010) on credit bureaus in Guatemala


Gine

and Yang (
JDE
?)on

biometrics in credit markets in Malawi


I4 Index Insurance Innovation Initiative projects in Peru (Boucher,
Trivelli

&
Carter), Ethiopia (McIntosh, Sarris & Ahmad) and Mali (
Bellemer
,
Guirkinger

&
Carter)

3.
All of these rely on real institutions (hard!) and spatially randomized
rollout

4.
Stay tuned!





Conclusions

1.
Challenge for contemporary development economics is to
keep its balance of rigor and relevance:


Experiments rightly play a powerful role in modern development
economics in complementing theory and observational data to
understand underlying structure and enable descriptive,
predictive and prescriptive analysis.


Behavioral experiments, in particular, can play a still
-
greater role
in teasing out credible estimates of otherwise
-
unobservable
parameters that matter to estimating behavioral and welfare
response.

2.
Some rollback of the RCT obsession would be welcome.
There is no ‘gold standard’ of perfect identification, and
we need to beware of the blind pursuit of exogenous
variation lest it crucify development economics on a cross
of
golden irrelevance.





Thank you for your comments