Natural Experiment Policy Evaluation: A Critique

Ilya Strebulaev and Christopher Hennessy

No 10455, CEPR Discussion Papers from C.E.P.R. Discussion Papers

Abstract: We argue exogenous random treatment is insufficient for valid inference regarding the sign and magnitude of causal effects in dynamic environments. In such settings, treatment responses must be understood as contingent upon the typically unmodeled policy generating process. With binary assignment, this results in quantitatively significant attenuation bias. With more than two policy states, treatment responses can be biased downward, upward, or have the wrong sign. Further, it is not only generally invalid to extrapolate elasticities across policy processes, as argued by Lucas (1976), but also to extrapolate within the same policy process. We derive auxiliary assumptions beyond exogeneity for valid inference in dynamic settings. If all possible policy transitions are rare events, treatment responses approximate causal effects. However, reliance on rare events is overly-restrictive as the necessary and sufficient conditions for equality of treatment responses and causal effects is that policy variable changes have mean zero. If these conditions are not met, we show how treatment responses can nevertheless be corrected and mapped back to causal effects or extrapolated to forecast responses to future policy changes.

Keywords: Natural experiments; Policy evaluations; Dynamic environments; Random treatment; Causal effects (search for similar items in EconPapers)
Date: 2015-03
New Economics Papers: this item is included in nep-mfd
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7) Track citations by RSS feed

Downloads: (external link)
https://cepr.org/publications/DP10455 (application/pdf)
CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cpr:ceprdp:10455

Ordering information: This working paper can be ordered from
https://cepr.org/publications/DP10455

Access Statistics for this paper

More papers in CEPR Discussion Papers from C.E.P.R. Discussion Papers Centre for Economic Policy Research, 33 Great Sutton Street, London EC1V 0DX.
Bibliographic data for series maintained by ().