[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/hal/cesptp/hal-03098058.html
   My bibliography  Save this paper

Policy Evaluation Using Causal Inference Methods

Author

Listed:
  • Denis Fougère

    (CNRS - Centre National de la Recherche Scientifique, OSC - Observatoire sociologique du changement (Sciences Po, CNRS) - Sciences Po - Sciences Po - CNRS - Centre National de la Recherche Scientifique, LIEPP - Laboratoire interdisciplinaire d'évaluation des politiques publiques (Sciences Po) - Sciences Po - Sciences Po, CEPR - Center for Economic Policy Research, IZA - Forschungsinstitut zur Zukunft der Arbeit - Institute of Labor Economics)

  • Nicolas Jacquemet

    (PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École des Ponts ParisTech - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, UP1 - Université Paris 1 Panthéon-Sorbonne, CES - Centre d'économie de la Sorbonne - UP1 - Université Paris 1 Panthéon-Sorbonne - CNRS - Centre National de la Recherche Scientifique)

Abstract
This chapter describes the main impact evaluation methods, both experimental and quasi-experimental, and the statistical model underlying them. Some of the most important methodological advances to have recently been put forward in this field of research are presented. We focus not only on the need to pay particular attention to the accuracy of the estimated effects, but also on the requirement to replicate assessments, carried out by experimentation or quasi-experimentation, in order to distinguish false positives from proven effects.

Suggested Citation

  • Denis Fougère & Nicolas Jacquemet, 2021. "Policy Evaluation Using Causal Inference Methods," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-03098058, HAL.
    • Handle: RePEc:hal:cesptp:hal-03098058
    Note: View the original document on HAL open archive server: https://hal.science/hal-03098058
    as

    Download full text from publisher

    File URL: https://hal.science/hal-03098058/document
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    2. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107016064, September.
    3. Lechner, Michael, 2018. "Modified Causal Forests for Estimating Heterogeneous Causal Effects," IZA Discussion Papers 12040, Institute of Labor Economics (IZA).
    4. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    5. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    6. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107638105, September.
    7. Joshua D. Angrist, 2004. "Treatment effect heterogeneity in theory and practice," Economic Journal, Royal Economic Society, vol. 114(494), pages 52-83, March.
    8. Steven Glazerman & Dan M. Levy & David Myers, 2003. "Nonexperimental Versus Experimental Estimates of Earnings Impacts," The ANNALS of the American Academy of Political and Social Science, , vol. 589(1), pages 63-93, September.
    9. List, John A, et al, 2001. "Academic Economists Behaving Badly? A Survey on Three Areas of Unethical Behavior," Economic Inquiry, Western Economic Association International, vol. 39(1), pages 162-170, January.
    10. Charles F. Manski, 2013. "Identification of treatment response with social interactions," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-23, February.
    11. Nicolas Jacquemet & Olivier L'Haridon, 2018. "Experimental Economics. Method and Applications," Post-Print halshs-02303376, HAL.
    12. Yingying Dong & Arthur Lewbel, 2015. "Identifying the Effect of Changing the Policy Threshold in Regression Discontinuity Models," The Review of Economics and Statistics, MIT Press, vol. 97(5), pages 1081-1092, December.
    13. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    14. Michael C Knaus & Michael Lechner & Anthony Strittmatter, 2021. "Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence," The Econometrics Journal, Royal Economic Society, vol. 24(1), pages 134-161.
    15. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    16. Duncan D. Chaplin & Thomas D. Cook & Jelena Zurovac & Jared S. Coopersmith & Mariel M. Finucane & Lauren N. Vollmer & Rebecca E. Morris, 2018. "The Internal And External Validity Of The Regression Discontinuity Design: A Meta‐Analysis Of 15 Within‐Study Comparisons," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 37(2), pages 403-429, March.
    17. Michael C. Knaus & Michael Lechner & Anthony Strittmatter, 2022. "Heterogeneous Employment Effects of Job Search Programs: A Machine Learning Approach," Journal of Human Resources, University of Wisconsin Press, vol. 57(2), pages 597-636.
    18. A. Belloni & V. Chernozhukov & I. Fernández‐Val & C. Hansen, 2017. "Program Evaluation and Causal Inference With High‐Dimensional Data," Econometrica, Econometric Society, vol. 85, pages 233-298, January.
    19. L. Liu & M. G. Hudgens & S. Becker-Dreps, 2016. "On inverse probability-weighted estimators in the presence of interference," Biometrika, Biometrika Trust, vol. 103(4), pages 829-842.
    20. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    21. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    22. James Bisbee & Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2017. "Local Instruments, Global Extrapolation: External Validity of the Labor Supply-Fertility Local Average Treatment Effect," Journal of Labor Economics, University of Chicago Press, vol. 35(S1), pages 99-147.
    23. Valentin Amrhein & Sander Greenland & Blake McShane, 2019. "Scientists rise up against statistical significance," Nature, Nature, vol. 567(7748), pages 305-307, March.
    24. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    25. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    26. Angrist, Joshua D. & Krueger, Alan B., 1999. "Empirical strategies in labor economics," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 23, pages 1277-1366, Elsevier.
    27. Sobel, Michael E., 2006. "What Do Randomized Studies of Housing Mobility Demonstrate?: Causal Inference in the Face of Interference," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1398-1407, December.
    28. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107016057, September.
    29. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107674165, September.
    30. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    31. Lan Liu & Michael G. Hudgens, 2014. "Large Sample Randomization Inference of Causal Effects in the Presence of Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(505), pages 288-301, March.
    32. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    33. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    34. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    35. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    36. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    37. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    38. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107627314, September.
    39. Acemoglu,Daron & Arellano,Manuel & Dekel,Eddie (ed.), 2013. "Advances in Economics and Econometrics," Cambridge Books, Cambridge University Press, number 9781107016040, September.
    40. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    41. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    42. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    43. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    44. Marinho Bertanha & Guido W. Imbens, 2020. "External Validity in Fuzzy Regression Discontinuity Designs," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(3), pages 593-612, July.
    45. Hong, Guanglei & Raudenbush, Stephen W., 2006. "Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 901-910, September.
    46. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    47. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "Inference on Treatment Effects after Selection among High-Dimensional Controlsâ€," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(2), pages 608-650.
    48. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    49. Rachael Meager, 2019. "Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 11(1), pages 57-91, January.
    50. Lee, Myoung-jae, 2016. "Matching, Regression Discontinuity, Difference in Differences, and Beyond," OUP Catalogue, Oxford University Press, number 9780190258740.
    51. Ziliak, Stephen T. & McCloskey, Deirdre N., 2004. "Size matters: the standard error of regressions in the American Economic Review," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 527-546, November.
    52. Farrell, Max H., 2015. "Robust inference on average treatment effects with possibly more covariates than observations," Journal of Econometrics, Elsevier, vol. 189(1), pages 1-23.
    53. Joshua D. Angrist & Kathryn Graddy & Guido W. Imbens, 2000. "The Interpretation of Instrumental Variables Estimators in Simultaneous Equations Models with an Application to the Demand for Fish," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 67(3), pages 499-527.
    54. Carolina Perez-Heydrich & Michael G. Hudgens & M. Elizabeth Halloran & John D. Clemens & Mohammad Ali & Michael E. Emch, 2014. "Assessing effects of cholera vaccination in the presence of interference," Biometrics, The International Biometric Society, vol. 70(3), pages 731-741, September.
    55. Sarah Baird & J. Aislinn Bohren & Craig McIntosh & Berk Özler, 2018. "Optimal Design of Experiments in the Presence of Interference," The Review of Economics and Statistics, MIT Press, vol. 100(5), pages 844-860, December.
    56. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    57. Rigdon, Joseph & Hudgens, Michael G., 2015. "Exact confidence intervals in the presence of interference," Statistics & Probability Letters, Elsevier, vol. 105(C), pages 130-135.
    58. Meager, Rachael, 2019. "Understanding the average impact of microcredit expansions: a Bayesian hierarchical analysis of seven randomized experiments," LSE Research Online Documents on Economics 88190, London School of Economics and Political Science, LSE Library.
    59. repec:mpr:mprres:3694 is not listed on IDEAS
    60. Mathias Lundin & Maria Karlsson, 2014. "Estimation of causal effects in observational studies with interference between units," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 23(3), pages 417-433, August.
    61. Sofrygin Oleg & van der Laan Mark J., 2017. "Semi-Parametric Estimation and Inference for the Mean Outcome of the Single Time-Point Intervention in a Causally Connected Population," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-35, March.
    62. Joshua D. Angrist & Miikka Rokkanen, 2015. "Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1331-1344, December.
    63. Jinyong Hahn & Ruoyao Shi, 2017. "Synthetic Control and Inference," Econometrics, MDPI, vol. 5(4), pages 1-12, November.
    64. Roth, Alvin E, 1994. "Lets Keep the Con out of Experimental Econ.: A Methodological Note," Empirical Economics, Springer, vol. 19(2), pages 279-289.
    65. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    66. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ferentinos, Konstantinos & Gibberd, Alex & Guin, Benjamin, 2021. "Climate policy and transition risk in the housing market," Bank of England working papers 918, Bank of England.
    2. Ruben Fotso, 2022. "Evaluating the indirect effects of cluster-based innovation policies: the case of the Technological Research Institutes in France," The Journal of Technology Transfer, Springer, vol. 47(4), pages 1070-1114, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Denis Fougère & Nicolas Jacquemet, 2019. "Causal Inference and Impact Evaluation," Economie et Statistique / Economics and Statistics, Institut National de la Statistique et des Etudes Economiques (INSEE), issue 510-511-5, pages 181-200.
    2. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    3. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    4. Committee, Nobel Prize, 2021. "Answering causal questions using observational data," Nobel Prize in Economics documents 2021-2, Nobel Prize Committee.
    5. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    6. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    7. Chen, Xuan & Flores, Carlos A. & Flores-Lagunes, Alfonso, 2015. "Going Beyond LATE: Bounding Average Treatment Effects of Job Corps Training," IZA Discussion Papers 9511, Institute of Labor Economics (IZA).
    8. Meager, Rachael, 2019. "Understanding the average impact of microcredit expansions: a Bayesian hierarchical analysis of seven randomized experiments," LSE Research Online Documents on Economics 88190, London School of Economics and Political Science, LSE Library.
    9. Guido W. Imbens, 2022. "Causality in Econometrics: Choice vs Chance," Econometrica, Econometric Society, vol. 90(6), pages 2541-2566, November.
    10. Tomasz Serwach, 2022. "The European Union and within-country income inequalities. The case of the New Member States," Working Papers hal-03548416, HAL.
    11. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    12. Kaspar Wüthrich, 2020. "A Comparison of Two Quantile Models With Endogeneity," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(2), pages 443-456, April.
    13. Bo, Hao & Galiani, Sebastian, 2021. "Assessing external validity," Research in Economics, Elsevier, vol. 75(3), pages 274-285.
    14. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    15. Brian Fabo & Martina Jancokova & Elisabeth Kempf & Lubos Pastor, 2020. "Fifty Shades of QE: Conflicts of Interest in Economic Research," Working Papers 2020-128, Becker Friedman Institute for Research In Economics.
    16. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    17. Yang, Jui-Chung & Chuang, Hui-Ching & Kuan, Chung-Ming, 2020. "Double machine learning with gradient boosting and its application to the Big N audit quality effect," Journal of Econometrics, Elsevier, vol. 216(1), pages 268-283.
    18. Tarek Azzam & Michael Bates & David Fairris, 2019. "Do Learning Communities Increase First Year College Retention? Testing Sample Selection and External Validity of Randomized Control Trials," Working Papers 202002, University of California at Riverside, Department of Economics.
    19. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    20. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.

    More about this item

    Keywords

    Causal inference; Evaluation methods; Causal effects;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C2 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables
    • C3 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables
    • C54 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Quantitative Policy Modeling

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:cesptp:hal-03098058. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.