[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2303.11399.html
   My bibliography  Save this paper

How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on Over 60 Replicated Studies

Author

Listed:
  • Apoorva Lal
  • Mac Lockhart
  • Yiqing Xu
  • Ziwen Zu
Abstract
Instrumental variable (IV) strategies are widely used in political science to establish causal relationships. However, the identifying assumptions required by an IV design are demanding, and it remains challenging for researchers to assess their validity. In this paper, we replicate 67 papers published in three top journals in political science during 2010-2022 and identify several troubling patterns. First, researchers often overestimate the strength of their IVs due to non-i.i.d. errors, such as a clustering structure. Second, the most commonly used t-test for the two-stage-least-squares (2SLS) estimates often severely underestimates uncertainty. Using more robust inferential methods, we find that around 19-30% of the 2SLS estimates in our sample are underpowered. Third, in the majority of the replicated studies, the 2SLS estimates are much larger than the ordinary-least-squares estimates, and their ratio is negatively correlated with the strength of the IVs in studies where the IVs are not experimentally generated, suggesting potential violations of unconfoundedness or the exclusion restriction. To help researchers avoid these pitfalls, we provide a checklist for better practice.

Suggested Citation

  • Apoorva Lal & Mac Lockhart & Yiqing Xu & Ziwen Zu, 2023. "How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on Over 60 Replicated Studies," Papers 2303.11399, arXiv.org, revised Nov 2023.
    • Handle: RePEc:arx:papers:2303.11399
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2303.11399
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hager, Anselm & Krakowski, Krzysztof, 2022. "Does State Repression Spark Protests? Evidence from Secret Police Surveillance in Communist Poland," American Political Science Review, Cambridge University Press, vol. 116(2), pages 564-579, May.
    2. Croke, Kevin & Grossman, Guy & Larreguy, Horacio A. & Marshall, John, 2016. "Deliberate Disengagement: How Education Can Decrease Political Participation in Electoral Authoritarian Regimes," American Political Science Review, Cambridge University Press, vol. 110(3), pages 579-600, August.
    3. Ana L. De La O, 2013. "Do Conditional Cash Transfers Affect Electoral Behavior? Evidence from a Randomized Experiment in Mexico," American Journal of Political Science, John Wiley & Sons, vol. 57(1), pages 1-14, January.
    4. Maurice J. G. Bun & Frank Windmeijer, 2010. "The weak instrument problem of the system GMM estimator in dynamic panel data models," Econometrics Journal, Royal Economic Society, vol. 13(1), pages 95-126, February.
    5. Chernozhukov, Victor & Hansen, Christian, 2008. "The reduced form: A simple approach to inference with weak instruments," Economics Letters, Elsevier, vol. 100(1), pages 68-71, July.
    6. Yuhki Tajima, 2013. "The Institutional Basis of Intercommunal Order: Evidence from Indonesia's Democratic Transition," American Journal of Political Science, John Wiley & Sons, vol. 57(1), pages 104-119, January.
    7. Marcelo J. Moreira, 2003. "A Conditional Likelihood Ratio Test for Structural Models," Econometrica, Econometric Society, vol. 71(4), pages 1027-1048, July.
    8. Marbach, Moritz & Hangartner, Dominik, 2020. "Profiling Compliers and Noncompliers for Instrumental-Variable Analysis," Political Analysis, Cambridge University Press, vol. 28(3), pages 435-444, July.
    9. Gerber, Alan S. & Huber, Gregory A. & Washington, Ebonya, 2010. "Party Affiliation, Partisanship, and Political Beliefs: A Field Experiment," American Political Science Review, Cambridge University Press, vol. 104(4), pages 720-744, November.
    10. Dieterle, Steven G. & Snell, Andy, 2016. "A simple diagnostic to investigate instrument validity and heterogeneous effects when using a single instrument," Labour Economics, Elsevier, vol. 42(C), pages 76-86.
    11. Baccini, Leonardo & Weymouth, Stephen, 2021. "Gone For Good: Deindustrialization, White Voter Backlash, and US Presidential Voting," American Political Science Review, Cambridge University Press, vol. 115(2), pages 550-567, May.
    12. Laitin, David D. & Ramachandran, Rajesh, 2016. "Language Policy and Human Development," American Political Science Review, Cambridge University Press, vol. 110(3), pages 457-480, August.
    13. José Luis Montiel Olea & Carolin Pflueger, 2013. "A Robust Test for Weak Instruments," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(3), pages 358-369, July.
    14. Leah C. Stokes, 2016. "Electoral Backlash against Climate Policy: A Natural Experiment on Retrospective Voting and Local Resistance to Public Policy," American Journal of Political Science, John Wiley & Sons, vol. 60(4), pages 958-974, October.
    15. Gwyneth H. McClendon, 2014. "Social Esteem and Participation in Contentious Politics: A Field Experiment at an LGBT Pride Rally," American Journal of Political Science, John Wiley & Sons, vol. 58(2), pages 279-290, April.
    16. Dower, Paul Castañeda & Finkel, Evgeny & Gehlbach, Scott & Nafziger, Steven, 2018. "Collective Action and Representation in Autocracies: Evidence from Russia’s Great Reforms," American Political Science Review, Cambridge University Press, vol. 112(1), pages 125-147, February.
    17. Anselm Hager & Hanno Hilbig, 2019. "Do Inheritance Customs Affect Political and Social Inequality?," American Journal of Political Science, John Wiley & Sons, vol. 63(4), pages 758-773, October.
    18. Matthew Adam Kocher & Thomas B. Pepinsky & Stathis N. Kalyvas, 2011. "Aerial Bombing and Counterinsurgency in the Vietnam War," American Journal of Political Science, John Wiley & Sons, vol. 55(2), pages 201-218, April.
    19. Blattman, Christopher & Hartman, Alexandra C. & Blair, Robert A., 2014. "How to Promote Order and Property Rights under Weak Rule of Law? An Experiment in Changing Dispute Resolution Behavior through Community Education," American Political Science Review, Cambridge University Press, vol. 108(1), pages 100-120, February.
    20. Alexander Coppock & Donald P. Green, 2016. "Is Voting Habit Forming? New Evidence from Experiments and Regression Discontinuities," American Journal of Political Science, John Wiley & Sons, vol. 60(4), pages 1044-1062, October.
    21. Ritter, Emily Hencken & Conrad, Courtenay R., 2016. "Preventing and Responding to Dissent: The Observational Challenges of Explaining Strategic Repression," American Political Science Review, Cambridge University Press, vol. 110(1), pages 85-99, February.
    22. Allison Carnegie & Nikolay Marinov, 2017. "Foreign Aid, Human Rights, and Democracy Promotion: Evidence from a Natural Experiment," American Journal of Political Science, John Wiley & Sons, vol. 61(3), pages 671-683, July.
    23. Miguel R. Rueda, 2017. "Small Aggregates, Big Manipulation: Vote Buying Enforcement and Collective Monitoring," American Journal of Political Science, John Wiley & Sons, vol. 61(1), pages 163-177, January.
    24. Erling Barth & Henning Finseraas & Karl O. Moene, 2015. "Political Reinforcement: How Rising Inequality Curbs Manifested Welfare Generosity," American Journal of Political Science, John Wiley & Sons, vol. 59(3), pages 565-577, July.
    25. Renard Sexton & Rachel L. Wellhausen & Michael G. Findley, 2019. "How Government Reactions to Violence Worsen Social Welfare: Evidence from Peru," American Journal of Political Science, John Wiley & Sons, vol. 63(2), pages 353-367, April.
    26. Hainmueller, Jens & Mummolo, Jonathan & Xu, Yiqing, 2019. "How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice," Political Analysis, Cambridge University Press, vol. 27(2), pages 163-192, April.
    27. Kåre Vernby, 2013. "Inclusion and Public Policy: Evidence from Sweden’s Introduction of Noncitizen Suffrage," American Journal of Political Science, John Wiley & Sons, vol. 57(1), pages 15-29, January.
    28. Li, Quan & Reuveny, Rafael, 2003. "Economic Globalization and Democracy: An Empirical Analysis," British Journal of Political Science, Cambridge University Press, vol. 33(1), pages 29-54, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Romagnoli, Matteo, 2024. "Clean sweep: Electricity liberalization and the direction of technological change in the electricity sector," Research Policy, Elsevier, vol. 53(8).
    2. Erdem Dogukan Yilmaz & Tim Meyer & Milan Miric, 2023. "Preventing Others from Commercializing Your Innovation: Evidence from Creative Commons Licenses," Papers 2309.00536, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antoine, Bertille & Lavergne, Pascal, 2023. "Identification-robust nonparametric inference in a linear IV model," Journal of Econometrics, Elsevier, vol. 235(1), pages 1-24.
    2. Bertille Antoine & Pascal Lavergne, 2020. "Identification-Robust Nonparametric Interference in a Linear IV Model," Discussion Papers dp20-03, Department of Economics, Simon Fraser University.
    3. Wang, Wenjie, 2021. "Wild Bootstrap for Instrumental Variables Regression with Weak Instruments and Few Clusters," MPRA Paper 106227, University Library of Munich, Germany.
    4. Keith Finlay & Leandro M. Magnusson, 2019. "Two applications of wild bootstrap methods to improve inference in cluster‐IV models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 34(6), pages 911-933, September.
    5. Murray Michael P., 2017. "Linear Model IV Estimation When Instruments Are Many or Weak," Journal of Econometric Methods, De Gruyter, vol. 6(1), pages 1-22, January.
    6. Keith Finlay & Leandro M. Magnusson, 2014. "Bootstrap Methods for Inference with Cluster-Sample IV Models," Economics Discussion / Working Papers 14-12, The University of Western Australia, Department of Economics.
    7. Luis R. Martinez & Jonas Jessen & Guo Xu, 2023. "A Glimpse of Freedom: Allied Occupation and Political Resistance in East Germany," American Economic Journal: Applied Economics, American Economic Association, vol. 15(1), pages 68-106, January.
    8. Mueller, Hannes & Rauh, Christopher, 2018. "Reading Between the Lines: Prediction of Political Violence Using Newspaper Text," American Political Science Review, Cambridge University Press, vol. 112(2), pages 358-375, May.
    9. Bensch, Gunther & Gotz, Gunnar & Peters, Jörg, 2020. "Effects of rural electrification on employment: A comment on Dinkelman (2011)," Ruhr Economic Papers 840, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    10. Wenjie Wang & Yichong Zhang, 2021. "Wild Bootstrap for Instrumental Variables Regressions with Weak and Few Clusters," Papers 2108.13707, arXiv.org, revised Jan 2024.
    11. Pearce Edwards, 2021. "The politics of nonviolent mobilization: Campaigns, competition, and social movement resources," Journal of Peace Research, Peace Research Institute Oslo, vol. 58(5), pages 945-961, September.
    12. Omar Al-Ubaydli & Faith Fatchen & John List, 2024. "Using Field Experiments to Understand the Impact of Institutions on Economic Growth," Natural Field Experiments 00787, The Field Experiments Website.
    13. David Cingranelli & Skip Mark & Mark Gibney & Peter Haschke & Reed Wood & Daniel Arnon, 2019. "Human Rights Violations and Violent Internal Conflict," Social Sciences, MDPI, vol. 8(2), pages 1-33, January.
    14. María Angelica Bautista & Felipe González & Luis R. Martínez & Pablo Muñoz & Mounu Prem, 2018. "The Geography of Repression and Support for Democracy: Evidence from the Pinochet Dictatorship," Working papers 5, Red Investigadores de Economía.
    15. Nazrullaeva, Eugenia & Harrison, Mark, 2023. "If You Do Not Change Your Behavior: Preventive Repression in Lithuania under Soviet Rule," CAGE Online Working Paper Series 664, Competitive Advantage in the Global Economy (CAGE).
    16. Kai Jäger, 2020. "When Do Campaign Effects Persist for Years? Evidence from a Natural Experiment," American Journal of Political Science, John Wiley & Sons, vol. 64(4), pages 836-851, October.
    17. Regis Barnichon & Geert Mesters, 2020. "Identifying Modern Macro Equations with Old Shocks," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 135(4), pages 2255-2298.
    18. Konstantinos Angelopoulos & George Economides, 2008. "Fiscal policy, rent seeking, and growth under electoral uncertainty: theory and evidence from the OECD," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1375-1405, November.
    19. Gandjon Fankem, Gislain Stéphane & Feyom, Cédric, 2024. "Does trade openness improve or worsen public governance in sub-Saharan Africa?," International Economics, Elsevier, vol. 178(C).
    20. Elizabeth U. Cascio & Na'ama Shenhav, 2020. "A Century of the American Woman Voter: Sex Gaps in Political Participation, Preferences, and Partisanship since Women's Enfranchisement," Journal of Economic Perspectives, American Economic Association, vol. 34(2), pages 24-48, Spring.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2303.11399. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.