[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp14923.html
   My bibliography  Save this paper

Visual Inference and Graphical Representation in Regression Discontinuity Designs

Author

Listed:
  • Korting, Christina

    (University of Delaware)

  • Lieberman, Carl

    (U.S. Census Bureau)

  • Matsudaira, Jordan

    (Columbia University)

  • Pei, Zhuan

    (Cornell University)

  • Shen, Yi

    (University of Waterloo)

Abstract
Despite the widespread use of graphs in empirical research, little is known about readers' ability to process the statistical information they are meant to convey ("visual inference"). We study visual inference within the context of regression discontinuity (RD) designs by measuring how accurately readers identify discontinuities in graphs produced from data generating processes calibrated on 11 published papers from leading economics journals. First, we assess the effects of different graphical representation methods on visual inference using randomized experiments. We find that bin widths and fit lines have the largest impacts on whether participants correctly perceive the presence or absence of a discontinuity. Incorporating the experimental results into two decision theoretical criteria adapted from the recent economics literature, we find that using small bins with no fit lines to construct RD graphs performs well and recommend it as a starting point to practitioners. Second, we compare visual inference with widely used econometric inference procedures. We find that visual inference achieves similar or lower type I error rates and complements econometric inference.

Suggested Citation

  • Korting, Christina & Lieberman, Carl & Matsudaira, Jordan & Pei, Zhuan & Shen, Yi, 2021. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," IZA Discussion Papers 14923, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp14923
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp14923.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    2. Timothy B. Armstrong & Michal Kolesár, 2018. "Optimal Inference in a Class of Regression Models," Econometrica, Econometric Society, vol. 86(2), pages 655-683, March.
    3. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1593-1660.
    4. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    5. David Card & Andrew Johnston & Pauline Leung & Alexandre Mas & Zhuan Pei, 2015. "The Effect of Unemployment Benefits on the Duration of Unemployment Insurance Receipt: New Evidence from a Regression Kink Design in Missouri, 2003-2013," American Economic Review, American Economic Association, vol. 105(5), pages 126-130, May.
    6. Benabou, Roland & Falk, Armin & Tirole, Jean, 2018. "Narratives, Imperatives, and Moral Reasoning," IZA Discussion Papers 11665, Institute of Labor Economics (IZA).
    7. Joshua Schwartzstein & Adi Sunderam, 2021. "Using Models to Persuade," American Economic Review, American Economic Association, vol. 111(1), pages 276-323, January.
    8. David Card & David S. Lee & Zhuan Pei, 2009. "Quasi-Experimental Identification and Estimation in the Regression Kink Design," Working Papers 1206, Princeton University, Department of Economics, Industrial Relations Section..
    9. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.
    10. David Card & David S. Lee & Zhuan Pei, 2009. "Quasi-Experimental Identification and Estimation in the Regression Kink Design," Working Papers 1206, Princeton University, Department of Economics, Industrial Relations Section..
    11. Cameron, A. Colin & Gelbach, Jonah B. & Miller, Douglas L., 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 238-249.
    12. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Rocío Titiunik, 2019. "Regression Discontinuity Designs Using Covariates," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 442-451, July.
    13. Mahbubul Majumder & Heike Hofmann & Dianne Cook, 2013. "Validation of Visual Statistical Inference, Applied to Linear Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(503), pages 942-956, September.
    14. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    15. Raj Chetty & Nathaniel Hendren & Patrick Kline & Emmanuel Saez, 2014. "Where is the land of Opportunity? The Geography of Intergenerational Mobility in the United States," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(4), pages 1553-1623.
    16. Toru Kitagawa & Shosei Sakaguchi & Aleksey Tetenov, 2021. "Constrained Classification and Policy Learning," Papers 2106.12886, arXiv.org, revised Jul 2023.
    17. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    18. Zhuan Pei & Jörn-Steffen Pischke & Hannes Schwandt, 2019. "Poorly Measured Confounders are More Useful on the Left than on the Right," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 205-216, April.
    19. Andrew Gelman & Guido Imbens, 2019. "Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(3), pages 447-456, July.
    20. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    21. Sebastian Calonico & Matias D Cattaneo & Max H Farrell, 2020. "Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 192-210.
    22. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    23. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    24. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    25. Janet Currie & Henrik Kleven & Esmée Zwiers, 2020. "Technology and Big Data Are Changing Economics: Mining Text to Track Methods," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 42-48, May.
    26. Jinyong Hahn & Petra Todd & Wilbert Van der Klaauw, 1999. "Evaluating the Effect of an Antidiscrimination Law Using a Regression-Discontinuity Design," NBER Working Papers 7131, National Bureau of Economic Research, Inc.
    27. Isaiah Andrews & Matthew Gentzkow & Jesse M. Shapiro, 2020. "Transparency in Structural Research," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 711-722, October.
    28. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    29. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," CESifo Working Paper Series 7262, CESifo.
    30. Zhuan Pei & Yi Shen, 2017. "The Devil is in the Tails: Regression Discontinuity Design with Measurement Error in the Assignment Variable," Advances in Econometrics, in: Regression Discontinuity Designs, volume 38, pages 455-502, Emerald Group Publishing Limited.
    31. Chen, Daniel L. & Schonger, Martin & Wickens, Chris, 2016. "oTree—An open-source platform for laboratory, online, and field experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 9(C), pages 88-97.
    32. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    33. Matthew Rabin & Richard H. Thaler, 2013. "Anomalies: Risk aversion," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 27, pages 467-480, World Scientific Publishing Co. Pte. Ltd..
    34. Fan, Jianqing & Yao, Qiwei, 1998. "Efficient estimation of conditional variance functions in stochastic regression," LSE Research Online Documents on Economics 6635, London School of Economics and Political Science, LSE Library.
    35. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    36. Guido Imbens & Stefan Wager, 2019. "Optimized Regression Discontinuity Designs," The Review of Economics and Statistics, MIT Press, vol. 101(2), pages 264-278, May.
    37. McCrary, Justin, 2008. "Manipulation of the running variable in the regression discontinuity design: A density test," Journal of Econometrics, Elsevier, vol. 142(2), pages 698-714, February.
    38. Ted O'Donoghue & Jason Somerville, 2018. "Modeling Risk Aversion in Economics," Journal of Economic Perspectives, American Economic Association, vol. 32(2), pages 91-114, Spring.
    39. Nickell, Stephen J, 1981. "Biases in Dynamic Models with Fixed Effects," Econometrica, Econometric Society, vol. 49(6), pages 1417-1426, November.
    40. Isaiah Andrews & Jesse M. Shapiro, 2021. "A Model of Scientific Communication," Econometrica, Econometric Society, vol. 89(5), pages 2117-2142, September.
    41. Joshua D. Angrist & Miikka Rokkanen, 2015. "Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1331-1344, December.
    42. Abhijit V. Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2020. "A Theory of Experimenters: Robustness, Randomization, and Balance," American Economic Review, American Economic Association, vol. 110(4), pages 1206-1230, April.
    43. Jeffrey T. Leek & Roger D. Peng, 2015. "Statistics: P values are just the tip of the iceberg," Nature, Nature, vol. 520(7549), pages 612-612, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.
    2. Hanna Virtanen & Mikko Silliman & Tiina Kuuppelomäki & Kristiina Huttunen, "undated". "Education, Gender, and Family Formation," Working Papers 340, Työn ja talouden tutkimus LABORE, The Labour Institute for Economic Research LABORE.
    3. Matias D. Cattaneo & Luke Keele & Rocio Titiunik, 2023. "A Guide to Regression Discontinuity Designs in Medical Applications," Papers 2302.07413, arXiv.org, revised May 2023.
    4. Leung, Pauline, 2022. "State responses to federal matching grants: The case of medicaid," Journal of Public Economics, Elsevier, vol. 216(C).
    5. Guiffard, Jean-Baptiste, 2024. "Valuing the virtual: The impact of fiber to the home on property prices in France," Telecommunications Policy, Elsevier, vol. 48(4).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang He & Otávio Bartalotti, 2020. "Wild bootstrap for fuzzy regression discontinuity designs: obtaining robust bias-corrected confidence intervals," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 211-231.
    2. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    3. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    4. Blaise Melly & Rafael Lalive, 2020. "Estimation, Inference, and Interpretation in the Regression Discontinuity Design," Diskussionsschriften dp2016, Universitaet Bern, Departement Volkswirtschaft.
    5. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    6. Babii, Andrii & Kumar, Rohit, 2023. "Isotonic regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 234(2), pages 371-393.
    7. Yiqi Liu & Yuan Qi, 2023. "Using Forests in Multivariate Regression Discontinuity Designs," Papers 2303.11721, arXiv.org, revised Jul 2024.
    8. Adam C. Sales & Ben B. Hansen, 2020. "Limitless Regression Discontinuity," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 143-174, April.
    9. David Wuepper & Robert Finger, 2023. "Regression discontinuity designs in agricultural and environmental economics," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 50(1), pages 1-28.
    10. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    11. Dor Leventer & Daniel Nevo, 2024. "Correcting invalid regression discontinuity designs with multiple time period data," Papers 2408.05847, arXiv.org.
    12. Dean Eckles & Nikolaos Ignatiadis & Stefan Wager & Han Wu, 2020. "Noise-Induced Randomization in Regression Discontinuity Designs," Papers 2004.09458, arXiv.org, revised Nov 2023.
    13. Chen, Yi & Zhao, Yi, 2022. "The timing of first marriage and subsequent life outcomes: Evidence from a natural experiment," Journal of Comparative Economics, Elsevier, vol. 50(3), pages 713-731.
    14. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    15. Atı̇la Abdulkadı̇roğlu & Joshua D. Angrist & Yusuke Narita & Parag Pathak, 2022. "Breaking Ties: Regression Discontinuity Design Meets Market Design," Econometrica, Econometric Society, vol. 90(1), pages 117-151, January.
    16. Pastore, Chiara & Jones, Andrew M., 2023. "Human capital consequences of missing out on a grammar school education," Economic Modelling, Elsevier, vol. 126(C).
    17. Michihito Ando, 2017. "How much should we trust regression-kink-design estimates?," Empirical Economics, Springer, vol. 53(3), pages 1287-1322, November.
    18. Likai Chen & Georg Keilbar & Liangjun Su & Weining Wang, 2023. "Inference on many jumps in nonparametric panel regression models," Papers 2312.01162, arXiv.org, revised Aug 2024.
    19. Otávio Bartalotti, 2013. "Theory and Practice of Inference in Regression Discontinuity: A Fixed-Bandwidth Asymptotics Approach," Working Papers 1302, Tulane University, Department of Economics, revised Nov 2013.
    20. Prakash, Nishith & Rockmore, Marc & Uppal, Yogesh, 2019. "Do criminally accused politicians affect economic outcomes? Evidence from India," Journal of Development Economics, Elsevier, vol. 141(C).

    More about this item

    Keywords

    graphical methods; visual inference; regression discontinuity design; expert prediction; statistical decision theory; scientific communication;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp14923. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.