[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/pri/indrel/638.html
   My bibliography  Save this paper

Visual Inference and Graphical Representation in Regression Discontinuity Designs

Author

Listed:
  • Christina Korting

    (Cornell University)

  • Carl Lieberman

    (Princeton University)

  • Jordan Matsudaira

    (Columbia University)

  • Zhuan Pei

    (Cornell University)

  • Yi Shen

    (University of Waterloo)

Abstract
Despite the widespread use of graphs in empirical research, little is known about readers' ability to process the statistical information they are meant to convey ("visual inference"). In this paper, we evaluate several key aspects of visual inference in regression discontinuity (RD) designs by measuring how well readers can identify discontinuities in graphs. First, we assess the effects of graphical representation methods on visual inference, using randomized experiments crowdsourcing discontinuity classifications with graphs produced from data generating processes calibrated on datasets from 11 published papers. Second, we evaluate visual inference by both experts and non-experts and study experts’ ability to predict our experimental results. We find that experts perform comparably to non-experts and partly anticipate the effects of graphical methods. Third, we compare experts’ visual inference to commonly used econometric procedures in RD designs and observe that it achieves similar or lower type I error rates. Fourth, we conduct an eyetracking study to further understand RD visual inference, but it does not reveal gaze patterns that robustly predict successful inference. We also evaluate visual inference in the closely related regression kink design.

Suggested Citation

  • Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2020. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," Working Papers 638, Princeton University, Department of Economics, Industrial Relations Section..
    • Handle: RePEc:pri:indrel:638
    as

    Download full text from publisher

    File URL: https://dataspace.princeton.edu/bitstream/88435/dsp013j3335157/3/638.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Timothy B. Armstrong & Michal Kolesár, 2018. "Optimal Inference in a Class of Regression Models," Econometrica, Econometric Society, vol. 86(2), pages 655-683, March.
    4. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Rocío Titiunik, 2019. "Regression Discontinuity Designs Using Covariates," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 442-451, July.
    5. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    6. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," NBER Working Papers 25072, National Bureau of Economic Research, Inc.
    7. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1593-1660.
    8. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    9. David Card & Andrew Johnston & Pauline Leung & Alexandre Mas & Zhuan Pei, 2015. "The Effect of Unemployment Benefits on the Duration of Unemployment Insurance Receipt: New Evidence from a Regression Kink Design in Missouri, 2003-2013," American Economic Review, American Economic Association, vol. 105(5), pages 126-130, May.
    10. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(2), pages 238-249, April.
    11. Benabou, Roland & Falk, Armin & Tirole, Jean, 2018. "Narratives, Imperatives, and Moral Reasoning," IZA Discussion Papers 11665, Institute of Labor Economics (IZA).
    12. Joshua Schwartzstein & Adi Sunderam, 2021. "Using Models to Persuade," American Economic Review, American Economic Association, vol. 111(1), pages 276-323, January.
    13. Sebastian Calonico & Matias D Cattaneo & Max H Farrell, 2020. "Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 192-210.
    14. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    15. David Card & David S. Lee & Zhuan Pei, 2009. "Quasi-Experimental Identification and Estimation in the Regression Kink Design," Working Papers 1206, Princeton University, Department of Economics, Industrial Relations Section..
    16. Matthew Rabin & Richard H. Thaler, 2013. "Anomalies: Risk aversion," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 27, pages 467-480, World Scientific Publishing Co. Pte. Ltd..
    17. Zhuan Pei & Yi Shen, 2017. "The Devil is in the Tails: Regression Discontinuity Design with Measurement Error in the Assignment Variable," Advances in Econometrics, in: Regression Discontinuity Designs, volume 38, pages 455-502, Emerald Group Publishing Limited.
    18. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.
    19. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    20. David Card & David S. Lee & Zhuan Pei, 2009. "Quasi-Experimental Identification and Estimation in the Regression Kink Design," Working Papers 1206, Princeton University, Department of Economics, Industrial Relations Section..
    21. Raj Chetty & Nathaniel Hendren & Patrick Kline & Emmanuel Saez, 2014. "Where is the land of Opportunity? The Geography of Intergenerational Mobility in the United States," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(4), pages 1553-1623.
    22. Mahbubul Majumder & Heike Hofmann & Dianne Cook, 2013. "Validation of Visual Statistical Inference, Applied to Linear Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(503), pages 942-956, September.
    23. Zhuan Pei & Jörn-Steffen Pischke & Hannes Schwandt, 2019. "Poorly Measured Confounders are More Useful on the Left than on the Right," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 205-216, April.
    24. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    25. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    26. Fan, Jianqing & Yao, Qiwei, 1998. "Efficient estimation of conditional variance functions in stochastic regression," LSE Research Online Documents on Economics 6635, London School of Economics and Political Science, LSE Library.
    27. Toru Kitagawa & Shosei Sakaguchi & Aleksey Tetenov, 2021. "Constrained Classification and Policy Learning," Papers 2106.12886, arXiv.org, revised Jul 2023.
    28. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    29. Guido Imbens & Stefan Wager, 2019. "Optimized Regression Discontinuity Designs," The Review of Economics and Statistics, MIT Press, vol. 101(2), pages 264-278, May.
    30. Chen, Daniel L. & Schonger, Martin & Wickens, Chris, 2016. "oTree—An open-source platform for laboratory, online, and field experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 9(C), pages 88-97.
    31. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    32. Andrew Gelman & Guido Imbens, 2019. "Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(3), pages 447-456, July.
    33. Isaiah Andrews & Matthew Gentzkow & Jesse M. Shapiro, 2020. "Transparency in Structural Research," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 711-722, October.
    34. McCrary, Justin, 2008. "Manipulation of the running variable in the regression discontinuity design: A density test," Journal of Econometrics, Elsevier, vol. 142(2), pages 698-714, February.
    35. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    36. Ted O'Donoghue & Jason Somerville, 2018. "Modeling Risk Aversion in Economics," Journal of Economic Perspectives, American Economic Association, vol. 32(2), pages 91-114, Spring.
    37. Nickell, Stephen J, 1981. "Biases in Dynamic Models with Fixed Effects," Econometrica, Econometric Society, vol. 49(6), pages 1417-1426, November.
    38. Janet Currie & Henrik Kleven & Esmée Zwiers, 2020. "Technology and Big Data Are Changing Economics: Mining Text to Track Methods," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 42-48, May.
    39. Isaiah Andrews & Jesse M. Shapiro, 2021. "A Model of Scientific Communication," Econometrica, Econometric Society, vol. 89(5), pages 2117-2142, September.
    40. Jinyong Hahn & Petra Todd & Wilbert Van der Klaauw, 1999. "Evaluating the Effect of an Antidiscrimination Law Using a Regression-Discontinuity Design," NBER Working Papers 7131, National Bureau of Economic Research, Inc.
    41. Joshua D. Angrist & Miikka Rokkanen, 2015. "Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1331-1344, December.
    42. Abhijit V. Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2020. "A Theory of Experimenters: Robustness, Randomization, and Balance," American Economic Review, American Economic Association, vol. 110(4), pages 1206-1230, April.
    43. Jeffrey T. Leek & Roger D. Peng, 2015. "Statistics: P values are just the tip of the iceberg," Nature, Nature, vol. 520(7549), pages 612-612, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.
    2. Hanna Virtanen & Mikko Silliman & Tiina Kuuppelomäki & Kristiina Huttunen, "undated". "Education, Gender, and Family Formation," Working Papers 340, Työn ja talouden tutkimus LABORE, The Labour Institute for Economic Research LABORE.
    3. Matias D. Cattaneo & Luke Keele & Rocio Titiunik, 2023. "A Guide to Regression Discontinuity Designs in Medical Applications," Papers 2302.07413, arXiv.org, revised May 2023.
    4. Leung, Pauline, 2022. "State responses to federal matching grants: The case of medicaid," Journal of Public Economics, Elsevier, vol. 216(C).
    5. Guiffard, Jean-Baptiste, 2024. "Valuing the virtual: The impact of fiber to the home on property prices in France," Telecommunications Policy, Elsevier, vol. 48(4).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang He & Otávio Bartalotti, 2020. "Wild bootstrap for fuzzy regression discontinuity designs: obtaining robust bias-corrected confidence intervals," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 211-231.
    2. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    3. Blaise Melly & Rafael Lalive, 2020. "Estimation, Inference, and Interpretation in the Regression Discontinuity Design," Diskussionsschriften dp2016, Universitaet Bern, Departement Volkswirtschaft.
    4. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    5. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    6. Babii, Andrii & Kumar, Rohit, 2023. "Isotonic regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 234(2), pages 371-393.
    7. Yiqi Liu & Yuan Qi, 2023. "Using Forests in Multivariate Regression Discontinuity Designs," Papers 2303.11721, arXiv.org, revised Jul 2024.
    8. Adam C. Sales & Ben B. Hansen, 2020. "Limitless Regression Discontinuity," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 143-174, April.
    9. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    10. Dor Leventer & Daniel Nevo, 2024. "Correcting invalid regression discontinuity designs with multiple time period data," Papers 2408.05847, arXiv.org.
    11. Dean Eckles & Nikolaos Ignatiadis & Stefan Wager & Han Wu, 2020. "Noise-Induced Randomization in Regression Discontinuity Designs," Papers 2004.09458, arXiv.org, revised Nov 2023.
    12. David Wuepper & Robert Finger, 2023. "Regression discontinuity designs in agricultural and environmental economics," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 50(1), pages 1-28.
    13. Chen, Yi & Zhao, Yi, 2022. "The timing of first marriage and subsequent life outcomes: Evidence from a natural experiment," Journal of Comparative Economics, Elsevier, vol. 50(3), pages 713-731.
    14. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    15. Atı̇la Abdulkadı̇roğlu & Joshua D. Angrist & Yusuke Narita & Parag Pathak, 2022. "Breaking Ties: Regression Discontinuity Design Meets Market Design," Econometrica, Econometric Society, vol. 90(1), pages 117-151, January.
    16. Prakash, Nishith & Rockmore, Marc & Uppal, Yogesh, 2019. "Do criminally accused politicians affect economic outcomes? Evidence from India," Journal of Development Economics, Elsevier, vol. 141(C).
    17. Pastore, Chiara & Jones, Andrew M., 2023. "Human capital consequences of missing out on a grammar school education," Economic Modelling, Elsevier, vol. 126(C).
    18. Otávio Bartalotti, 2013. "Theory and Practice of Inference in Regression Discontinuity: A Fixed-Bandwidth Asymptotics Approach," Working Papers 1302, Tulane University, Department of Economics, revised Nov 2013.
    19. Crespo Cristian, 2020. "Beyond Manipulation: Administrative Sorting in Regression Discontinuity Designs," Journal of Causal Inference, De Gruyter, vol. 8(1), pages 164-181, January.
    20. Naven, Matthew & Whalen, Daniel, 2022. "The signaling value of university rankings: Evidence from top 14 law schools," Economics of Education Review, Elsevier, vol. 89(C).

    More about this item

    Keywords

    Regression Discontinuity Design; Regression Kink Design; Graphical Methods; VisualInference; Eyetracking; Expert Prediction;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pri:indrel:638. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Bobray Bordelon (email available below). General contact details of provider: https://edirc.repec.org/data/irprius.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.