[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/zbw/esprep/194798.html
   My bibliography  Save this paper

Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method

Author

Listed:
  • Furukawa, Chishio
Abstract
This paper questions the conventional wisdom that publication bias must result from the biased preferences of researchers. When readers only compare the number of positive and negative results of papers to make their decisions, even unbiased researchers will omit noisy null results and inflate some marginally insignificant estimates. Nonetheless, the equilibrium with such publication bias is socially optimal. The model predicts that published non-positive results are either precise null results or noisy but extreme negative results. This paper shows this prediction holds with some data, and proposes a new stem-based bias correction method that is robust to this and other publication selection processes.

Suggested Citation

  • Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    • Handle: RePEc:zbw:esprep:194798
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/194798/1/draft%20March%202019%20v2.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Marco Battaglini & Rebecca B. Morton & Thomas R. Palfrey, 2010. "The Swing Voter's Curse in the Laboratory," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 77(1), pages 61-89.
    2. Lucas W. Davis, 2008. "The Effect of Driving Restrictions on Air Quality in Mexico City," Journal of Political Economy, University of Chicago Press, vol. 116(1), pages 38-81, February.
    3. Colin Camerer & Linda Babcock & George Loewenstein & Richard Thaler, 1997. "Labor Supply of New York City Cabdrivers: One Day at a Time," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 407-441.
    4. Alberto Abadie, 2020. "Statistical Nonsignificance in Empirical Economics," American Economic Review: Insights, American Economic Association, vol. 2(2), pages 193-208, June.
    5. Fudenberg, Drew & Levine, David, 1998. "Learning in games," European Economic Review, Elsevier, vol. 42(3-5), pages 631-639, May.
    6. Emeric Henry, 2009. "Strategic Disclosure of Research Results: The Cost of Proving Your Honesty," Economic Journal, Royal Economic Society, vol. 119(539), pages 1036-1064, July.
    7. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    8. Eddie Dekel & Michele Piccione, 2000. "Sequential Voting Procedures in Symmetric Binary Elections," Journal of Political Economy, University of Chicago Press, vol. 108(1), pages 34-55, February.
    9. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    10. Kei Kawai & Yasutora Watanabe, 2013. "Inferring Strategic Voting," American Economic Review, American Economic Association, vol. 103(2), pages 624-662, April.
    11. Sanchez-Pages, Santiago & Vorsatz, Marc, 2007. "An experimental study of truth-telling in a sender-receiver game," Games and Economic Behavior, Elsevier, vol. 61(1), pages 86-112, October.
    12. Rema Hanna & Esther Duflo & Michael Greenstone, 2016. "Up in Smoke: The Influence of Household Behavior on the Long-Run Impact of Improved Cooking Stoves," American Economic Journal: Economic Policy, American Economic Association, vol. 8(1), pages 80-114, February.
    13. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    14. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    15. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    16. Gerber, Alan & Malhotra, Neil, 2008. "Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals," Quarterly Journal of Political Science, now publishers, vol. 3(3), pages 313-326, October.
    17. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    18. Havranek, Tomas & Irsova, Zuzana & Janda, Karel & Zilberman, David, 2015. "Selective reporting and the social cost of carbon," Energy Economics, Elsevier, vol. 51(C), pages 394-406.
    19. Vijay Krishna & John Morgan, 2001. "A Model of Expertise," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 116(2), pages 747-775.
    20. Laroche, P., 2000. "What do Unions do to Productivity? A Meta-Analysis," Papers 2000-5, Groupe de recherche en économie financière et en gestion des entreprises, Universite Nancy 2.
    21. Hristos Doucouliagos & Richard Freeman & Patrice Laroche & T.D. Stanley, 2018. "How Credible Is Trade Union Research? Forty Years of Evidence on the Monopoly–Voice Trade-Off," Post-Print hal-02137429, HAL.
    22. Joost de Winter & Riender Happee, 2013. "Why Selective Publication of Statistically Significant Results Can Be Effective," PLOS ONE, Public Library of Science, vol. 8(6), pages 1-7, June.
    23. Shapiro, Jesse M., 2016. "Special interests and the media: Theory and an application to climate change," Journal of Public Economics, Elsevier, vol. 144(C), pages 91-108.
    24. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    25. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    26. Justin McCrary & Garret Christensen & Daniele Fanelli, 2016. "Conservative Tests under Satisficing Models of Publication Bias," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-10, February.
    27. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    28. Hao Li & Sherwin Rosen & Wing Suen, 2001. "Conflicts and Common Interests in Committees," American Economic Review, American Economic Association, vol. 91(5), pages 1478-1497, December.
    29. repec:hal:wpspec:info:hdl:2441/eu4vqp9ompqllr09iatr74eao is not listed on IDEAS
    30. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    31. Austen-Smith, David & Banks, Jeffrey S., 1996. "Information Aggregation, Rationality, and the Condorcet Jury Theorem," American Political Science Review, Cambridge University Press, vol. 90(1), pages 34-45, March.
    32. Timothy J. Feddersen, 2004. "Rational Choice Theory and the Paradox of Not Voting," Journal of Economic Perspectives, American Economic Association, vol. 18(1), pages 99-112, Winter.
    33. Jürgen Barth & Thomas Munder & Heike Gerger & Eveline Nüesch & Sven Trelle & Hansjörg Znoj & Peter Jüni & Pim Cuijpers, 2013. "Comparative Efficacy of Seven Psychotherapeutic Interventions for Patients with Depression: A Network Meta-Analysis," PLOS Medicine, Public Library of Science, vol. 10(5), pages 1-17, May.
    34. John Morgan & Phillip C. Stocken, 2008. "Information Aggregation in Polls," American Economic Review, American Economic Association, vol. 98(3), pages 864-896, June.
    35. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    36. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    37. Hristos Doucouliagos & Richard B. Freeman & Patrice Laroche & T. D. Stanley, 2018. "How Credible Is Trade Union Research? Forty Years of Evidence on the Monopoly–Voice Trade-Off," ILR Review, Cornell University, ILR School, vol. 71(2), pages 287-305, March.
    38. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    39. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    40. Torun Dewan & Andrea Galeotti & Christian Ghiglino & Francesco Squintani, 2015. "Information Aggregation and Optimal Structure of the Executive," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 475-494, February.
    41. John J. Siegfried, 2012. "Minutes of the Meeting of the Executive Committee: Chicago, IL, January 5, 2012," American Economic Review, American Economic Association, vol. 102(3), pages 645-652, May.
    42. Fafchamps, Marcel & Labonne, Julien, 2017. "Using Split Samples to Improve Inference on Causal Effects," Political Analysis, Cambridge University Press, vol. 25(4), pages 465-482, October.
    43. Kristin J. Forbes, 2000. "A Reassessment of the Relationship between Inequality and Growth," American Economic Review, American Economic Association, vol. 90(4), pages 869-887, September.
    44. Freeman, Richard, 1988. "Union density and economic performance : An analysis of U.S. States," European Economic Review, Elsevier, vol. 32(2-3), pages 707-716, March.
    45. Fryer Roland & Jackson Matthew O., 2008. "A Categorical Model of Cognition and Biased Decision Making," The B.E. Journal of Theoretical Economics, De Gruyter, vol. 8(1), pages 1-44, February.
    46. Blakeley B. McShane & David Gal, 2017. "Rejoinder: Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 904-908, July.
    47. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    48. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    49. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    50. Crawford, Vincent, 1998. "A Survey of Experiments on Communication via Cheap Talk," Journal of Economic Theory, Elsevier, vol. 78(2), pages 286-298, February.
    51. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    52. Wing Suen, 2004. "The Self-Perpetuation of Biased Beliefs," Economic Journal, Royal Economic Society, vol. 114(495), pages 377-396, April.
    53. Powell, G. Bingham, 1986. "American Voter Turnout in Comparative Perspective," American Political Science Review, Cambridge University Press, vol. 80(1), pages 17-43, March.
    54. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    55. Aleksey Tetenov, 2016. "An economic theory of statistical testing," CeMMAP working papers CWP50/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    56. Drew Fudenberg & David K. Levine, 1998. "The Theory of Learning in Games," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262061945, April.
    57. Hilda Bastian & Paul Glasziou & Iain Chalmers, 2010. "Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?," PLOS Medicine, Public Library of Science, vol. 7(9), pages 1-6, September.
    58. repec:fth:prinin:425 is not listed on IDEAS
    59. Marcel A L M van Assen & Robbie C M van Aert & Michèle B Nuijten & Jelte M Wicherts, 2014. "Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results," PLOS ONE, Public Library of Science, vol. 9(1), pages 1-5, January.
    60. repec:hal:spmain:info:hdl:2441/eu4vqp9ompqllr09iatr74eao is not listed on IDEAS
    61. Austen-Smith David, 1993. "Interested Experts and Policy Advice: Multiple Referrals under Open Rule," Games and Economic Behavior, Elsevier, vol. 5(1), pages 3-43, January.
    62. Gilligan, Thomas W & Krehbiel, Keith, 1987. "Collective Decisionmaking and Standing Committees: An Informational Rationale for Restrictive Amendment Procedures," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 3(2), pages 287-335, Fall.
    63. Emeric Henry, 2009. "Disclosure of research results: the cost of proving your honesty," Post-Print hal-01023670, HAL.
    64. Torun Dewan & Andrea Galeotti & Christian Ghiglino & Francesco Squintani, 2015. "Information aggregation and optimal structure of the executive," LSE Research Online Documents on Economics 59632, London School of Economics and Political Science, LSE Library.
    65. Marco Battaglini, 2017. "Public Protests and Policy Making," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(1), pages 485-549.
    66. Crawford, Vincent P & Sobel, Joel, 1982. "Strategic Information Transmission," Econometrica, Econometric Society, vol. 50(6), pages 1431-1451, November.
    67. Ignacio Esponda Jr. & Emanuel Vespa Jr., 2014. "Hypothetical Thinking and Information Extraction in the Laboratory," American Economic Journal: Microeconomics, American Economic Association, vol. 6(4), pages 180-202, November.
    68. Blakeley B. McShane & David Gal, 2017. "Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 885-895, July.
    69. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    2. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    3. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    4. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    5. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    6. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    7. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    8. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    9. Tomas Havranek & Zuzana Irsova & Lubica Laslopova & Olesia Zeynalova, 2020. "Skilled and Unskilled Labor Are Less Substitutable than Commonly Thought," Working Papers IES 2020/29, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    10. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    11. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    12. Kroupova, Katerina & Havranek, Tomas & Irsova, Zuzana, 2024. "Student Employment and Education: A Meta-Analysis," Economics of Education Review, Elsevier, vol. 100(C).
    13. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    14. Brodeur, Abel & Cook, Nikolai & Hartley, Jonathan & Heyes, Anthony, 2022. "Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?," MetaArXiv uxf39, Center for Open Science.
    15. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    16. Doucouliagos, Hristos & Hinz, Thomas & Zigova, Katarina, 2022. "Bias and careers: Evidence from the aid effectiveness literature," European Journal of Political Economy, Elsevier, vol. 71(C).
    17. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    18. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    19. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    20. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.

    More about this item

    Keywords

    publication bias; information aggregation; meta-analysis; bias correction method;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • D71 - Microeconomics - - Analysis of Collective Decision-Making - - - Social Choice; Clubs; Committees; Associations
    • D82 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Asymmetric and Private Information; Mechanism Design
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    NEP fields

    This paper has been announced in the following NEP Reports:

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Meta-Research in Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:esprep:194798. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.