Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply
Author
Suggested Citation
DOI: 10.1257/aer.20220277
Download full text from publisher
References listed on IDEAS
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016.
"Star Wars: The Empirics Strike Back,"
American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2012. "Star wars: The empirics strike back," Working Papers halshs-00710122, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2015. "Star Wars: The Empirics Strike Back," PSE Working Papers halshs-01158500, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," PSE-Ecole d'économie de Paris (Postprint) hal-01447851, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2015. "Star Wars: The Empirics Strike Back," Working Papers halshs-01158500, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2015. "Star Wars: The Empirics Strike Back," AMSE Working Papers 1523, Aix-Marseille School of Economics, France, revised May 2015.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2012. "Star wars: The empirics strike back," PSE Working Papers halshs-00710122, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2012. "Star wars: The empirics strike back," SciencePo Working papers Main halshs-00710122, HAL.
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2015. "Star Wars: The Empirics Strike Back," Working Papers 1505E, University of Ottawa, Department of Economics.
- Brodeur, Abel & Lé, Mathias & Sangnier, Marc & Zylberberg, Yanos, 2013. "Star Wars: The Empirics Strike Back," IZA Discussion Papers 7268, Institute of Labor Economics (IZA).
- Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," Post-Print hal-01447851, HAL.
- Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
Citations
Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
Cited by:
- Danielle V. Handel & Eric A. Hanushek, 2024.
"Contexts of Convenience: Generalizing from Published Evaluations of School Finance Policies,"
Evaluation Review, , vol. 48(3), pages 461-494, June.
- Danielle V. Handel & Eric A. Hanushek, 2023. "Contexts of Convenience: Generalizing from Published Evaluations of School Finance Policies," NBER Working Papers 31653, National Bureau of Economic Research, Inc.
- Danielle V. Handel & Eric A. Hanushek, 2023. "Contexts of Convenience: Generalizing from Published Evaluations of School Finance Policies," CESifo Working Paper Series 10664, CESifo.
- Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023.
"Do economists replicate?,"
Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
- Fiala, Nathan & Neubauer, Florian & Peters, Jörg, 2022. "Do economists replicate?," Ruhr Economic Papers 939, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
- Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do Economists Replicate?," I4R Discussion Paper Series 13, The Institute for Replication (I4R).
- Mußhoff, Oliver & Ölkers, Tim & Kirchner, Ella, 2024. "A note on the discussion regarding terrorism and land use in agriculture," Land Use Policy, Elsevier, vol. 144(C).
- Ankel-Peters, Jörg & Schmidt, Christoph M., 2023.
"Rural electrification, the credibility revolution, and the limits of evidence-based policy,"
Ruhr Economic Papers
1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
- Ankel-Peters, Jörg & Schmidt, Christoph M., 2024. "Rural Electrification, the Credibility Revolution, and the Limits of Evidence-Based Policy," I4R Discussion Paper Series 123, The Institute for Replication (I4R).
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- Stefano DellaVigna & Elizabeth Linos, 2022.
"RCTs to Scale: Comprehensive Evidence From Two Nudge Units,"
Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
- Stefano DellaVigna & Elizabeth Linos, 2020. "RCTs to Scale: Comprehensive Evidence from Two Nudge Units," NBER Working Papers 27594, National Bureau of Economic Research, Inc.
- Bruns, Stephan & Herwartz, Helmut & Ioannidis, John P.A. & Islam, Chris-Gabriel & Raters, Fabian H. C., 2023. "Statistical reporting errors in economics," MetaArXiv mbx62, Center for Open Science.
- Christoph Huber & Christian König-Kersting & Matteo M. Marini, 2022. "Experimenting with Financial Professionals," Working Papers 2022-07, Faculty of Economics and Statistics, Universität Innsbruck, revised Jun 2024.
- Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022.
"We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments,"
LCERPA Working Papers
am0133, Laurier Centre for Economic Research and Policy Analysis.
- Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," MetaArXiv a9vhr, Center for Open Science.
- Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," IZA Discussion Papers 15478, Institute of Labor Economics (IZA).
- Uwe Hassler & Marc‐Oliver Pohle, 2022.
"Unlucky Number 13? Manipulating Evidence Subject to Snooping,"
International Statistical Review, International Statistical Institute, vol. 90(2), pages 397-410, August.
- Uwe Hassler & Marc-Oliver Pohle, 2020. "Unlucky Number 13? Manipulating Evidence Subject to Snooping," Papers 2009.02198, arXiv.org.
- Adam Gorajek & Joel Bank & Andrew Staib & Benjamin Malin & Hamish Fitchett, 2021.
"Star Wars at Central Banks,"
RBA Research Discussion Papers
rdp2021-02, Reserve Bank of Australia.
- Joel Bank & Hamish Fitchett & Adam Gorajek & Benjamin Malin & Andrew Staib, 2021. "Star Wars at Central Banks," Reserve Bank of New Zealand Discussion Paper Series DP2021/01, Reserve Bank of New Zealand.
- Joel Bank & Hamish Fitchett & Adam Gorajek & Benjamin A. Malin & Andrew Staib, 2021. "Star Wars at Central Banks," Staff Report 620, Federal Reserve Bank of Minneapolis.
- Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
- Gechert, Sebastian & Mey, Bianka & Opatrny, Matej & Havranek, Tomas & Stanley, T. D. & Bom, Pedro R. D. & Doucouliagos, Hristos & Heimberger, Philipp & Irsova, Zuzana & Rachinger, Heiko J., 2023.
"Conventional Wisdom, Meta-Analysis, and Research Revision in Economics,"
EconStor Preprints
280745, ZBW - Leibniz Information Centre for Economics.
- Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T. D. Stanley & Pedro R. D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2023. "Conventional Wisdom, Meta-Analysis, and Research Revision in Economics," Chemnitz Economic Papers 061, Department of Economics, Chemnitz University of Technology.
- Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T.D. Stanley & Pedro R.D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2024. "Conventional Wisdom, Meta-Analysis, and Research Revision in Economics," FMM Working Paper 95-2024, IMK at the Hans Boeckler Foundation, Macroeconomic Policy Institute.
- Dreber, Anna & Johannesson, Magnus, 2023.
"A framework for evaluating reproducibility and replicability in economics,"
Ruhr Economic Papers
1055, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
- Dreber, Anna & Johannesson, Magnus, 2023. "A framework for evaluating reproducibility and replicability in economics," I4R Discussion Paper Series 38, The Institute for Replication (I4R).
- Graham Elliott & Nikolay Kudrin & Kaspar Wüthrich, 2022.
"Detecting p‐Hacking,"
Econometrica, Econometric Society, vol. 90(2), pages 887-906, March.
- Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2019. "Detecting p-hacking," Papers 1906.06711, arXiv.org, revised May 2021.
- Elliott, Graham & Kudrin, Nikolay & Wüthrich, Kaspar, 2022. "Detecting p‐Hacking," University of California at San Diego, Economics Working Paper Series qt2p04s3dr, Department of Economics, UC San Diego.
- Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024.
"Meta-analysis of Empirical Estimates of Loss Aversion,"
Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
- Brown, Alexander L. & Imai, Taisuke & Vieider, Ferdinand & Camerer, Colin, 2020. "Meta-Analysis of Empirical Estimates of Loss-Aversion," MetaArXiv hnefr, Center for Open Science.
- Brown, Alexander L. & Imai, Taisuke & Vieider, Ferdinand M. & Camerer, Colin F., 2021. "Meta-Analysis of Empirical Estimates of Loss-Aversion," Rationality and Competition Discussion Paper Series 275, CRC TRR 190 Rationality and Competition.
- Alexander, L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin Camerer, 2021. "Meta-Analysis of Empirical Estimates of Loss-Aversion," CESifo Working Paper Series 8848, CESifo.
- Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022.
"We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments,"
GLO Discussion Paper Series
1157, Global Labor Organization (GLO).
- Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," I4R Discussion Paper Series 8, The Institute for Replication (I4R).
- Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023.
"Unpacking P-hacking and Publication Bias,"
American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
- Brodeur, Abel & Carrell, Scott & Figlio, David & Lusher, Lester, 2023. "Unpacking P-Hacking and Publication Bias," I4R Discussion Paper Series 52, The Institute for Replication (I4R).
- Abel Brodeur & Scott E. Carrell & David N. Figlio & Lester R. Lusher, 2023. "Unpacking P-Hacking and Publication Bias," NBER Working Papers 31548, National Bureau of Economic Research, Inc.
- Brodeur, Abel & Carrell, Scott E. & Figlio, David N. & Lusher, Lester, 2023. "Unpacking P-Hacking and Publication Bias," IZA Discussion Papers 16369, Institute of Labor Economics (IZA).
- Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P & Hepplewhite, Matthew & de la Guardia, Fernando Hoces, 2024.
"Promoting Reproducibility and Replicability in Political Science,"
Department of Economics, Working Paper Series
qt23n3n3dg, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
- Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S. & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P. & Hepplewhite, Matthew & Hoces de la Guardia, Fernan, 2024. "Promoting Reproducibility and Replicability in Political Science," I4R Discussion Paper Series 100, The Institute for Replication (I4R).
- Adam Gorajek & Benjamin A. Malin, 2021. "Comment on "Star Wars: The Empirics Strike Back"," Staff Report 629, Federal Reserve Bank of Minneapolis.
- Roggenkamp, Hauke C., 2024. "Revisiting ‘Growth and Inequality in Public Good Provision’—Reproducing and Generalizing Through Inconvenient Online Experimentation," OSF Preprints 6rn97, Center for Open Science.
- Anna Dreber & Magnus Johannesson & Yifan Yang, 2024.
"Selective reporting of placebo tests in top economics journals,"
Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
- Dreber, Anna & Johannesson, Magnus & Yang, Yifan, 2023. "Selective Reporting of Placebo Tests in Top Economics Journals," I4R Discussion Paper Series 31, The Institute for Replication (I4R).
- Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2024.
"The Null Result Penalty,"
The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
- Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2023. "The Null Result Penalty," The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
- Felix Chopra & Ingar K. Haaland & Christopher Roth & Andreas Stegmann, 2022. "The Null Result Penalty," CESifo Working Paper Series 9776, CESifo.
- Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2022. "The Null Result Penalty," ECONtribute Discussion Papers Series 169, University of Bonn and University of Cologne, Germany.
- Chopra, Felix & Haaland, Ingar & Roth, Christopher & Stegmann, Andreas, 2022. "The Null Result Penalty," CEPR Discussion Papers 17331, C.E.P.R. Discussion Papers.
- Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "The Power of Tests for Detecting $p$-Hacking," Papers 2205.07950, arXiv.org, revised Apr 2024.
- Antinyan, Armenak & Asatryan, Zareh, 2019.
"Nudging for tax compliance: A meta-analysis,"
ZEW Discussion Papers
19-055, ZEW - Leibniz Centre for European Economic Research.
- Antinyan, Armenak & Asatryan, Zareh, 2024. "Nudging for tax compliance: A meta-analysis," ZEW Discussion Papers 24-054, ZEW - Leibniz Centre for European Economic Research.
- Armenak Antinyan & Zareh Asatryan, 2020. "Nudging for Tax Compliance: A Meta-Analysis," CESifo Working Paper Series 8500, CESifo.
More about this item
JEL classification:
- A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
- C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
- C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
Statistics
Access and download statisticsCorrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:aecrev:v:112:y:2022:i:9:p:3137-39. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.