[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/hal/journl/hal-03014999.html
   My bibliography  Save this paper

How to make a pie: Reproductible research for empirical economics and econometrics

Author

Listed:
  • Valérie Orozco

    (TSE-R - Toulouse School of Economics - UT Capitole - Université Toulouse Capitole - UT - Université de Toulouse - EHESS - École des hautes études en sciences sociales - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

  • Christophe Bontemps

    (TSE-R - Toulouse School of Economics - UT Capitole - Université Toulouse Capitole - UT - Université de Toulouse - EHESS - École des hautes études en sciences sociales - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

  • Élise Maigné

    (US ODR - Observatoire des Programmes Communautaires de Développement Rural - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

  • Virginie Piguet

    (CESAER - Centre d'Economie et de Sociologie Rurales Appliquées à l'Agriculture et aux Espaces Ruraux - AgroSup Dijon - Institut National Supérieur des Sciences Agronomiques, de l'Alimentation et de l'Environnement - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

  • Annie Hofstetter

    (CEE-M - Centre d'Economie de l'Environnement - Montpellier - UM - Université de Montpellier - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement - Institut Agro - Montpellier SupAgro - Institut Agro - Institut national d'enseignement supérieur pour l'agriculture, l'alimentation et l'environnement)

  • Anne Lacroix

    (GAEL - Laboratoire d'Economie Appliquée de Grenoble - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement - UGA - Université Grenoble Alpes - Grenoble INP - Institut polytechnique de Grenoble - Grenoble Institute of Technology - UGA - Université Grenoble Alpes)

  • Fabrice Levert

    (SMART-LERECO - Structures et Marché Agricoles, Ressources et Territoires - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement - INSTITUT AGRO Agrocampus Ouest - Institut Agro - Institut national d'enseignement supérieur pour l'agriculture, l'alimentation et l'environnement)

  • Jean‐marc Rousselle

    (CEE-M - Centre d'Economie de l'Environnement - Montpellier - UM - Université de Montpellier - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement - Institut Agro - Montpellier SupAgro - Institut Agro - Institut national d'enseignement supérieur pour l'agriculture, l'alimentation et l'environnement)

Abstract
Empirical economics and econometrics (EEE) research now relies primarily on the application of code to data sets. Handling the workflow that links data sets, programs, results and finally manuscript(s) is essential if one wishes to reproduce results. Herein, we highlight the importance of "reproducible research" in EEE and propose three simple principles to follow: organize your work, code for others and automate as much as you can. The first principle, "organize your work", deals with the overall organization of files and the documentation of a research workflow. "Code for others" emphasizes that we should take care in how we write code that has to be read by others or later by our future self. Finally, "automate as much as you can" is a proposal to avoid any manual treatment and to automate most, if not all, of the steps used in a research process to reduce errors and increase reproducibility. As software is not always the problem and will never be the solution, we illustrate these principles with good habits and tools, with a particular focus on their implementation in most popular software and languages in applied economics.

Suggested Citation

  • Valérie Orozco & Christophe Bontemps & Élise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐marc Rousselle, 2020. "How to make a pie: Reproductible research for empirical economics and econometrics," Post-Print hal-03014999, HAL.
    • Handle: RePEc:hal:journl:hal-03014999
    DOI: 10.1111/joes.12389
    Note: View the original document on HAL open archive server: https://hal.inrae.fr/hal-03014999
    as

    Download full text from publisher

    File URL: https://hal.inrae.fr/hal-03014999/document
    Download Restriction: no
    File URL: https://libkey.io/10.1111/joes.12389?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. John J. Donohue III & Steven D. Levitt, 2008. "Measurement Error, Legalized Abortion, and the Decline in Crime: A Response to Foote and Goetz," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 425-440.
    2. Ben Jann, 2017. "Creating HTML or Markdown documents from within Stata using webdoc," Stata Journal, StataCorp LP, vol. 17(1), pages 3-38, March.
    3. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 38(2), pages 257-279.
    4. Richard Van Noorden, 2011. "Science publishing: The trouble with retractions," Nature, Nature, vol. 478(7367), pages 26-28, October.
    5. Steven D. Levitt, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effects of Police on Crime: Reply," American Economic Review, American Economic Association, vol. 92(4), pages 1244-1250, September.
    6. Andrew C. Chang & Phillip Li, 2017. "A Preanalysis Plan to Replicate Sixty Economics Research Papers That Worked Half of the Time," American Economic Review, American Economic Association, vol. 107(5), pages 60-64, May.
    7. Lars Vilhuber, 2024. "Report of the AEA Data Editor," AEA Papers and Proceedings, American Economic Association, vol. 114, pages 878-890, May.
    8. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    9. Caroline M. Hoxby, 2000. "Does Competition among Public Schools Benefit Students and Taxpayers?," American Economic Review, American Economic Association, vol. 90(5), pages 1209-1238, December.
    10. Christophe Pérignon & Kamel Gadouche & Christophe Hurlin & Roxane Silberman & Eric Debonnel, 2019. "Certify reproducibility with confidential data," Post-Print hal-03528358, HAL.
    11. E. F. Haghish, 2016. "markdoc: Literate programming in Stata," Stata Journal, StataCorp LP, vol. 16(4), pages 964-988, December.
    12. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    13. E. F. Haghish, 2016. "Rethinking literate programming in statistics," Stata Journal, StataCorp LP, vol. 16(4), pages 938-963, December.
    14. Carmen M. Reinhart & Kenneth S. Rogoff, 2010. "Growth in a Time of Debt," American Economic Review, American Economic Association, vol. 100(2), pages 573-578, May.
    15. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    16. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    17. John J. Donohue III & Steven D. Levitt, 2001. "The Impact of Legalized Abortion on Crime," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 116(2), pages 379-420.
    18. Barreto,Humberto & Howland,Frank, 2006. "Introductory Econometrics," Cambridge Books, Cambridge University Press, number 9780521843195, September.
    19. Ben Jann, 2016. "Creating LaTeX documents from within Stata using texdoc," Stata Journal, StataCorp LP, vol. 16(2), pages 245-263, June.
    20. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    21. Pascaline Dupas & Jonathan Robinson, 2013. "Savings Constraints and Microenterprise Development: Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 5(1), pages 163-192, January.
    22. Ben Jann, 2016. "texdoc 2.0: An update on creating LaTeX documents from within Stata," United Kingdom Stata Users' Group Meetings 2016 04, Stata Users Group.
    23. Schulte, Eric & Davison, Dan & Dye, Thomas & Dominik, Carsten, 2012. "A Multi-Language Computing Environment for Literate Programming and Reproducible Research," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 46(i03).
    24. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    25. Daniel S. Hamermesh, 2013. "Six Decades of Top Economics Publishing: Who and How?," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 162-172, March.
    26. Roseline Bilina & Steve Lawford, 2012. "Python for Unified Research in Econometrics and Statistics," Econometric Reviews, Taylor & Francis Journals, vol. 31(5), pages 558-591, September.
    27. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    28. Lenth, Russell V. & Højsgaard, Søren, 2007. "SASWeave: Literate Programming Using SAS," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 19(i08).
    29. Brian C. Martinson & Melissa S. Anderson & Raymond de Vries, 2005. "Scientists behaving badly," Nature, Nature, vol. 435(7043), pages 737-738, June.
    30. Denis Huschka, 2013. "Why should we share our data, how can it be organized, and what are the challenges ahead?," RatSWD Working Papers 216, German Data Forum (RatSWD).
    31. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    32. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    33. Vlaeminck, Sven & Herrmann, Lisa-Kristin, 2015. "Data Policies and Data Archives: A New Paradigm for Academic Publishing in Economic Sciences?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 145-155.
    34. B.D. McCullough, 2009. "Open Access Economics Journals and the Market for Reproducible Economic Research," Economic Analysis and Policy, Elsevier, vol. 39(1), pages 117-126, March.
    35. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    36. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1988. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project: Reply," American Economic Review, American Economic Association, vol. 78(5), pages 1162-1163, December.
    37. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    38. Russell Lenth & Søren Højsgaard, 2011. "Reproducible statistical analysis with multiple languages," Computational Statistics, Springer, vol. 26(3), pages 419-426, September.
    39. McCullough, B. D., 2018. "Quis custodiet ipsos custodes? Despite evidence to the contrary, the American Economic Review concluded that all was well with its archive," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-13.
    40. Germán Rodríguez, 2017. "Literate data analysis with Stata and Markdown," Stata Journal, StataCorp LP, vol. 17(3), pages 600-618, September.
    41. Christopher L. Foote & Christopher F. Goetz, 2008. "The Impact of Legalized Abortion on Crime: Comment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 407-423.
    42. Roger Koenker & Achim Zeileis, 2009. "On reproducible econometric research," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 24(5), pages 833-847.
    43. Evan Meredith & Jeffrey S. Racine, 2009. "Towards reproducible econometric research: the Sweave framework," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 24(2), pages 366-374, March.
    44. J. Scott Long, 2009. "The Workflow of Data Analysis Using Stata," Stata Press books, StataCorp LP, number wdaus, March.
    45. Hunter, John E, 2001. "The Desperate Need for Replications," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(1), pages 149-158, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Raymond P. Guiteras & Ahnjeong Kim & Brian Quistorff & Clayson Shumway, 2023. "statacons: An SCons-based build tool for Stata," Stata Journal, StataCorp LP, vol. 23(1), pages 148-196, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Valérie Orozco & Christophe Bontemps & Élise Maigné & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.
    2. Colliard, Jean-Edouard & Hurlin, Christophe & Pérignon, Christophe, 2019. "Reproducibility Certification in Economics Research," HEC Research Papers Series 1345, HEC Paris.
    3. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    4. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    5. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    6. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    7. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    8. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    9. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    10. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    11. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    12. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    13. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    14. Sylvérie Herbert & Hautahi Kingi & Flavio Stanchi & Lars Vilhuber, 2024. "Reproduce to validate: A comprehensive study on the reproducibility of economics research," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 57(3), pages 961-988, August.
    15. Fišar, Miloš & Greiner, Ben & Huber, Christoph & Katok, Elena & Ozkes, Ali & Management Science Reproducibility Collaboration, 2023. "Reproducibility in Management Science," Department for Strategy and Innovation Working Paper Series 03/2023, WU Vienna University of Economics and Business.
    16. Vlaeminck, Sven & Herrmann, Lisa-Kristin, 2015. "Data Policies and Data Archives: A New Paradigm for Academic Publishing in Economic Sciences?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 145-155.
    17. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    18. Alejandro Gaviria & Carlos Medina & Jorge Tamayo, 2010. "Assessing the Link between Adolescent Fertility and Urban Crime," Borradores de Economia 6860, Banco de la Republica.
    19. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    20. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:journl:hal-03014999. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.