[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/266186.html
   My bibliography  Save this article

How to keep it adequate: A protocol for ensuring validity in agent-based simulation

Author

Listed:
  • Troost, Christian
  • Huber, Robert
  • Bell, Andrew R.
  • van Delden, Hedwig
  • Filatova, Tatiana
  • Le, Quang Bao
  • Lippe, Melvin
  • Niamir, Leila
  • Polhill, J. Gareth
  • Sun, Zhanli
  • Berger, Thomas
Abstract
There has so far been no shared understanding of validity in agent-based simulation. We here conceptualise validation as systematically substantiating the premises on which conclusions from simulation analysis for a particular modelling context are built. Given such a systematic perspective, validity of agent-based models cannot be ensured if validation is merely understood as an isolated step in the modelling process. Rather, valid conclusions from simulation analysis require context-adequate method choices at all steps of the simulation analysis including model construction, model and parameter inference, uncertainty analysis and simulation. We present a twelve-step protocol to highlight the (often hidden) premises for methodological choices and their link to the modelling context. It is designed to aid modelers in understanding their context and in choosing and documenting context-adequate and mutually consistent methods throughout the modelling process. Its purpose is to assist reviewers and the community as a whole in assessing and discussing context-adequacy.

Suggested Citation

  • Troost, Christian & Huber, Robert & Bell, Andrew R. & van Delden, Hedwig & Filatova, Tatiana & Le, Quang Bao & Lippe, Melvin & Niamir, Leila & Polhill, J. Gareth & Sun, Zhanli & Berger, Thomas, 2023. "How to keep it adequate: A protocol for ensuring validity in agent-based simulation," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 159, pages 1-21.
    • Handle: RePEc:zbw:espost:266186
    DOI: 10.1016/j.envsoft.2022.105559
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/266186/1/Troost_2023_validity_agent_based_simulation.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.1016/j.envsoft.2022.105559?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rand, William & Rust, Roland T., 2011. "Agent-based modeling in marketing: Guidelines for rigor," International Journal of Research in Marketing, Elsevier, vol. 28(3), pages 181-193.
    2. Leila Niamir & Gregor Kiesewetter & Fabian Wagner & Wolfgang Schöpp & Tatiana Filatova & Alexey Voinov & Hans Bressers, 2020. "Assessing the macroeconomic impacts of individual behavioral changes on carbon emissions," Climatic Change, Springer, vol. 158(2), pages 141-160, January.
    3. Augusiak, Jacqueline & Van den Brink, Paul J. & Grimm, Volker, 2014. "Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach," Ecological Modelling, Elsevier, vol. 280(C), pages 117-128.
    4. Kukacka, Jiri & Barunik, Jozef, 2017. "Estimation of financial agent-based models with simulated maximum likelihood," Journal of Economic Dynamics and Control, Elsevier, vol. 85(C), pages 21-45.
    5. Saul I. Gass, 1983. "Feature Article—Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis," Operations Research, INFORMS, vol. 31(4), pages 603-631, August.
    6. McCloskey, Donald N, 1985. "The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests," American Economic Review, American Economic Association, vol. 75(2), pages 201-205, May.
    7. Onggo, Bhakti Stephan & Karatas, Mumtaz, 2016. "Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation," European Journal of Operational Research, Elsevier, vol. 254(2), pages 517-531.
    8. Charles F. Manski, 2019. "Treatment Choice With Trial Data: Statistical Decision Theory Should Supplant Hypothesis Testing," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 296-304, March.
    9. Ernesto Carrella & Richard Bailey & Jens Koed Madsen, 2020. "Calibrating Agent-Based Models with Linear Regressions," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 23(1), pages 1-7.
    10. Midgley, David & Marks, Robert & Kunchamwar, Dinesh, 2007. "Building and assurance of agent-based models: An example and challenge to the field," Journal of Business Research, Elsevier, vol. 60(8), pages 884-893, August.
    11. Rosenzweig, Mark & Udry, Christopher, 2016. "External Validity in a Stochastic World," Center Discussion Papers 242440, Yale University, Economic Growth Center.
    12. Thomas Brenner & Claudia Werker, 2007. "A Taxonomy of Inference in Simulation Models," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 227-244, October.
    13. McCarl, Bruce A. & Apland, Jeffrey, 1986. "Validation Of Linear Programming Models," Southern Journal of Agricultural Economics, Southern Agricultural Economics Association, vol. 18(2), pages 1-10, December.
    14. van Vliet, Jasper & Hagen-Zanker, Alex & Hurkens, Jelle & van Delden, Hedwig, 2013. "A fuzzy set approach to assess the predictive accuracy of land use simulations," Ecological Modelling, Elsevier, vol. 261, pages 32-42.
    15. Volker Grimm & Steven F. Railsback & Christian E. Vincenot & Uta Berger & Cara Gallagher & Donald L. DeAngelis & Bruce Edmonds & Jiaqi Ge & Jarl Giske & Jürgen Groeneveld & Alice S.A. Johnston & Alex, 2020. "The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 23(2), pages 1-7.
    16. Jan C. Thiele & Winfried Kurth & Volker Grimm, 2014. "Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and 'R'," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 17(3), pages 1-11.
    17. Finn E. Kydland & Edward C. Prescott, 1996. "The Computational Experiment: An Econometric Tool," Journal of Economic Perspectives, American Economic Association, vol. 10(1), pages 69-85, Winter.
    18. Engle, Robert F. & Hendry, David F., 1993. "Testing superexogeneity and invariance in regression models," Journal of Econometrics, Elsevier, vol. 56(1-2), pages 119-139, March.
    19. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    20. Grimm, Volker & Berger, Uta & DeAngelis, Donald L. & Polhill, J. Gary & Giske, Jarl & Railsback, Steven F., 2010. "The ODD protocol: A review and first update," Ecological Modelling, Elsevier, vol. 221(23), pages 2760-2768.
    21. Abhijit Banerjee & Sylvain Chassang & Erik Snowberg, 2016. "Decision Theoretic Approaches to Experiment Design and External Validity," NBER Working Papers 22167, National Bureau of Economic Research, Inc.
    22. Aumann, Craig A., 2007. "A methodology for developing simulation models of complex systems," Ecological Modelling, Elsevier, vol. 202(3), pages 385-396.
    23. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    24. Li An & Volker Grimm & Billie L. Turner II, 2020. "Editorial: Meeting Grand Challenges in Agent-Based Models," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 23(1), pages 1-13.
    25. McCloskey, Donald N, 1983. "The Rhetoric of Economics," Journal of Economic Literature, American Economic Association, vol. 21(2), pages 481-517, June.
    26. Mössinger, Johannes & Troost, Christian & Berger, Thomas, 2022. "Bridging the gap between models and users: A lightweight mobile interface for optimized farming decisions in interactive modeling sessions," Agricultural Systems, Elsevier, vol. 195(C).
    27. Scott Moss & Bruce Edmonds, 2005. "Towards Good Social Science," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 8(4), pages 1-13.
    28. van der Vaart, Elske & Beaumont, Mark A. & Johnston, Alice S.A. & Sibly, Richard M., 2015. "Calibration and evaluation of individual-based models using Approximate Bayesian Computation," Ecological Modelling, Elsevier, vol. 312(C), pages 182-190.
    29. Lars Peter Hansen & James J. Heckman, 1996. "The Empirical Foundations of Calibration," Journal of Economic Perspectives, American Economic Association, vol. 10(1), pages 87-104, Winter.
    30. Natasha Stout & Sue Goldie, 2008. "Keeping the noise down: common random numbers for disease simulation modeling," Health Care Management Science, Springer, vol. 11(4), pages 399-406, December.
    31. Thomas Berger & Christian Troost, 2014. "Agent-based Modelling of Climate Adaptation and Mitigation Options in Agriculture," Journal of Agricultural Economics, Wiley Blackwell, vol. 65(2), pages 323-348, June.
    32. Simon Deichsel & Andreas Pyka, 2009. "A Pragmatic Reading of Friedman's Methodological Essay and What It Tells Us for the Discussion of ABMs," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 12(4), pages 1-6.
    33. Pavlo R. Blavatskyy & Ganna Pogrebna, 2010. "Models of stochastic choice and decision theories: why both are important for analyzing decisions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 963-986.
    34. Christian Troost & Thomas Berger, 2015. "Dealing with Uncertainty in Agent-Based Simulation: Farm-Level Modeling of Adaptation to Climate Change in Southwest Germany," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 97(3), pages 833-854.
    35. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    36. Lester V. Manderscheid, 1965. "Significance Levels—0.05, 0.01, or?," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 47(5), pages 1381-1385.
    37. Borgonovo, Emanuele & Plischke, Elmar, 2016. "Sensitivity analysis: A review of recent advances," European Journal of Operational Research, Elsevier, vol. 248(3), pages 869-887.
    38. Hands,D. Wade, 2001. "Reflection without Rules," Cambridge Books, Cambridge University Press, number 9780521797962.
    39. James Nolan & Dawn Parker & G. Cornelis Van Kooten & Thomas Berger, 2009. "An Overview of Computational Modeling in Agricultural and Resource Economics," Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 57(4), pages 417-429, December.
    40. Grimm, Volker & Augusiak, Jacqueline & Focks, Andreas & Frank, Béatrice M. & Gabsi, Faten & Johnston, Alice S.A. & Liu, Chun & Martin, Benjamin T. & Meli, Mattia & Radchuk, Viktoriia & Thorbek, Pernil, 2014. "Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE," Ecological Modelling, Elsevier, vol. 280(C), pages 129-139.
    41. Troost, Christian & Berger, Thomas, 2015. "Process-based simulation of regional agricultural supply functions in Southwestern Germany using farm-level and agent-based models," 2015 Conference, August 9-14, 2015, Milan, Italy 211929, International Association of Agricultural Economists.
    42. Ward, Eric J., 2008. "A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools," Ecological Modelling, Elsevier, vol. 211(1), pages 1-10.
    43. Thomas Berger & Christian Troost & Tesfamicheal Wossen & Evgeny Latynskiy & Kindie Tesfaye & Sika Gbegbelegbe, 2017. "Can smallholder farmers adapt to climate variability, and how effective are policy interventions? Agent-based simulation results for Ethiopia," Agricultural Economics, International Association of Agricultural Economists, vol. 48(6), pages 693-706, November.
    44. Lucas, Robert Jr, 1976. "Econometric policy evaluation: A critique," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 1(1), pages 19-46, January.
    45. Bruce Edmonds & Christophe Le Page & Mike Bithell & Edmund Chattoe-Brown & Volker Grimm & Ruth Meyer & Cristina Montañola-Sales & Paul Ormerod & Hilton Root & Flaminio Squazzoni, 2019. "Different Modelling Purposes," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 22(3), pages 1-6.
    46. Jule Thober & Birgit Müller & Jürgen Groeneveld & Volker Grimm, 2017. "Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(2), pages 1-8.
    47. Carauta, Marcelo & Troost, Christian & Guzman-Bustamante, Ivan & Hampf, Anna & Libera, Affonso & Meurer, Katharina & Bönecke, Eric & Franko, Uwe & Ribeiro Rodrigues, Renato de Aragão & Berger, Thomas, 2021. "Climate-related land use policies in Brazil: How much has been achieved with economic incentives in agriculture?," Land Use Policy, Elsevier, vol. 109(C).
    48. Calum Brown & Peter Alexander & Sascha Holzhauer & Mark D. A. Rounsevell, 2017. "Behavioral models of climate change adaptation and mitigation in land‐based sectors," Wiley Interdisciplinary Reviews: Climate Change, John Wiley & Sons, vol. 8(2), March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cordelia Kreft & Robert Huber & David Schäfer & Robert Finger, 2024. "Quantifying the impact of farmers' social networks on the effectiveness of climate change mitigation policies in agriculture," Journal of Agricultural Economics, Wiley Blackwell, vol. 75(1), pages 298-322, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robert Huber & Hang Xiong & Kevin Keller & Robert Finger, 2022. "Bridging behavioural factors and standard bio‐economic modelling in an agent‐based modelling framework," Journal of Agricultural Economics, Wiley Blackwell, vol. 73(1), pages 35-63, February.
    2. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    3. Huber, Robert & Bakker, Martha & Balmann, Alfons & Berger, Thomas & Bithell, Mike & Brown, Calum & Grêt-Regamey, Adrienne & Xiong, Hang & Le, Quang Bao & Mack, Gabriele & Meyfroidt, Patrick & Millingt, 2018. "Representation of decision-making in European agricultural agent-based models," Agricultural Systems, Elsevier, vol. 167(C), pages 143-160.
    4. Lapp, Maya & Long, Colby, 2022. "A new approach to agent-based models of Community Resource Management based on the analysis of cheating, monitoring, and sanctioning," Ecological Modelling, Elsevier, vol. 468(C).
    5. Planque, Benjamin & Aarflot, Johanna M. & Buttay, Lucie & Carroll, JoLynn & Fransner, Filippa & Hansen, Cecilie & Husson, Bérengère & Langangen, Øystein & Lindstrøm, Ulf & Pedersen, Torstein & Primice, 2022. "A standard protocol for describing the evaluation of ecological models," Ecological Modelling, Elsevier, vol. 471(C).
    6. An, Li & Grimm, Volker & Sullivan, Abigail & Turner II, B.L. & Malleson, Nicolas & Heppenstall, Alison & Vincenot, Christian & Robinson, Derek & Ye, Xinyue & Liu, Jianguo & Lindkvist, Emilie & Tang, W, 2021. "Challenges, tasks, and opportunities in modeling agent-based complex systems," Ecological Modelling, Elsevier, vol. 457(C).
    7. Watson, Joseph W & Boyd, Robin & Dutta, Ritabrata & Vasdekis, Georgios & Walker, Nicola D. & Roy, Shovonlal & Everitt, Richard & Hyder, Kieran & Sibly, Richard M, 2022. "Incorporating environmental variability in a spatially-explicit individual-based model of European sea bass✰," Ecological Modelling, Elsevier, vol. 466(C).
    8. Utomo, Dhanan Sarwo & Onggo, Bhakti Stephan & Eldridge, Stephen, 2018. "Applications of agent-based modelling and simulation in the agri-food supply chains," European Journal of Operational Research, Elsevier, vol. 269(3), pages 794-805.
    9. Mario Martinoli & Alessio Moneta & Gianluca Pallante, 2022. "Calibration and Validation of Macroeconomic Simulation Models by Statistical Causal Search," LEM Papers Series 2022/33, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    10. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    11. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    12. Coronese, Matteo & Occelli, Martina & Lamperti, Francesco & Roventini, Andrea, 2023. "AgriLOVE: Agriculture, land-use and technical change in an evolutionary, agent-based model," Ecological Economics, Elsevier, vol. 208(C).
    13. Grimm, Volker & Berger, Uta, 2016. "Structural realism, emergence, and predictions in next-generation ecological modelling: Synthesis from a special issue," Ecological Modelling, Elsevier, vol. 326(C), pages 177-187.
    14. Crouse, Kristin N. & Desai, Nisarg P. & Cassidy, Kira A. & Stahler, Erin E. & Lehman, Clarence L. & Wilson, Michael L., 2022. "Larger territories reduce mortality risk for chimpanzees, wolves, and agents: Multiple lines of evidence in a model validation framework," Ecological Modelling, Elsevier, vol. 471(C).
    15. repec:hal:spmain:info:hdl:2441/20hflp7eqn97boh50no50tv67n is not listed on IDEAS
    16. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    17. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    18. Lorscheid, Iris & Meyer, Matthias, 2016. "Divide and conquer: Configuring submodels for valid and efficient analyses of complex simulation models," Ecological Modelling, Elsevier, vol. 326(C), pages 152-161.
    19. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    20. Boult, Victoria L. & Quaife, Tristan & Fishlock, Vicki & Moss, Cynthia J. & Lee, Phyllis C. & Sibly, Richard M., 2018. "Individual-based modelling of elephant population dynamics using remote sensing to estimate food availability," Ecological Modelling, Elsevier, vol. 387(C), pages 187-195.
    21. Noeldeke, Beatrice & Winter, Etti & Ntawuhiganayo, Elisée Bahati, 2022. "Representing human decision-making in agent-based simulation models: Agroforestry adoption in rural Rwanda," Ecological Economics, Elsevier, vol. 200(C).
    22. Ayllón, Daniel & Railsback, Steven F. & Vincenzi, Simone & Groeneveld, Jürgen & Almodóvar, Ana & Grimm, Volker, 2016. "InSTREAM-Gen: Modelling eco-evolutionary dynamics of trout populations under anthropogenic environmental change," Ecological Modelling, Elsevier, vol. 326(C), pages 36-53.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:266186. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.