[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/hal/pseptp/halshs-04409145.html
   My bibliography  Save this paper

Neural and computational underpinnings of biased confidence in human reinforcement learning

Author

Listed:
  • Chih-Chung Ting

    (UHH - Universität Hamburg)

  • Nahuel Salem-Garcia

    (CISA - Swiss Center for Affective Sciences - UNIGE - Université de Genève = University of Geneva)

  • Stefano Palminteri

    (ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres)

  • Jan Engelmann

    (ASE - Amsterdam School of Economics - UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam)

  • Maël Lebreton

    (CISA - Swiss Center for Affective Sciences - UNIGE - Université de Genève = University of Geneva, PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École des Ponts ParisTech - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, PJSE - Paris Jourdan Sciences Economiques - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École des Ponts ParisTech - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

Abstract
While navigating a fundamentally uncertain world, humans and animals constantly evaluate the probability of their decisions, actions or statements being correct. When explicitly elicited, these confidence estimates typically correlates positively with neural activity in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

Suggested Citation

  • Chih-Chung Ting & Nahuel Salem-Garcia & Stefano Palminteri & Jan Engelmann & Maël Lebreton, 2023. "Neural and computational underpinnings of biased confidence in human reinforcement learning," PSE-Ecole d'économie de Paris (Postprint) halshs-04409145, HAL.
    • Handle: RePEc:hal:pseptp:halshs-04409145
    DOI: 10.1038/s41467-023-42589-5
    as

    Download full text from publisher

    To our knowledge, this item is not available for download. To find whether it is available, there are three options:
    1. Check below whether another version of this item is available online.
    2. Check on the provider's web page whether it is in fact available.
    3. Perform a search for a similarly titled item that would be available.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano Palminteri & Mehdi Khamassi & Mateus Joffily & Giorgio Coricelli, 2015. "Contextual modulation of value signals in reward and punishment learning," Nature Communications, Nature, vol. 6(1), pages 1-14, November.
    2. Maël Lebreton & Sophie Bavard & Jean Daunizeau & Stefano Palminteri, 2019. "Assessing inter-individual differences with task-related functional neuroimaging," Nature Human Behaviour, Nature, vol. 3(9), pages 897-905, September.
    3. Jean Daunizeau & Vincent Adam & Lionel Rigoux, 2014. "VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data," PLOS Computational Biology, Public Library of Science, vol. 10(1), pages 1-16, January.
    4. Marine Hainguerlot & Jean-Christophe Vergnaud & Vincent de Gardelle, 2018. "Metacognitive ability predicts learning cue-stimulus associations in the absence of external feedback," PSE-Ecole d'économie de Paris (Postprint) hal-01761531, HAL.
    5. Sophie Bavard & Maël Lebreton & Mehdi Khamassi & Giorgio Coricelli & Stefano Palminteri, 2018. "Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
    6. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-01306258, HAL.
    7. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    8. Nadescha Trudel & Jacqueline Scholl & Miriam C. Klein-Flügge & Elsa Fouragnan & Lev Tankelevitch & Marco K. Wittmann & Matthew F. S. Rushworth, 2021. "Polarity of uncertainty representation during exploration and exploitation in ventromedial prefrontal cortex," Nature Human Behaviour, Nature, vol. 5(1), pages 83-98, January.
    9. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    10. Germain Lefebvre & Maël Lebreton & Florent Meyniel & Sacha Bourgeois-Gironde & Stefano Palminteri, 2017. "Behavioural and neural characterization of optimistic reinforcement learning," Nature Human Behaviour, Nature, vol. 1(4), pages 1-9, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
    2. Jean-Pierre Benoît & Juan Dubra & Giorgia Romagnoli, 2022. "Belief Elicitation When More than Money Matters: Controlling for "Control"," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 837-888, August.
    3. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    4. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    5. Stefano Palminteri & Germain Lefebvre & Emma J Kilford & Sarah-Jayne Blakemore, 2017. "Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing," PLOS Computational Biology, Public Library of Science, vol. 13(8), pages 1-22, August.
    6. Jan B. Engelmann & Maël Lebreton & Nahuel A. Salem-Garcia & Peter Schwardmann & Joël J. van der Weele, 2024. "Anticipatory Anxiety and Wishful Thinking," American Economic Review, American Economic Association, vol. 114(4), pages 926-960, April.
    7. Johann Lussange & Stefano Vrizzi & Stefano Palminteri & Boris Gutkin, 2024. "Modelling crypto markets by multi-agent reinforcement learning," Papers 2402.10803, arXiv.org.
    8. Juan Dubra & Jean-Pierre Benoît & Giorgia Romagnoli, 2019. "Belief elicitation when more than money matters," Documentos de Trabajo/Working Papers 1901, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..
    9. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    10. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    11. Gneezy, Uri & Saccardo, Silvia & Serra-Garcia, Marta & van Veldhuizen, Roel, 2020. "Bribing the Self," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 120, pages 311-324.
    12. Thomas Garcia & Sébastien Massoni, 2017. "Aiming to choose correctly or to choose wisely? The optimality-accuracy trade-off in decisions under uncertainty," Working Papers halshs-01631540, HAL.
    13. Lefebvre, Germain & Nioche, Aurélien & Bourgeois-Gironde, Sacha & Palminteri, Stefano, 2018. "An Empirical Investigation of the Emergence of Money: Contrasting Temporal Difference and Opportunity Cost Reinforcement Learning," MPRA Paper 85586, University Library of Munich, Germany.
    14. Johann Lussange & Boris Gutkin, 2023. "Order book regulatory impact on stock market quality: a multi-agent reinforcement learning perspective," Papers 2302.04184, arXiv.org.
    15. Fezzi, Carlo & Menapace, Luisa & Raffaelli, Roberta, 2021. "Estimating risk preferences integrating insurance choices with subjective beliefs," European Economic Review, Elsevier, vol. 135(C).
    16. Markus M. Möbius & Muriel Niederle & Paul Niehaus & Tanya S. Rosenblat, 2022. "Managing Self-Confidence: Theory and Experimental Evidence," Management Science, INFORMS, vol. 68(11), pages 7793-7817, November.
    17. Dmitri Vinogradov & Yousef Makhlouf, 2021. "Signaling probabilities in ambiguity: who reacts to vague news?," Theory and Decision, Springer, vol. 90(3), pages 371-404, May.
    18. Zahra Murad & Charitini Stavropoulou & Graham Cookson, 2019. "Incentives and gender in a multi-task setting: An experimental study with real-effort tasks," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-18, March.
    19. Bucciol, Alessandro & Quercia, Simone & Sconti, Alessia, 2021. "Promoting financial literacy among the elderly: Consequences on confidence," Journal of Economic Psychology, Elsevier, vol. 87(C).
    20. Murad, Zahra & Starmer, Chris, 2021. "Confidence snowballing and relative performance feedback," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 550-572.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:pseptp:halshs-04409145. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Caroline Bauer (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.