[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/zbw/safewp/315.html
   My bibliography  Save this paper

Expl(AI)ned: The impact of explainable Artificial Intelligence on cognitive processes

Author

Listed:
  • Bauer, Kevin
  • von Zahn, Moritz
  • Hinz, Oliver
Abstract
Due to a growing number of initiatives and regulations, predictions of modern Artificial Intelligence (AI) systems increasingly come with explanations about why they behave the way they do. In this paper, we explore the impact of feature-based explanations on users' information processing. We designed two complementary empirical studies where participants either made incentivized decisions on their own, with the aid of opaque predictions, or with explained predictions. In Study 1, laypeople engaged in the deliberately abstract investment game task. In Study 2, experts from the real-estate industry estimated listing prices for real German apartments. Our results indicate that the provision of feature-based explanations paves the way for AI systems to reshape users' sense-making of information and understanding of the world around them. Specifically, explanations change users' situational weighting of available information and evoke mental model adjustments. Crucially, mental model adjustments are subject to the confirmation bias so that misconceptions can persist and even accumulate, possibly leading to suboptimal or biased decisions. Additionally, mental model adjustments create spillover effects that alter user behavior in related yet disparate domains. Overall, this paper provides important insights into potential downstream consequences of the broad employment of modern explainable AI methods. In particular, side effects of mental model adjustments present a potential risk of manipulating user behavior and promoting discriminatory inclinations. Our findings may inform the refinement of current efforts of companies building AI systems and regulators that aim to mitigate problems associated with the black-box nature of many modern AI systems.

Suggested Citation

  • Bauer, Kevin & von Zahn, Moritz & Hinz, Oliver, 2022. "Expl(AI)ned: The impact of explainable Artificial Intelligence on cognitive processes," SAFE Working Paper Series 315, Leibniz Institute for Financial Research SAFE, revised 2022.
    • Handle: RePEc:zbw:safewp:315
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/268750/1/1837094039.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mitchell Hoffman & Lisa B Kahn & Danielle Li, 2018. "Discretion in Hiring," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(2), pages 765-800.
    2. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    3. repec:ner:ucllon:http://discovery.ucl.ac.uk/17678/ is not listed on IDEAS
    4. Berkeley J. Dietvorst & Joseph P. Simmons & Cade Massey, 2018. "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science, INFORMS, vol. 64(3), pages 1155-1170, March.
    5. Adam, Marc T.P. & Krämer, Jan & Müller, Marius B., 2015. "Auction Fever! How Time Pressure and Social Competition Affect Bidders’ Arousal and Bids in Retail Auctions," Journal of Retailing, Elsevier, vol. 91(3), pages 468-485.
    6. Andrew Prahl & Lyn Van Swol, 2017. "Understanding algorithm aversion: When is advice from automation discounted?," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 36(6), pages 691-702, September.
    7. Ernst Fehr & Urs Fischbacher, 2003. "The nature of human altruism," Nature, Nature, vol. 425(6960), pages 785-791, October.
    8. Aghion, Philippe & Tirole, Jean, 1997. "Formal and Real Authority in Organizations," Journal of Political Economy, University of Chicago Press, vol. 105(1), pages 1-29, February.
    9. Epstein Larry G & Noor Jawwad & Sandroni Alvaro, 2010. "Non-Bayesian Learning," The B.E. Journal of Theoretical Economics, De Gruyter, vol. 10(1), pages 1-20, January.
    10. Berg Joyce & Dickhaut John & McCabe Kevin, 1995. "Trust, Reciprocity, and Social History," Games and Economic Behavior, Elsevier, vol. 10(1), pages 122-142, July.
    11. Roland Bénabou & Jean Tirole, 2011. "Identity, Morals, and Taboos: Beliefs as Assets," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 805-855.
    12. Florian Zimmermann, 2020. "The Dynamics of Motivated Beliefs," American Economic Review, American Economic Association, vol. 110(2), pages 337-361, February.
    13. Logg, Jennifer M. & Minson, Julia A. & Moore, Don A., 2019. "Algorithm appreciation: People prefer algorithmic to human judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 90-103.
    14. Jason Chan & Jing Wang, 2018. "Hiring Preferences in Online Labor Markets: Evidence of a Female Hiring Bias," Management Science, INFORMS, vol. 64(7), pages 2973-2994, July.
    15. Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(1), pages 237-293.
    16. Matthew Rabin, 2013. "Incorporating Limited Rationality into Economics," Journal of Economic Literature, American Economic Association, vol. 51(2), pages 528-543, June.
    17. Baker, George & Gibbons, Robert & Murphy, Kevin J, 1999. "Informal Authority in Organizations," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 15(1), pages 56-73, April.
    18. Timo Henckel & Gordon D. Menzies & Peter G. Moffatt & Daniel J. Zizzo, 2022. "Belief adjustment: a double hurdle model and experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 26-67, February.
    19. Jan De Spiegeleer & Dilip B. Madan & Sofie Reyners & Wim Schoutens, 2018. "Machine learning for quantitative finance: fast derivative pricing, hedging and fitting," Quantitative Finance, Taylor & Francis Journals, vol. 18(10), pages 1635-1643, October.
    20. Viswanath Venkatesh & Fred D. Davis, 2000. "A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies," Management Science, INFORMS, vol. 46(2), pages 186-204, February.
    21. Jasbir S. Dhaliwal & Izak Benbasat, 1996. "The Use and Effects of Knowledge-Based System Explanations: Theoretical Foundations and a Framework for Empirical Evaluation," Information Systems Research, INFORMS, vol. 7(3), pages 342-362, September.
    22. Matthew Rabin & Joel L. Schrag, 1999. "First Impressions Matter: A Model of Confirmatory Bias," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(1), pages 37-82.
    23. Bonaccio, Silvia & Dalal, Reeshad S., 2006. "Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 101(2), pages 127-151, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kevin Bauer & Andrej Gill, 2024. "Mirror, Mirror on the Wall: Algorithmic Assessments, Transparency, and Self-Fulfilling Prophecies," Information Systems Research, INFORMS, vol. 35(1), pages 226-248, March.
    2. Bauer, Kevin & Gill, Andrej, 2021. "Mirror, mirror on the wall: Machine predictions and self-fulfilling prophecies," SAFE Working Paper Series 313, Leibniz Institute for Financial Research SAFE.
    3. Kevin Bauer & Moritz von Zahn & Oliver Hinz, 2023. "Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing," Information Systems Research, INFORMS, vol. 34(4), pages 1582-1602, December.
    4. Chugunova, Marina & Sele, Daniela, 2022. "We and It: An interdisciplinary review of the experimental evidence on how humans interact with machines," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 99(C).
    5. Markus Jung & Mischa Seiter, 2021. "Towards a better understanding on mitigating algorithm aversion in forecasting: an experimental study," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 32(4), pages 495-516, December.
    6. Merle, Aurélie & St-Onge, Anik & Sénécal, Sylvain, 2022. "Does it pay to be honest? The effect of retailer-provided negative feedback on consumers’ product choice and shopping experience," Journal of Business Research, Elsevier, vol. 147(C), pages 532-543.
    7. Ekaterina Jussupow & Kai Spohrer & Armin Heinzl & Joshua Gawlitza, 2021. "Augmenting Medical Diagnosis Decisions? An Investigation into Physicians’ Decision-Making Process with Artificial Intelligence," Information Systems Research, INFORMS, vol. 32(3), pages 713-735, September.
    8. Bauer, Kevin & Nofer, Michael & Abdel-Karim, Benjamin M. & Hinz, Oliver, 2022. "The effects of discontinuing machine learning decision support," SAFE Working Paper Series 370, Leibniz Institute for Financial Research SAFE.
    9. Keding, Christoph & Meissner, Philip, 2021. "Managerial overreliance on AI-augmented decision-making processes: How the use of AI-based advisory systems shapes choice behavior in R&D investment decisions," Technological Forecasting and Social Change, Elsevier, vol. 171(C).
    10. Christiane B. Haubitz & Cedric A. Lehmann & Andreas Fügener & Ulrich W. Thonemann, 2021. "The Risk of Algorithm Transparency: How Algorithm Complexity Drives the Effects on Use of Advice," ECONtribute Discussion Papers Series 078, University of Bonn and University of Cologne, Germany.
    11. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    12. Pascal Oliver Heßler & Jella Pfeiffer & Sebastian Hafenbrädl, 2022. "When Self-Humanization Leads to Algorithm Aversion," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(3), pages 275-292, June.
    13. Fumagalli, Elena & Rezaei, Sarah & Salomons, Anna, 2022. "OK computer: Worker perceptions of algorithmic recruitment," Research Policy, Elsevier, vol. 51(2).
    14. Mahmud, Hasan & Islam, A.K.M. Najmul & Ahmed, Syed Ishtiaque & Smolander, Kari, 2022. "What influences algorithmic decision-making? A systematic literature review on algorithm aversion," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    15. Kohei Kawaguchi, 2021. "When Will Workers Follow an Algorithm? A Field Experiment with a Retail Business," Management Science, INFORMS, vol. 67(3), pages 1670-1695, March.
    16. Benedikt Berger & Martin Adam & Alexander Rühr & Alexander Benlian, 2021. "Watch Me Improve—Algorithm Aversion and Demonstrating the Ability to Learn," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(1), pages 55-68, February.
    17. Robert M. Gillenkirch & Julia Ortner & Sebastian Robert & Louis Velthuis, 2023. "Designing incentives and performance measurement for advisors: How to make decision-makers listen to advice," Working Papers 2304, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.
    18. Christoph Keding, 2021. "Understanding the interplay of artificial intelligence and strategic management: four decades of research in review," Management Review Quarterly, Springer, vol. 71(1), pages 91-134, February.
    19. Scott Schanke & Gordon Burtch & Gautam Ray, 2021. "Estimating the Impact of “Humanizing” Customer Service Chatbots," Information Systems Research, INFORMS, vol. 32(3), pages 736-751, September.
    20. Bogliacino, Francesco & Codagnone, Cristiano, 2021. "Microfoundations, behaviour, and evolution: Evidence from experiments," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 372-385.

    More about this item

    Keywords

    Explainable Artificial Intelligence; User Behavior; Information Processing; Mental Models;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:safewp:315. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/csafede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.