[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/inm/ormksc/v43y2024i2p254-266.html
   My bibliography  Save this article

Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis

Author

Listed:
  • Peiyao Li

    (Haas School of Business, University of California, Berkeley, California 94720)

  • Noah Castelo

    (Alberta School of Business, University of Alberta, Edmonton, Alberta T6G 2R6, Canada)

  • Zsolt Katona

    (Haas School of Business, University of California, Berkeley, California 94720)

  • Miklos Sarvary

    (Columbia Business School, Columbia University, New York, New York 10027)

Abstract
This paper explores the potential of large language models (LLMs) to substitute for human participants in market research. Such LLMs can be used to generate text given a prompt. We argue that perceptual analysis is a particularly promising use case for such automated market research for certain product categories. The proposed new method generates outputs that closely match those generated from human surveys: agreement rates between human- and LLM- generated data sets reach over 75%. Moreover, this applies for perceptual analysis based on both brand similarity measures and product attribute ratings. The paper demonstrates that, for some categories, this new method of fully or partially automated market research will increase the efficiency of market research by meaningfully speeding up the process and potentially reducing the cost. Further results also suggest that with an ever larger training corpus applied to large language models, LLM-based market research will be applicable to answer more nuanced questions based on demographic variables or contextual variation that would be prohibitively expensive or infeasible with human respondents.

Suggested Citation

  • Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    • Handle: RePEc:inm:ormksc:v:43:y:2024:i:2:p:254-266
    DOI: 10.1287/mksc.2023.0454
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mksc.2023.0454
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mksc.2023.0454?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jonah Berger & Grant Packard & Reihane Boghrati & Ming Hsu & Ashlee Humphreys & Andrea Luangrath & Sarah Moore & Gideon Nave & Christopher Olivola & Matthew Rocklage, 2022. "Marketing insights from text analysis," Marketing Letters, Springer, vol. 33(3), pages 365-377, September.
    2. Ashlee Humphreys & Rebecca Jen-Hui Wang & Eileen FischerEditor & Linda PriceAssociate Editor, 2018. "Automated Text Analysis for Consumer Research," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 44(6), pages 1274-1306.
    3. Aron Culotta & Jennifer Cutler, 2016. "Mining Brand Perceptions from Twitter Social Networks," Marketing Science, INFORMS, vol. 35(3), pages 343-362, May.
    4. Jonah Berger & Grant Packard & Reihane Boghrati & Ming Hsu & Ashlee Humphreys & Andrea Luangrath & Sarah Moore & Gideon Nave & Christopher Olivola & Matthew Rocklage, 2022. "Correction to: Marketing insights from text analysis," Marketing Letters, Springer, vol. 33(3), pages 379-379, September.
    5. Oded Netzer & Ronen Feldman & Jacob Goldenberg & Moshe Fresko, 2012. "Mine Your Own Business: Market-Structure Surveillance Through Text Mining," Marketing Science, INFORMS, vol. 31(3), pages 521-543, May.
    6. Liu Liu & Daria Dzyabura & Natalie Mizik, 2020. "Visual Listening In: Extracting Brand Image Portrayed on Social Media," Marketing Science, INFORMS, vol. 39(4), pages 669-686, July.
    7. Jia Liu & Olivier Toubia, 2018. "A Semantic Approach for Estimating Consumer Content Preferences from Online Search Queries," Marketing Science, INFORMS, vol. 37(6), pages 930-952, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hermann, Erik & Puntoni, Stefano, 2024. "Artificial intelligence and consumer behavior: From predictive to generative AI," Journal of Business Research, Elsevier, vol. 180(C).
    2. Jiangbo Yu & Graeme McKinley, 2024. "Synthetic Participatory Planning of Shared Automated Electric Mobility Systems," Sustainability, MDPI, vol. 16(13), pages 1-32, June.
    3. Shuaiyu Chen & T. Clifton Green & Huseyin Gulen & Dexin Zhou, 2024. "What Does ChatGPT Make of Historical Stock Returns? Extrapolation and Miscalibration in LLM Stock Return Forecasts," Papers 2409.11540, arXiv.org.
    4. Ning Li & Huaikang Zhou & Mingze Xu, 2024. "From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management," Papers 2408.05328, arXiv.org.
    5. Daniel Albert & Stephan Billinger, 2024. "Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models," Papers 2410.06932, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Grewal, Dhruv & Herhausen, Dennis & Ludwig, Stephan & Villarroel Ordenes, Francisco, 2022. "The Future of Digital Communication Research: Considering Dynamics and Multimodality," Journal of Retailing, Elsevier, vol. 98(2), pages 224-240.
    2. Ning Zhong & David A. Schweidel, 2020. "Capturing Changes in Social Media Content: A Multiple Latent Changepoint Topic Model," Marketing Science, INFORMS, vol. 39(4), pages 827-846, July.
    3. Hui Li & Jian Ni & Fangzhu Yang, 2024. "Product Design Using Generative Adversarial Network: Incorporating Consumer Preference and External Data," Papers 2405.15929, arXiv.org, revised Jun 2024.
    4. Maximilian Matthe & Daniel M. Ringel & Bernd Skiera, 2023. "Mapping Market Structure Evolution," Marketing Science, INFORMS, vol. 42(3), pages 589-613, May.
    5. Katsumata, Sotaro & Nishimoto, Akihiro & Kannan, P.K., 2023. "Brand competitiveness and resilience to exogenous shock: Usage of smartphone apps during the COVID-19 pandemic," Journal of Retailing and Consumer Services, Elsevier, vol. 75(C).
    6. Venkatesh Shankar & Sohil Parsana, 2022. "An overview and empirical comparison of natural language processing (NLP) models and an introduction to and empirical application of autoencoder models in marketing," Journal of the Academy of Marketing Science, Springer, vol. 50(6), pages 1324-1350, November.
    7. Pamuksuz, Utku & Yun, Joseph T. & Humphreys, Ashlee, 2021. "A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning," Journal of Interactive Marketing, Elsevier, vol. 56(C), pages 55-69.
    8. Fernando, Angeline Gautami & Aw, Eugene Cheng-Xi, 2023. "What do consumers want? A methodological framework to identify determinant product attributes from consumers’ online questions," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    9. Ishita Chakraborty & Minkyung Kim & K. Sudhir, 2019. "Attribute Sentiment Scoring With Online Text Reviews : Accounting for Language Structure and Attribute Self-Selection," Cowles Foundation Discussion Papers 2176, Cowles Foundation for Research in Economics, Yale University.
    10. Bitty Balducci & Detelina Marinova, 2018. "Unstructured data in marketing," Journal of the Academy of Marketing Science, Springer, vol. 46(4), pages 557-590, July.
    11. Zecong Ma & Sergio Palacios, 2021. "Image-mining: exploring the impact of video content on the success of crowdfunding," Journal of Marketing Analytics, Palgrave Macmillan, vol. 9(4), pages 265-285, December.
    12. Yoganathan, Vignesh & Osburg, Victoria-Sophie, 2024. "The mind in the machine: Estimating mind perception's effect on user satisfaction with voice-based conversational agents," Journal of Business Research, Elsevier, vol. 175(C).
    13. Paramveer S. Dhillon & Sinan Aral, 2021. "Modeling Dynamic User Interests: A Neural Matrix Factorization Approach," Marketing Science, INFORMS, vol. 40(6), pages 1059-1080, November.
    14. Xuan Gong & Yunchan Zhu & Rizwan Ali & Ruijin Guo, 2019. "Capturing Associations and Sustainable Competitiveness of Brands from Social Tags," Sustainability, MDPI, vol. 11(6), pages 1-20, March.
    15. Todd Pezzuti & James M. Leonhardt, 2023. "What’s not to like? Negations in brand messages increase consumer engagement," Journal of the Academy of Marketing Science, Springer, vol. 51(3), pages 675-694, May.
    16. Carlson, Keith & Kopalle, Praveen K. & Riddell, Allen & Rockmore, Daniel & Vana, Prasad, 2023. "Complementing human effort in online reviews: A deep learning approach to automatic content generation and review synthesis," International Journal of Research in Marketing, Elsevier, vol. 40(1), pages 54-74.
    17. Alzate, Miriam & Arce-Urriza, Marta & Cebollada, Javier, 2022. "Mining the text of online consumer reviews to analyze brand image and brand positioning," Journal of Retailing and Consumer Services, Elsevier, vol. 67(C).
    18. Mitra, Satanik & Jenamani, Mamata, 2020. "OBIM: A computational model to estimate brand image from online consumer review," Journal of Business Research, Elsevier, vol. 114(C), pages 213-226.
    19. Li, Xi & Shi, Mengze & Wang, Xin (Shane), 2019. "Video mining: Measuring visual information using automatic methods," International Journal of Research in Marketing, Elsevier, vol. 36(2), pages 216-231.
    20. Gensler, Sonja & Oehring, Karlo & Wiesel, Thorsten, 2024. "Reported and communicated shifts in strategic emphasis and firm performance," International Journal of Research in Marketing, Elsevier, vol. 41(2), pages 220-240.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormksc:v:43:y:2024:i:2:p:254-266. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.