[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/23470.html
   My bibliography  Save this paper

Measuring and Bounding Experimenter Demand

Author

Listed:
  • Jonathan de Quidt
  • Johannes Haushofer
  • Christopher Roth
Abstract
We propose a technique for assessing robustness of behavioral measures and treatment effects to experimenter demand effects. The premise is that by deliberately inducing demand in a structured way we can measure its influence and construct plausible bounds on demand-free behavior. We provide formal restrictions on choice that validate our method, and a Bayesian model that microfounds them. Seven pre-registered experiments with eleven canonical laboratory games and around 19,000 participants demonstrate the technique. We also illustrate how demand sensitivity varies by task, participant pool, gender, real versus hypothetical incentives, and participant attentiveness, and provide both reduced-form and structural analyses of demand effects.

Suggested Citation

  • Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2017. "Measuring and Bounding Experimenter Demand," NBER Working Papers 23470, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:23470
    Note: DEV LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w23470.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Aurelio Bruzzo, 2018. "Recenti iniziative europee ed italiane per la valorizzazione del patrimonio culturale," Working Papers 2018127, University of Ferrara, Department of Economics.
    2. Stefano DellaVigna & John A. List & Ulrike Malmendier, 2012. "Testing for Altruism and Social Pressure in Charitable Giving," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 1-56.
    3. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    4. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard H, 1986. "Fairness and the Assumptions of Economics," The Journal of Business, University of Chicago Press, vol. 59(4), pages 285-300, October.
    5. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    6. Eran Shmaya & Leeat Yariv, 2016. "Experiments on Decisions under Uncertainty: A Theoretical Framework," American Economic Review, American Economic Association, vol. 106(7), pages 1775-1801, July.
    7. Aviv Nevo & Adam M. Rosen, 2012. "Identification With Imperfect Instruments," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 659-671, August.
    8. Ingvild Almås & Alexander W. Cappelen & Bertil Tungodden, 2020. "Cutthroat Capitalism versus Cuddly Socialism: Are Americans More Meritocratic and Efficiency-Seeking than Scandinavians?," Journal of Political Economy, University of Chicago Press, vol. 128(5), pages 1753-1788.
    9. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    10. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    11. Thaler, Richard H, 1988. "The Ultimatum Game," Journal of Economic Perspectives, American Economic Association, vol. 2(4), pages 195-206, Fall.
    12. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    13. Urs Fischbacher & Franziska Föllmi-Heusi, 2013. "Lies In Disguise—An Experimental Study On Cheating," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 525-547, June.
    14. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    15. Stefano DellaVigna & Devin Pope, 2018. "What Motivates Effort? Evidence and Expert Forecasts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(2), pages 1029-1069.
    16. John A. List & Robert P. Berrens & Alok K. Bohara & Joe Kerkvliet, 2004. "Examining the Role of Social Isolation on Stated Preferences," American Economic Review, American Economic Association, vol. 94(3), pages 741-752, June.
    17. James Andreoni & Charles Sprenger, 2012. "Estimating Time Preferences from Convex Budgets," American Economic Review, American Economic Association, vol. 102(7), pages 3333-3356, December.
    18. Guido W. Imbens & Charles F. Manski, 2004. "Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 72(6), pages 1845-1857, November.
    19. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    20. repec:feb:framed:0087 is not listed on IDEAS
    21. Guth, Werner & Schmittberger, Rolf & Schwarze, Bernd, 1982. "An experimental analysis of ultimatum bargaining," Journal of Economic Behavior & Organization, Elsevier, vol. 3(4), pages 367-388, December.
    22. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    23. Berg Joyce & Dickhaut John & McCabe Kevin, 1995. "Trust, Reciprocity, and Social History," Games and Economic Behavior, Elsevier, vol. 10(1), pages 122-142, July.
    24. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    25. Hoffman, Elizabeth & McCabe, Kevin A & Smith, Vernon L, 1996. "On Expectations and the Monetary Stakes in Ultimatum Games," International Journal of Game Theory, Springer;Game Theory Society, vol. 25(3), pages 289-301.
    26. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    27. Stefano Dellavigna & John A. List & Ulrike Malmendier & Gautam Rao, 2017. "Voting to Tell Others," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 84(1), pages 143-181.
    28. Binmore, K & Shaked, A & Sutton, J, 1985. "Testing Noncooperative Bargaining Theory: A Preliminary Study," American Economic Review, American Economic Association, vol. 75(5), pages 1178-1180, December.
    29. Gerber, Alan S. & Green, Donald P. & Larimer, Christopher W., 2008. "Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment," American Political Science Review, Cambridge University Press, vol. 102(1), pages 33-48, February.
    30. Nicholas Bardsley, 2008. "Dictator game giving: altruism or artefact?," Experimental Economics, Springer;Economic Science Association, vol. 11(2), pages 122-133, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Emin Karagözoğlu & Ümit Barış Urhan, 2017. "The Effect of Stake Size in Experimental Bargaining and Distribution Games: A Survey," Group Decision and Negotiation, Springer, vol. 26(2), pages 285-325, March.
    3. Weimann Joachim, 2015. "Die Rolle von Verhaltensökonomik und experimenteller Forschung in Wirtschaftswissenschaft und Politikberatung," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 16(3), pages 231-252, October.
    4. Bharat Chandar & Uri Gneezy & John A. List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," NBER Working Papers 26380, National Bureau of Economic Research, Inc.
    5. Carlos Alós-Ferrer & Jaume García-Segarra & Alexander Ritschel, 2018. "The Big Robber Game," ECON - Working Papers 291, Department of Economics - University of Zurich.
    6. Aycinena, Diego & Bogliacino, Francesco & Kimbrough, Erik O., 2024. "Measuring norms: Assessing the threat of social desirability bias to the Bicchieri and Xiao elicitation method," Journal of Economic Behavior & Organization, Elsevier, vol. 222(C), pages 225-239.
    7. Larney, Andrea & Rotella, Amanda & Barclay, Pat, 2019. "Stake size effects in ultimatum game and dictator game offers: A meta-analysis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 61-72.
    8. Brit Grosskopf & Graeme Pearce, 2020. "Do You Mind Me Paying Less? Measuring Other-Regarding Preferences in the Market for Taxis," Management Science, INFORMS, vol. 66(11), pages 5059-5074, November.
    9. Walkowitz, Gari, 2019. "On the Validity of Probabilistic (and Cost-Saving) Incentives in Dictator Games: A Systematic Test," MPRA Paper 91541, University Library of Munich, Germany.
    10. Rami Zwick & Xiao-Ping Chen, 1999. "What Price Fairness? A Bargaining Study," Management Science, INFORMS, vol. 45(6), pages 804-823, June.
    11. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    12. Thomas Buser, 2010. "Handedness predicts Social Preferences: Evidence connecting the Lab to the Field," Tinbergen Institute Discussion Papers 10-119/3, Tinbergen Institute.
    13. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    14. Walkowitz, Gari, 2021. "Dictator game variants with probabilistic (and cost-saving) payoffs: A systematic test," Journal of Economic Psychology, Elsevier, vol. 85(C).
    15. Burnham, Terence C., 2013. "Toward a neo-Darwinian synthesis of neoclassical and behavioral economics," Journal of Economic Behavior & Organization, Elsevier, vol. 90(S), pages 113-127.
    16. Pin, Paolo & Rotesi, Tiziano, 2023. "App-based experiments," Journal of Economic Psychology, Elsevier, vol. 99(C).
    17. Walkowitz, Gari, 2017. "On the Validity of Cost-Saving Methods in Dictator-Game Experiments: A Systematic Test," MPRA Paper 83309, University Library of Munich, Germany.
    18. Fredrik Carlsson & Haoran He & Peter Martinsson, 2013. "Easy come, easy go," Experimental Economics, Springer;Economic Science Association, vol. 16(2), pages 190-207, June.
    19. Mehmood, Sultan & Naseer, Shaheen & Chen, Daniel L., 2024. "Altruism in governance: Insights from randomized training for Pakistan's junior ministers," Journal of Development Economics, Elsevier, vol. 170(C).
    20. Murnighan, J. Keith & Wang, Long, 2016. "The social world as an experimental game," Organizational Behavior and Human Decision Processes, Elsevier, vol. 136(C), pages 80-94.

    More about this item

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:23470. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.