[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/inm/ororsc/v29y2018i6p1010-1032.html
   My bibliography  Save this article

Learning from Mixed Signals in Online Innovation Communities

Author

Listed:
  • Christoph Riedl

    (D'Amore-McKim School of Business, and College of Computer and Information Science, Northeastern University, Boston, Massachusetts 02115; Institute for Quantitative Social Science, Harvard University, Cambridge, Massachusetts 02138;)

  • Victor P. Seidel

    (F.W. Olin Graduate School of Business, Babson College, Babson Park, Massachusetts 02457; Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138; Said Business School, University of Oxford, Oxford OX1 1HP, United Kingdom)

Abstract
We study how contributors to innovation contests improve their performance through direct experience and by observing others as they synthesize learnable signals from different sources. Our research draws on a 10-year panel of more than 55,000 individuals participating in a firm-hosted online innovation community sponsoring creative t-shirt design contests. Our data set contains almost 180,000 submissions that reflect signals of direct performance evaluation from both the community and the firm. Our data set also contains almost 150 million ratings that reflect signals for learning from observing the completed work of others. We have three key findings. First, we find a period of initial investment with decreased performance. This is because individuals struggle to synthesize learnable signals from early performance evaluation. This finding is contrary to other studies that report faster learning from early direct experience when improvements are easiest to achieve. Second, we find that individuals consistently improve their performance from observing others’ good examples. However, whether they improve from observing others’ bad examples depends on their ability to correctly recognize that work as being of low quality. Third, we find that individuals can successfully integrate signals about what is valued by the firm hosting the community, not just about what is valued by the community. We thus provide important insights into the mechanisms of how individuals learn in crowdsourced innovation and provide important qualifications for the often-heralded theme of “learning from failures.”

Suggested Citation

  • Christoph Riedl & Victor P. Seidel, 2018. "Learning from Mixed Signals in Online Innovation Communities," Organization Science, INFORMS, vol. 29(6), pages 1010-1032, December.
    • Handle: RePEc:inm:ororsc:v:29:y:2018:i:6:p:1010-1032
    DOI: 10.1287/orsc.2018.1219
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/orsc.2018.1219
    Download Restriction: no
    File URL: https://libkey.io/10.1287/orsc.2018.1219?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Einhorn, Hj & Hogarth, Rm, 1981. "Behavioral Decision-Theory - Processes Of Judgment And Choice," Journal of Accounting Research, Wiley Blackwell, vol. 19(1), pages 1-31.
    2. Robert Moffitt & John Fitzgerald & Peter Gottschalk, 1999. "Sample Attrition in Panel Data: The Role of Selection on Observables," Annals of Economics and Statistics, GENES, issue 55-56, pages 129-152.
    3. John Fitzgerald & Peter Gottschalk & Robert Moffitt, 1998. "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 251-299.
    4. Foster, Andrew D & Rosenzweig, Mark R, 1995. "Learning by Doing and Learning from Others: Human Capital and Technical Change in Agriculture," Journal of Political Economy, University of Chicago Press, vol. 103(6), pages 1176-1209, December.
    5. Christian Hildebrand & Gerald Häubl & Andreas Herrmann & Jan R. Landwehr, 2013. "When Social Media Can Be Bad for You: Community Feedback Stifles Consumer Creativity and Reduces Satisfaction with Self-Designed Products," Information Systems Research, INFORMS, vol. 24(1), pages 14-29, March.
    6. Eric D. Darr & Linda Argote & Dennis Epple, 1995. "The Acquisition, Transfer, and Depreciation of Knowledge in Service Organizations: Productivity in Franchises," Management Science, INFORMS, vol. 41(11), pages 1750-1762, November.
    7. Argote, L. & Epple, D., 1990. "Learning Curves In Manufacturing," GSIA Working Papers 89-90-02, Carnegie Mellon University, Tepper School of Business.
    8. Bikhchandani, Sushil & Hirshleifer, David & Welch, Ivo, 1992. "A Theory of Fads, Fashion, Custom, and Cultural Change in Informational Cascades," Journal of Political Economy, University of Chicago Press, vol. 100(5), pages 992-1026, October.
    9. Boudreau, Kevin J. & Lakhani, Karim R., 2015. "“Open” disclosure of innovations, incentives and follow-on reuse: Theory on processes of cumulative innovation and a field experiment in computational biology," Research Policy, Elsevier, vol. 44(1), pages 4-19.
    10. Yates, Joanne & Orlikowski, Wanda J. & Woerner, Stephanie, 2003. "Virtual Organizing: Using Threads to Coordinate Distributed Work," Working papers 4320-03, Massachusetts Institute of Technology (MIT), Sloan School of Management.
    11. Kevin J. Boudreau & Karim R. Lakhani & Michael Menietti, 2016. "Performance responses to competition across skill levels in rank-order tournaments: field evidence and implications for tournament design," RAND Journal of Economics, RAND Corporation, vol. 47(1), pages 140-165, February.
    12. Laura J. Kornish & Karl T. Ulrich, 2011. "Opportunity Spaces in Innovation: Empirical Analysis of Large Samples of Ideas," Management Science, INFORMS, vol. 57(1), pages 107-128, January.
    13. Linda Argote & Bill McEvily & Ray Reagans, 2003. "Managing Knowledge in Organizations: An Integrative Framework and Review of Emerging Themes," Management Science, INFORMS, vol. 49(4), pages 571-582, April.
    14. Verbeek, Marno & Nijman, Theo, 1992. "Testing for Selectivity Bias in Panel Data Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 33(3), pages 681-703, August.
    15. Jeffrey M. Wooldridge, 2002. "Inverse probability weighted M-estimators for sample selection, attrition, and stratification," Portuguese Economic Journal, Springer;Instituto Superior de Economia e Gestao, vol. 1(2), pages 117-139, August.
    16. Rebecca Achee Thornton & Peter Thompson, 2001. "Learning from Experience and Learning from Others: An Exploration of Learning and Spillovers in Wartime Shipbuilding," American Economic Review, American Economic Association, vol. 91(5), pages 1350-1368, December.
    17. K. J. Arrow, 1971. "The Economic Implications of Learning by Doing," Palgrave Macmillan Books, in: F. H. Hahn (ed.), Readings in the Theory of Growth, chapter 11, pages 131-149, Palgrave Macmillan.
    18. Nijman, T.E. & Verbeek, M.J.C.M., 1992. "Testing for selectivity in panel data models," Other publications TiSEM 7ec34a6c-1d84-4052-971c-d, Tilburg University, School of Economics and Management.
    19. Dahlander, Linus & Piezunka, Henning, 2014. "Open to suggestions: How organizations elicit suggestions through proactive and reactive attention," Research Policy, Elsevier, vol. 43(5), pages 812-827.
    20. Victor P. Seidel & Benedikt Langner, 2015. "Using an online community for vehicle design: project variety and motivations to participate," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 24(3), pages 635-653.
    21. Jeffrey A. Roberts & Il-Horn Hann & Sandra A. Slaughter, 2006. "Understanding the Motivations, Participation, and Performance of Open Source Software Developers: A Longitudinal Study of the Apache Projects," Management Science, INFORMS, vol. 52(7), pages 984-999, July.
    22. James G. March, 1991. "Exploration and Exploitation in Organizational Learning," Organization Science, INFORMS, vol. 2(1), pages 71-87, February.
    23. Param Vir Singh & Yong Tan & Nara Youn, 2011. "A Hidden Markov Model of Developer Learning Dynamics in Open Source Software Projects," Information Systems Research, INFORMS, vol. 22(4), pages 790-807, December.
    24. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    25. Ikujiro Nonaka, 1994. "A Dynamic Theory of Organizational Knowledge Creation," Organization Science, INFORMS, vol. 5(1), pages 14-37, February.
    26. Sherwin Rosen, 1972. "Learning by Experience as Joint Production," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 86(3), pages 366-382.
    27. Ann Majchrzak & Arvind Malhotra, 2016. "Effect of Knowledge-Sharing Trajectories on Innovative Outcomes in Temporary Online Crowds," Information Systems Research, INFORMS, vol. 27(4), pages 685-703, December.
    28. Eelke Wiersma, 2007. "Conditions That Shape the Learning Curve: Factors That Increase the Ability and Opportunity to Learn," Management Science, INFORMS, vol. 53(12), pages 1903-1915, December.
    29. von Krogh, Georg & Spaeth, Sebastian & Lakhani, Karim R., 2003. "Community, joining, and specialization in open source software innovation: a case study," Research Policy, Elsevier, vol. 32(7), pages 1217-1241, July.
    30. Natalia Levina & Manuel Arriaga, 2014. "Distinction and Status Production on User-Generated Content Platforms: Using Bourdieu’s Theory of Cultural Production to Understand Social Dynamics in Online Fields," Information Systems Research, INFORMS, vol. 25(3), pages 468-488, September.
    31. Christian Terwiesch & Yi Xu, 2008. "Innovation Contests, Open Innovation, and Multiagent Problem Solving," Management Science, INFORMS, vol. 54(9), pages 1529-1543, September.
    32. Ivo Blohm & Christoph Riedl & Johann Füller & Jan Marco Leimeister, 2016. "Rate or Trade? Identifying Winning Ideas in Open Idea Sourcing," Information Systems Research, INFORMS, vol. 27(1), pages 27-48, March.
    33. Yan Huang & Param Vir Singh & Kannan Srinivasan, 2014. "Crowdsourcing New Product Ideas Under Consumer Learning," Management Science, INFORMS, vol. 60(9), pages 2138-2159, September.
    34. Michael Barrett & Eivor Oborn & Wanda Orlikowski, 2016. "Creating Value in Online Communities: The Sociomaterial Configuring of Strategy, Platform, and Stakeholder Engagement," Information Systems Research, INFORMS, vol. 27(4), pages 704-723, December.
    35. Kevin J. Boudreau & Nicola Lacetera & Karim R. Lakhani, 2011. "Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis," Management Science, INFORMS, vol. 57(5), pages 843-863, May.
    36. Lars Bo Jeppesen & Karim R. Lakhani, 2010. "Marginality and Problem-Solving Effectiveness in Broadcast Search," Organization Science, INFORMS, vol. 21(5), pages 1016-1033, October.
    37. Georg von Krogh & Eric von Hippel, 2006. "The Promise of Research on Open Source Software," Management Science, INFORMS, vol. 52(7), pages 975-983, July.
    38. Abhijit V. Banerjee, 1992. "A Simple Model of Herd Behavior," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 107(3), pages 797-817.
    39. Dennis Epple & Linda Argote & Kenneth Murphy, 1996. "An Empirical Investigation of the Microstructure of Knowledge Acquisition and Transfer Through Learning by Doing," Operations Research, INFORMS, vol. 44(1), pages 77-86, February.
    40. repec:adr:anecst:y:1999:i:55-56:p:05 is not listed on IDEAS
    41. Janice Nadler & Leigh Thompson & Leaf Van Boven, 2003. "Learning Negotiation Skills: Four Models of Knowledge Creation and Transfer," Management Science, INFORMS, vol. 49(4), pages 529-540, April.
    42. Rema Hanna & Sendhil Mullainathan & Joshua Schwartzstein, 2014. "Learning Through Noticing: Theory and Evidence from a Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(3), pages 1311-1353.
    43. Sushil Bikhchandani & David Hirshleifer & Ivo Welch, 1998. "Learning from the Behavior of Others: Conformity, Fads, and Informational Cascades," Journal of Economic Perspectives, American Economic Association, vol. 12(3), pages 151-170, Summer.
    44. Dahlander, Linus & Magnusson, Mats G., 2005. "Relationships between open source software companies and communities: Observations from Nordic firms," Research Policy, Elsevier, vol. 34(4), pages 481-493, May.
    45. Beth A. Bechky, 2003. "Sharing Meaning Across Occupational Communities: The Transformation of Understanding on a Production Floor," Organization Science, INFORMS, vol. 14(3), pages 312-330, June.
    46. Julia Bauer & Nikolaus Franke & Philipp Tuertscher, 2016. "Intellectual Property Norms in Online Communities: How User-Organized Intellectual Property Regulation Supports Innovation," Information Systems Research, INFORMS, vol. 27(4), pages 724-750, December.
    47. Lars Bo Jeppesen & Lars Frederiksen, 2006. "Why Do Users Contribute to Firm-Hosted User Communities? The Case of Computer-Controlled Music Instruments," Organization Science, INFORMS, vol. 17(1), pages 45-63, February.
    48. Stefan H. Thomke, 1998. "Managing Experimentation in the Design of New Products," Management Science, INFORMS, vol. 44(6), pages 743-762, June.
    49. Diwas KC & Bradley R. Staats & Francesca Gino, 2013. "Learning from My Success and from Others' Failure: Evidence from Minimally Invasive Cardiac Surgery," Management Science, INFORMS, vol. 59(11), pages 2435-2449, November.
    50. John Seely Brown & Paul Duguid, 1991. "Organizational Learning and Communities-of-Practice: Toward a Unified View of Working, Learning, and Innovation," Organization Science, INFORMS, vol. 2(1), pages 40-57, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David Clingingsmith & Will Drover & Scott Shane, 2023. "Examining the outcomes of entrepreneur pitch training: an exploratory field study," Small Business Economics, Springer, vol. 60(3), pages 947-974, March.
    2. Pankaj Kumar & Xiaojin Liu & Akbar Zaheer, 2022. "How much does the firm's alliance network matter?," Strategic Management Journal, Wiley Blackwell, vol. 43(8), pages 1433-1468, August.
    3. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.
    4. Quignon, Aurelien, 2023. "Crowd-based feedback and early-stage entrepreneurial performance: Evidence from a digital platform," Research Policy, Elsevier, vol. 52(7).
    5. Tommy Pan Fang & Andy Wu & David R. Clough, 2021. "Platform diffusion at temporary gatherings: Social coordination and ecosystem emergence," Strategic Management Journal, Wiley Blackwell, vol. 42(2), pages 233-272, February.
    6. Ren, Jie & Han, Yue & Genc, Yegin & Yeoh, William & Popovič, Aleš, 2021. "The boundary of crowdsourcing in the domain of creativity✰," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    7. Haijing Hao & Rema Padman & Baohong Sun & Rahul Telang, 2019. "Modeling social learning on consumers’ long-term usage of a mobile technology: a Bayesian estimation of a Bayesian learning model," Electronic Commerce Research, Springer, vol. 19(1), pages 1-21, March.
    8. Oliver Alexy, 2022. "How flat can it get? From better at flatter to the promise of the decentralized, boundaryless organization," Journal of Organization Design, Springer;Organizational Design Community, vol. 11(1), pages 31-36, March.
    9. Victor P. Seidel & Christoph Riedl, 2023. "How creative versus technical constraints affect individual learning in an online innovation community," Papers 2303.15163, arXiv.org.
    10. Smirnova, Inna & Reitzig, Markus & Alexy, Oliver, 2022. "What makes the right OSS contributor tick? Treatments to motivate high-skilled developers," Research Policy, Elsevier, vol. 51(1).
    11. Muninger, Marie-Isabelle & Mahr, Dominik & Hammedi, Wafa, 2022. "Social media use: A review of innovation management practices," Journal of Business Research, Elsevier, vol. 143(C), pages 140-156.
    12. Dahlander, Linus & Beretta, Michela & Thomas, Arne & Kazemi, Shahab & Fenger, Morten H.J. & Frederiksen, Lars, 2023. "Weeding out or picking winners in open innovation? Factors driving multi-stage crowd selection on LEGO ideas," Research Policy, Elsevier, vol. 52(10).
    13. Wachs, Johannes & Nitecki, Mariusz & Schueller, William & Polleres, Axel, 2022. "The Geography of Open Source Software: Evidence from GitHub," Technological Forecasting and Social Change, Elsevier, vol. 176(C).
    14. Tat Koon Koh & Muller Y. M. Cheung, 2022. "Seeker Exemplars and Quantitative Ideation Outcomes in Crowdsourcing Contests," Information Systems Research, INFORMS, vol. 33(1), pages 265-284, March.
    15. Christoph Riedl & Tom Grad & Christopher Lettl, 2024. "Competition and Collaboration in Crowdsourcing Communities: What happens when peers evaluate each other?," Papers 2404.14141, arXiv.org.
    16. Sheen S. Levine & Michael J. Prietula & Ann Majchrzak, 2022. "Advice in Crisis: Principles of Organizational and Entrepreneurial Resilience," Journal of Organization Design, Springer;Organizational Design Community, vol. 11(4), pages 145-168, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Samer Faraj & Georg von Krogh & Eric Monteiro & Karim R. Lakhani, 2016. "Special Section Introduction—Online Community as Space for Knowledge Flows," Information Systems Research, INFORMS, vol. 27(4), pages 668-684, December.
    2. Linda Argote & Sunkee Lee & Jisoo Park, 2021. "Organizational Learning Processes and Outcomes: Major Findings and Future Research Directions," Management Science, INFORMS, vol. 67(9), pages 5399-5429, September.
    3. Christoph Riedl & Tom Grad & Christopher Lettl, 2024. "Competition and Collaboration in Crowdsourcing Communities: What happens when peers evaluate each other?," Papers 2404.14141, arXiv.org.
    4. Yuan Jin & Ho Cheung Brian Lee & Sulin Ba & Jan Stallaert, 2021. "Winning by Learning? Effect of Knowledge Sharing in Crowdsourcing Contests," Information Systems Research, INFORMS, vol. 32(3), pages 836-859, September.
    5. Avner Ben-Ner & Stephanie Lluis, 2011. "Learning: What and How? An Empirical Study of Adjustments in Workplace Organization Structure," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 50(1), pages 76-108, January.
    6. Linda Argote & Ella Miron-Spektor, 2011. "Organizational Learning: From Experience to Knowledge," Organization Science, INFORMS, vol. 22(5), pages 1123-1137, October.
    7. Thompson, Peter, 2010. "Learning by Doing," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 1, chapter 0, pages 429-476, Elsevier.
    8. Ethan Mollick & Ramana Nanda, 2016. "Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts," Management Science, INFORMS, vol. 62(6), pages 1533-1553, June.
    9. Zaggl, Michael A., 2017. "Manipulation of explicit reputation in innovation and knowledge exchange communities: The example of referencing in science," Research Policy, Elsevier, vol. 46(5), pages 970-983.
    10. repec:eee:respol:v:48:y:2019:i:8:p:- is not listed on IDEAS
    11. Adrián Kovács & Bart Looy & Bruno Cassiman, 2015. "Exploring the scope of open innovation: a bibliometric review of a decade of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 951-983, September.
    12. Sheen S. Levine & Michael J. Prietula, 2012. "How Knowledge Transfer Impacts Performance: A Multilevel Model of Benefits and Liabilities," Organization Science, INFORMS, vol. 23(6), pages 1748-1766, December.
    13. Shunyuan Zhang & Param Vir Singh & Anindya Ghose, 2019. "A Structural Analysis of the Role of Superstars in Crowdsourcing Contests," Service Science, INFORMS, vol. 30(1), pages 15-33, March.
    14. Kexin Zhao & Bin Zhang & Xue Bai, 2018. "Estimating Contextual Motivating Factors in Virtual Interorganizational Communities of Practice: Peer Effects and Organizational Influences," Information Systems Research, INFORMS, vol. 29(4), pages 910-927, December.
    15. Vipul Aggarwal & Elina H. Hwang & Yong Tan, 2021. "Learning to Be Creative: A Mutually Exciting Spatiotemporal Point Process Model for Idea Generation in Open Innovation," Information Systems Research, INFORMS, vol. 32(4), pages 1214-1235, December.
    16. Dahlander, Linus & Beretta, Michela & Thomas, Arne & Kazemi, Shahab & Fenger, Morten H.J. & Frederiksen, Lars, 2023. "Weeding out or picking winners in open innovation? Factors driving multi-stage crowd selection on LEGO ideas," Research Policy, Elsevier, vol. 52(10).
    17. Param Vir Singh & Yong Tan & Nara Youn, 2011. "A Hidden Markov Model of Developer Learning Dynamics in Open Source Software Projects," Information Systems Research, INFORMS, vol. 22(4), pages 790-807, December.
    18. Megan Lawrence, 2020. "Replication using templates: Does the unit learn from itself, the template, or both?," Strategic Management Journal, Wiley Blackwell, vol. 41(11), pages 1955-1982, November.
    19. Patel, Chirag & Ahmad Husairi, Mariyani & Haon, Christophe & Oberoi, Poonam, 2023. "Monetary rewards and self-selection in design crowdsourcing contests: Managing participation, contribution appropriateness, and winning trade-offs," Technological Forecasting and Social Change, Elsevier, vol. 191(C).
    20. Smirnova, Inna & Reitzig, Markus & Alexy, Oliver, 2022. "What makes the right OSS contributor tick? Treatments to motivate high-skilled developers," Research Policy, Elsevier, vol. 51(1).
    21. Ho Cheung Brian Lee & Sulin Ba & Xinxin Li & Jan Stallaert, 2018. "Salience Bias in Crowdsourcing Contests," Information Systems Research, INFORMS, vol. 29(2), pages 401-418, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ororsc:v:29:y:2018:i:6:p:1010-1032. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.