Research:CSCW15 workshop/Notes

This page contains a proposal for a workshop to be submitted to CSCW'15 (deadline: August 8, 2014)

Open collaboration communities like Wikipedia, GalaxyZoo, Reddit, and Imgur support self-organized, cooperative work by distributed networks of volunteers. Both CSCW researchers who study these online communities and industry practitioners who support these communities through research need to understand phenomena related to governance, quality control, motivation, and socialization within these systems. However, there are few opportunities for academic and industry researchers to communicate and collaborate.

This workshop will bring researchers from these two arenas together to develop a framework for performing open collaboration research that supports the work of the communities being studied and also advances scientific understanding. Workshop participants will share methods, tools, data, and research findings in order to identify key research questions in open collaboration research and discuss best practices for academic/industry research partnerships.

Open collaboration system (OCS)

^[1] defines an open collaboration system as an online environment that

1. supports the collective production of an artifact
2. through a technologically mediated collaboration platform
3. that presents a low barrier to entry and exit, and
4. supports the emergence of persistent but malleable social structures.

Nature

• Identify key areas where research is necessary to support open collaboration
• Develop shared understanding of open questions for practitioners and academics

Data

• Pilot a data repository by publishing a small set of well-documented and compelling OC datasets
• Develop shared understanding of the kind of data that academic researchers have/need
• Develop shared understand of the kind of data that industry researchers have/need

Future collaboration

• Propose a framework for collaboration between researchers in industry/academia
• Academic researchers perform research that can provide actionable insights for indistry data scientists, community managers

• Researching Online Communities Workshop
• Nature and Data of Open Collaboration Workshop
• Open Collaboration Workshop
• Towards Industry/Academic Partnership on Researching Open Collaboration
• Sharing Data & Goals: Industry/Academic Partnership on Researching Open Collaboration

Open collaboration communities have inverted the traditional publishing model: Through the use of technologies built for the internet, information consumers can take part in (or even take ownership of) production itself. While removing barriers to participation dramatically boosts the number of person-hours that can be dedicated to production, it introduces a novel set of problems. These include <list topics from below here>

Allocation (of work)

• Nature:
  • Natural patterns of work & collaboration in open source software ^[2]
  • Natural patterns of newcomers to start by fixing mistakes & adding missing content in Wikipedia ^[3]
  • The progressive reduction in work-rate over time for Wikipedians ^[4]
  • Natural clustering of editing activity around Wikipedia articles related to breaking news events ^[5]
  • In Wikipedia different distributions of work on an article lead to different outcomes. ^[6]
  • Work investment per contributor tends to follow a powerlaw distribution in user-generated content communities ^[7]
• Technologies
  • The use of lists by Wikipedians to identify and direct work ^[8]
  • Wikipedia's SuggestBot helps editors find work to do that they're likely to find interesting ^[9]
  • Calling focus to areas in need of work was effective in increasing contribition to a geo-wiki ^[10]
  • (TODO) ^[11]

Regulation (of behavior)

• Development of norms -- regulate people's activities to work together
  • The wiki structure lends itself to the development and enforcement of norms ^[12]
  • On Wikipedia, norms are developed and first used by admins, then power users, and later, newcomers. ^[13]
  • While the development of formalized norms in Wikipedia was open at first, the process has become increasingly calcified. ^[14]

Quality management (of product)

• Dealing with spam/vandalism/damage
  • Wikipedia's distributed cognition system supports a stygmergic strategy of coordination around undesirable editor detection ^[15]
  • Wikipedia's quality control systems disproportionately affect newcomers^[14]
• Technologies
  • ^[16] design an automated system that, based on implicit review of content, assigned a "trustworthiness" score to Wikipedia's content.
  • ^[17] developed an semi-automated system for identifying and removing spammy contributions to Wikipedia. See also (en:WP:Huggle and en:User:ClueBot NG)
  • When Wikipedia's primary counter-vandalism bot went down the time to revert damage doubled. Humans picked up the slack, but where much slower than the bot. ^[18]

Community management

• Onboarding/socialization process
  • Legitimate peripheral participation (LPP) provides a lens to view the introduction of new members to a community of practice ^[19]
  • Studies of Wikipedia suggest that new editors self-socialize in ways described by LPP ^[3]
  • Successful newcomers to open source software projects follow a "join script" where they participate peripherally (mailing list and bug reports) before trying more substantial activities (patches/pull requests/applying for committer) ^[20]
  • A general framework for patterns of moving from a lurker/consumer to producer/community organizer. ^[21]
• Newcomer retention
  • In 2007, Wikipedia began to experience a sudden decline in the number of active contributors -- and the cause is decreased newcomer retention ^[22]
  • The reason for Wikipedia's declining newcomer retention is tied strongly to the use of automated counter-vandalism tools.^[14]
  • The design of Wikipedia's counter-vandalism tools frame newcomers as suspicious and this leads to bad treatment^[23]
  • Differing socialization tactics lead to different outcome in Wikipedia ^[24]
  • Response to inquiries leads to retention and increased activity. ^[25]
  • Newcomers struggle with the massive amount of formalized and implicit norms in the mature English Wikiepdia^[14] so Wikipedians developed a safe space for them to ask questions and receive support^[26]
    • Other efforts have focused on automated instruction, see https://en.wikipedia.org/wiki/Wikipedia:The_Wikipedia_Adventure.
• Motivation of experienced contributors
  • Anthony, Smith, and Williamson, 2007. The Quality of Open Source Production: Zealots and Good Samaritans in the Case of Wikipedia. Rationality and Society.
  • Glott, Ruediger; Schmidt, Phillipp; Ghosh, Rishab. "Wikipedia Survey - Overview of Results". Wikipedia Study. UNU-MERIT.
  • Nov, Oded (2007). "What Motivates Wikipedians?". Communications of the ACM 50 (11): 60–64.
  • Yang, Heng-Li; Lai, Cheng-Yu (November 2010). "Motivations of Wikipedia content contributors". Computers in Human Behavior 26 (6): 1377–1383.
  • See also https://en.wikipedia.org/wiki/Wikipedia_community

• Need for a core community
  • Discussion of the roles adopted by Wikipedians and the rates at which such roles manifest ^[27]
  • ^[21] argues that a core group of "Leaders" is necessary, and due to churn, must be regularly infused with new leaders.
• Technologies
  • https://en.wikipedia.org/wiki/Wikipedia:The_Wikipedia_Adventure gives newcomes an introduction to norms and policies.
  • ^[26] makes use of a robot and implicit signals of desirability for newcomers to extend invitations to good newcomers.
  • ^[23] calls attention to the poor treatment of newcomers and provides a means for mentors to quickly identify and rescue the most desirable newcomers.

Reflection -- Identity, setting goals and observing progress.

• Social translucence helps reflection.
• Tracking and reacting to large-scale trends
• Able to set and track goals

• open data vs. user privacy
• establishing shared definitions for top-line metrics across systems
• sharing and documenting public datasets

• institutional review & community review
• knowledge transfer (design implications)
• open access
• formal and informal frameworks for supporting research of open collaboration communities
• supporting open collaboration communities through research partnerships

• required. organizers prepare curated datasets for the workshop. Host on datahub.io
• required. participants submit short position papers: summarize relevant research they've done and/or pose questions, challenges, issues for group discussion
• required. organizers prepare lightning talks on their communities, activities, challenges, & research needs
• optional. participants share curated datasets on datahub.io
• optional. selected participants invited to prepare lightning talks about their position papers
• optional. leaders of related initiatives (e.g. Digital Ecosystems Research Partnership, Open Collaboration Data Factory) invited to give lightning talks about their projects

• produce awesome stuff on top of this data
• ..?
• profit :)

• relationship to DERP? Join but maintain more focused separate initiative?

co-organized with DERP founders, focus is complementary: this event will help build out the DERP network, and help develop reviewing standards, common metrics, and identify Big Questions Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• should we ask for position papers or other prep work to seed the conversation? Who from?
  • If the goal is to come up with open questions across OCS's, yes.

all participants will submit position papers, organizers will come prepared to present/articulate their own positions as well Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• should we ask industry reps to give short presentations of their research needs?
  • This sounds interesting. I guess by "research needs" you mean "important ongoing open problems for which research might yield actionable insights"?

yes, that's a good articulation of it :) all organizers will give lighting talks, and industry participants with compelling position papers will also be invited to participate Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• ask industry researchers to make their data resources available beforehand? (MVP: a summary page like Research:Data)
  • Do you want data to be available there? If so, why? Is that data to answer particular questions, or something more?

WMF, Reddit, Imgur (and possibly other organizers) will post some sample datasets on DataHub.io. These datasets can help anchor group discussion about shared top-line metrics, common challenges & important questions for OCs. They also provide a novel value proposition for attendees: "here is some cool Reddit data to play with, and here are some things you could do with it" Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• " two communities of practice " feels awkward. I don't know what you'd replace it with, though.

has been replaced with less jargon. Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• What does it mean to "support the work of these online communities through research"? Be more specific, if possible.

not sure how specific we can get in the space of the abstract. Mostly talking about the work that folks in Data Science/Community Manager roles do to understand community health, evaluate feature usage, etc. Agree that the proposal itself will need to expand on this. Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• Similarly "a framework that...supports the work of the communities being studied" doesn't make sense to me. If the goal is to support communities, shouldn't community managers (rather than just researchers at particular PP-communities) be the target? I've added them (remove if that's not your intention!)

Community managers are definitely a target audience. Even those who don't do data science themselves will have important insights to share. Jmorgan (WMF) (talk) 00:26, 2 July 2014 (UTC)

• "Datasets and APIs will be prepared in advance and shared between participants" <-- that's not a goal, that's a procedure

good point. changed it to "Pilot a data repository by publishing a small set of well-documented and compelling OC datasets". Jmorgan (WMF) (talk) 00:41, 2 July 2014 (UTC)

• I'm not sure what "Nature" means to you -- maybe something like "Real-world problems/challenges"? A different term would be better.

I agree... any ideas of how we would re-organize our themes? Jmorgan (WMF) (talk) 00:42, 2 July 2014 (UTC)

• Can you give an example of how technical innovation might be needed to contribute to Grand Challenges? I guess this is something about social and data translucence (e.g. usage statistics from an article being shown, which then informs priorities)

struggled with how to articulate this, but then ended up dropping "Grand Challenge" language altogether. We probably won't push the technical innovation angle at all. Jmorgan (WMF) (talk) 00:41, 2 July 2014 (UTC)

• For a one-day workshop, on top of a framework, this sounds like a lot: "Participants will also partner to share methods, tools and data, perform analyses, and develop best practices for open collaboration research." Are you foreseeing ongoing collaborations, and how will they be fostered?

Truth be told, we probably won't perform much analysis during the workshop: it won't be a true hackathon. Sharing methods, tools and data will happen through lightning talks and during afternoon breakout groups. Developing best practices (and identifying key research questions/needs) will happen through a roundtable discussion after the lightning talks, and probably also through breakout groups. Subsequent collaboration will be facilitated through partner initiatives (Digital Ecosystem Research Partnership and Open Collaboration Data Factory) and through personal relationships formed during the workshop. Also, we will create a mailing list :) Jmorgan (WMF) (talk) 00:41, 2 July 2014 (UTC)

• Aaron Halfaker (WMF)
• Jonathan Morgan (WMF)
• Dario Taraborelli (WMF)
• Tim Hwang (Imgur)
• Max Goodman (Reddit)
• ...

• Stuart Lynn (Zooniverse)

as co-organizers or participants

OpenIDEO

• JM: Nathon Maton (product manager)

Zooniverse

• JM: Chris Lintott (founder)

StackExchange

• DT: Jeff Atwood (co-founder)

Wikia

• AH: Kevin, Bhupendra & Mike from Analytics

GitHub

• DT: Arfon Smith (former Zooniverse CTO, now head of GitHub science)

Mozilla

• ??

OpenHatch

• TH: has some contacts

OpenStreetMap

• ??

Reddit

• Max Goodman Done

Fark

• TH: via derp

Twitch

• TH: via derp

probably as participants

• Sean Goggins
• Mako Hill
• Kevin Crowston
• Andrea Forte
• Elizabeth Gerber
• Brian Keegan
• Stuart Geiger
• Jodi Schneider
• Giovanni Ciampaglia
• Aaron Shaw
• Mark Zachry
• David McDonald
• Alex Leavitt
• Edith Law
• Bluma Gelley
• Gabe Mugar

this is just fodder for proposal-creation

• HCOMP 2014 Workshop on Crowdsourcing in Science, Public Health and Government (Organizers: Edith Law and Cliff Lampe)
• Online Community Data Hackathon (note: we may want to contact the organizers of this workshop, to see if they're planning it for this year. There's some overlap here… probably in a productive, creative way)
• Social media question-asking
• CSCW and education
• Data-intensive collaboration in Science and Engineering
• The Future of Collaborative Software Development
• Nth Annual Workshop on Collaborative Editing Systems (see also [1], [2], [3])
• The Changing Dynamics of Scientific Collaboration
• Cross-cultural studies of collaborative systems