research-article

Public Access

Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability

Authors:

Niloufar Salehi,

Melissa Valentine,

Michael BernsteinAuthors Info & Claims

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

Pages 1700 - 1713

https://doi.org/10.1145/2998181.2998300

Published: 25 February 2017 Publication History

Abstract

Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams -- familiarity with other members -- stands in contrast to crowd work's flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work.

References

[1]

Shoshana Altschuller, and Raquel Benbunan-Fich. "Trust, Performance, and the Communication Process in Ad Hoc Decision-Making Virtual Teams." Journal of Computer-Mediated Communication 16, no. 1 (2010): 27--47.

[2]

Linda Argote, Paul Ingram, John M. Levine, and Richard L. Moreland. "Knowledge transfer in organizations: Learning from the experience of others." Organizational behavior and human decision processes 82, no. 1 (2000): 1--8.

[3]

Linda Argote. "Input uncertainty and organizational coordination in hospital emergency units." Administrative science quarterly (1982): 420--434.

[4]

Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. "Soylent: a word processor with a crowd inside." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 313--322. ACM, 2010.

Digital Library

[5]

Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller et al. "VizWiz: nearly real-time answers to visual questions." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 333--342. ACM, 2010.

Digital Library

[6]

Jeffrey P. Bigham, Michael S. Bernstein, and Eytan Adar. "Human-Computer Interaction and Collective Intelligence." Handbook of Collective Intelligence (2015): 57.

[7]

Kathleen Carley. "Organizational learning and personnel turnover."Organization Science 3.1 (1992): 20--46.

Digital Library

[8]

Tom Finholt, Lee Sproull, and Sara Kiesler. "Communication and performance in ad hoc task groups." Intellectual teamwork: Social and technological foundations of cooperative work (1990): 291--325.

Digital Library

[9]

Susan G. Cohen, and Diane E. Bailey. "What makes teams work: Group effectiveness research from the shop floor to the executive suite." Journal of management 23, no. 3 (1997): 239--290.

[10]

Lincoln Dahlberg. "The Internet and democratic discourse: Exploring the prospects of online deliberative forums extending the public sphere." Information, Communication & Society 4, no. 4 (2001): 615--633.

[11]

Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. "Parallel prototyping leads to better design results, more divergence, and increased self-efficacy." ACM Transactions on Computer-Human Interaction (TOCHI) 17, no. 4 (2010): 18.

Digital Library

[12]

Scott B. Droege, and Jenny M. Hoobler. "Employee turnover and tacit knowledge diffusion: A network perspective." Journal of Managerial Issues(2003): 5064.

[13]

Amy C. Edmondson Teaming: How organizations learn, innovate, and compete in the knowledge economy. John Wiley & Sons, 2012.

[14]

Amy Edmondson. "Psychological safety and learning behavior in work teams." Administrative science quarterly 44, no. 2 (1999): 350--383.

[15]

J. Alberto Espinosa, Sandra A. Slaughter, Robert E. Kraut, and James D. Herbsleb. "Team knowledge and coordination in geographically distributed software development." Journal of Management Information Systems 24, no. 1 (2007): 135--169.

Digital Library

[16]

John E. Ettlie "The impact of interorganizational manpower flows on the innovation process." Management Science 31, no. 9 (1985): 10551071.

Digital Library

[17]

Mary L. Gray, Siddharth Suri, Syed S. Ali, and Deepti Kulkarni. "The Crowd is a Collaborative Network." Proceedings of Computer-Supported Cooperative Work (2016).

Digital Library

[18]

Deborah H. Gruenfeld, Elizabeth A. Mannix, Katherine Y. Williams, and Margaret A. Neale. "Group composition and decision making: How member familiarity and information distribution affect process and performance." Organizational behavior and human decision processes 67, no. 1 (1996): 1--15.

[19]

Rebecca M. Guidice, Joyce Thompson Heames, and Sheng Wang. "The indirect relationship between organizational-level knowledge worker turnover and innovation: An integrated application of related literature." The Learning Organization 16, no. 2 (2009): 143--167.

[20]

Daniel Haas, Jason Ansel, Lydia Gu, and Adam Marcus. "Argonaut: macrotask crowdsourcing for complex data processing." Proceedings of the VLDB Endowment 8, no. 12 (2015): 1642--1653.

Digital Library

[21]

J. Richard. Hackman, Leading teams: Setting the stage for great performances. Harvard Business Press, 2002.

[22]

D.A. Harrison, Mohammed, S., McGrath, J.E., Florey, A.T. and Vanderstoep, S.W., 2003. Time matters in team performance: Effects of member familiarity, entrainment, and task discontinuity on speed and quality. Personnel Psychology, 56(3), pp.633--669.

[23]

Mark Hefke, and Ljiljana Stojanovic. "An ontologybased approach for competence bundling and composition of ad-hoc teams in an organization." Proc. I-KNOW'04 (2004): 126--134.

[24]

Pamela J. Hinds, and Mark Mortensen. "Understanding conflict in geographically distributed teams: The moderating effects of shared identity, shared context, and spontaneous communication." Organization science 16, no. 3 (2005): 290--307.

Digital Library

[25]

Robert S. Huckman, and Bradley R. Staats. "Fluid tasks and fluid teams: The impact of diversity in experience and team familiarity on team performance." Manufacturing & Service Operations Management 13, no. 3 (2011): 310--328.

Digital Library

[26]

Robert S. Huckman, Bradley R. Staats, and David M. Upton. "Team familiarity, role experience, and performance: Evidence from Indian software services." Management science 55, no. 1 (2009): 85--100.

Digital Library

[27]

Aniket Kittur, Bongwon Suh, Bryan A. Pendleton, and Ed H. Chi. "He says, she says: conflict and coordination in Wikipedia." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 453--462. ACM, 2007.

Digital Library

[28]

Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. "Crowdforge: Crowdsourcing complex work." In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 43--52. ACM, 2011.

Digital Library

[29]

Aniket Kittur. "Crowdsourcing, collaboration and creativity." ACM Crossroads 17, no. 2 (2010): 22--26.

Digital Library

[30]

David A. Kravitz, and Barbara Martin. "Ringelmann rediscovered: The original article." (1986): 936.

[31]

Karim Lakhani, David A. Garvin, and Eric Lonstein. "Topcoder (a): Developing software through crowdsourcing." Harvard Business School General Management Unit Case 610-032 (2010).

[32]

Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. "Apparition: Crowdsourced user interfaces that come To life as you sketch them." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1925--1934. ACM, 2015.

Digital Library

[33]

Diane Wei Liang, Richard Moreland, and Linda Argote. "Group versus individual training and group performance: The mediating role of transactive memory." Personality and Social Psychology Bulletin 21, no. 4 (1995): 384--393.

[34]

Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. "Turkit: tools for iterative tasks on mechanical turk." In Proceedings of the ACM SIGKDD workshop on human computation, pp. 29--30. ACM, 2009.

Digital Library

[35]

Ioanna Lykourentzou, Shannon Wang, Robert E. Kraut, and Steven P. Dow. "Team Dating: A SelfOrganized Team Formation Strategy for Collaborative Crowdsourcing." In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1243--1249. ACM, 2016.

Digital Library

[36]

Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and Steven P. Dow. "Personality Matters: Balancing for Personality Types Leads to Better Outcomes for Crowd Teams." (2016).

[37]

Andrew Mao, Winter Mason, Siddharth Suri, and Duncan J. Watts. "An experimental study of team size and performance on a complex task." PloS one 11, no. 4 (2016): e0153048.

[38]

David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. "Being a turker." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 224--235. ACM, 2014.

Digital Library

[39]

Winter Mason, and Siddharth Suri. "Conducting behavioral research on Amazon's Mechanical Turk." Behavior research methods 44.1 (2012): 1--23.

[40]

Joseph E. McGrath, Holly Arrow, and Jennifer L. Berdahl. "The study of groups: past, present, and future." Personality and Social Psychology Review 4, no. 1 (2000): 95--105.

[41]

Mitzi M. Montoya-Weiss, Anne P. Massey, and Michael Song. "Getting it together: Temporal coordination and conflict management in global virtual teams." Academy of management Journal 44, no. 6 (2001): 1251--1262.

[42]

M. Mortensen 2014 "Constructing the Team: The Antecedents and Effects of Membership Model Divergence." Organization Science, 25: 909--931.

Digital Library

[43]

Ingrid Mulder, Janine Swaak, and Joseph Kessels. "In search of reflective behavior and shared understanding in ad hoc expert teams." CyberPsychology & Behavior 7, no. 2 (2004): 141--154.

[44]

Gary M. Olson, and Judith S. Olson. "Distance matters." Human-computer interaction 15, no. 2 (2000): 139--178.

Digital Library

[45]

Ray Reagans, Linda Argote, and Daria Brooks. "Individual experience and experience working together: Predicting learning rates from knowing who knows what and knowing how to work together." Management science 51.6 (2005): 869--881.

Digital Library

[46]

Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. "Expert crowdsourcing with flash teams." In Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 75--85. ACM, 2014.

Digital Library

[47]

Floor Rink, Aimée A. Kane, Naomi Ellemers, and Gerben Van der Vegt. "Team receptivity to newcomers: Five decades of evidence and future research themes." The Academy of Management Annals 7, no. 1 (2013): 247--293.

[48]

Elena Rocco, Thomas Finholt, Erik C. Hofer, and James Herbsleb. "Designing as if trust mattered." Collaboratory for Research on Electronic Work (CREW) Technical Report (2000).

[49]

Kate Starbird and Leysia Palen. "Voluntweeters: Selforganizing by digital volunteers in times of crisis." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1071--1080. ACM, 2011.

Digital Library

[50]

Andrew H. Van de Ven, Andre L. Delbecq, and Richard Koenig Jr. "Determinants of coordination modes within organizations." American sociological review (1976): 322--338.

[51]

Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. "Evidence for a collective intelligence factor in the performance of human groups." science 330, no. 6004 (2010): 686--688.

[52]

Galen Pickard, Wei Pan, Iyad Rahwan, Manuel Cebrian, Riley Crane, Anmol Madan, and Alex Pentland. "Time-critical social mobilization." Science 334, no. 6055 (2011): 509--512.

[53]

Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, and Kristy Milland. "We are dynamo: Overcoming stalling and friction in collective action for crowd workers." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1621--1630. ACM, 2015.

Digital Library

[54]

Haiyi Zhu, Steven P. Dow, Robert E. Kraut, and Aniket Kittur. "Reviewing versus doing: Learning and performance in crowd assessment." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 1445--1455. ACM, 2014.

Digital Library

[55]

http://www.pewinternet.org/2016/07/11/research-inthe-crowdsourcing-age-a-casestudy/http://www.pewinternet.org/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/

Cited By

Asencio RMesmer-Magnus JDeChurch LContractor N(2024)Boundary Transitions in Dynamic TeamworkAcademy of Management Review10.5465/amr.2019.047149:3(597-617)Online publication date: Jul-2024
https://doi.org/10.5465/amr.2019.0471
Smirnov APonomarev AShilov NLevashova TTeslya N(2024)A Conception of Collaborative Decision Support Systems: Approach and Platform ArchitectureКонцепция построения коллаборативных систем поддержки принятия решений: подход и архитектура платформыInformatics and AutomationИнформатика и автоматизация10.15622/ia.23.4.823:4(1139-1172)Online publication date: 26-Jun-2024
https://doi.org/10.15622/ia.23.4.8
Palea DGuo ANair AAnderson RCharagulla NSu NLee D(2024)Forming Shared Interest Pods: Barriers to Self-Assembly of Interest-Based Small Groups, and Dynamics of Retaining and Giving up Control to Find Collective FitProceedings of the ACM on Human-Computer Interaction10.1145/36869848:CSCW2(1-50)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686984
Show More Cited By

Index Terms

Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

The future of crowd work
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work

Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd ...
Human Collaboration on Crowdsourcing Platforms – a Content Analysis
HCI in Business, Government and Organizations. Information Systems and Analytics
Abstract
The crowdsourcing phenomenon offers the opportunity to address an open call to the crowd. Crowd workers may work together to find a solution that satisfies the open call. One of the major benefits for a crowdsourcer is the pool of crowd workers ...
Designing a Crowdsourcing System for the Elderly: A Gamified Approach to Speech Collection
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Despite the need for representation of older adults in crowdsourced data, crowd work is generally not designed for older adults and participation by older adults is low. In this paper, we demonstrate a process for designing crowd work for older adults; ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

February 2017

2556 pages

ISBN:9781450343350

DOI:10.1145/2998181

General Chairs:
Charlotte P. Lee
University of Washington
,
Steve Poltrock
Retired
,
Program Chairs:
Louise Barkhuus
Stockholm University and Cornell Tech
,
Marcos Borges
Universidade Federal do Rio de Janeiro
,
Wendy Kellogg
IBM T.J. Watson Research Center

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CSCW '17

Sponsor:

SIGCHI

CSCW '17: Computer Supported Cooperative Work and Social Computing

February 25 - March 1, 2017

Oregon, Portland, USA

Acceptance Rates

CSCW '17 Paper Acceptance Rate 183 of 530 submissions, 35%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
1,232
Total Downloads

Downloads (Last 12 months)168
Downloads (Last 6 weeks)16

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Asencio RMesmer-Magnus JDeChurch LContractor N(2024)Boundary Transitions in Dynamic TeamworkAcademy of Management Review10.5465/amr.2019.047149:3(597-617)Online publication date: Jul-2024
https://doi.org/10.5465/amr.2019.0471
Smirnov APonomarev AShilov NLevashova TTeslya N(2024)A Conception of Collaborative Decision Support Systems: Approach and Platform ArchitectureКонцепция построения коллаборативных систем поддержки принятия решений: подход и архитектура платформыInformatics and AutomationИнформатика и автоматизация10.15622/ia.23.4.823:4(1139-1172)Online publication date: 26-Jun-2024
https://doi.org/10.15622/ia.23.4.8
Palea DGuo ANair AAnderson RCharagulla NSu NLee D(2024)Forming Shared Interest Pods: Barriers to Self-Assembly of Interest-Based Small Groups, and Dynamics of Retaining and Giving up Control to Find Collective FitProceedings of the ACM on Human-Computer Interaction10.1145/36869848:CSCW2(1-50)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686984
Smirnov APonomarev ALevashova TTeslya NShilov N(2024)Platform Architecture for Human-AI Collaborative Decision SupportProceedings of the Eighth International Scientific Conference “Intelligent Information Technologies for Industry” (IITI’24), Volume 110.1007/978-3-031-77688-5_32(334-345)Online publication date: 20-Dec-2024
https://doi.org/10.1007/978-3-031-77688-5_32
Wu DHuang J(2024)Gig work and gig workers: An integrative review and agenda for future researchJournal of Organizational Behavior10.1002/job.277545:2(183-208)Online publication date: 24-Jan-2024
https://doi.org/10.1002/job.2775
Musick GGilman EDuan WMcNeese NKnijnenburg BO'Neill T(2023)Knowing Unknown Teammates: Exploring Anonymity and Explanations in a Teammate Information-Sharing Recommender SystemProceedings of the ACM on Human-Computer Interaction10.1145/36100757:CSCW2(1-34)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610075
Shi WKrishna Kumaran SSundaram HBailey B(2023)The Value of Activity Traces in Peer Evaluations: An Experimental StudyProceedings of the ACM on Human-Computer Interaction10.1145/35796277:CSCW1(1-39)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579627
Chiang CLu ZLi ZYin M(2023)Are Two Heads Better Than One in AI-Assisted Decision Making? Comparing the Behavior and Performance of Groups and Individuals in Human-AI Collaborative Recidivism Risk AssessmentProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581015(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581015
Garcia Calvo AKenney MZysman J(2023)Understanding work in the online platform economy: the narrow, the broad, and the systemic perspectivesIndustrial and Corporate Change10.1093/icc/dtad00532:4(795-814)Online publication date: 1-Feb-2023
https://doi.org/10.1093/icc/dtad005
Musick GHauptman AFlathmann CMcNeese NKnijnenburg B(2023)Recommendations with Benefits: Exploring Explanations in Information Sharing Recommender Systems for Temporary TeamsInternational Journal of Human–Computer Interaction10.1080/10447318.2023.227893340:23(8202-8218)Online publication date: 20-Nov-2023
https://doi.org/10.1080/10447318.2023.2278933
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents