research-article

On Moderating Software Crowdsourcing Challenges

Authors:

Cleidson R. B. de Souza,

Leticia S. Machado,

Ricardo Rodrigo M. MeloAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 4, Issue GROUP

Article No.: 14, Pages 1 - 22

https://doi.org/10.1145/3375194

Published: 04 January 2020 Publication History

Abstract

Crowdsourcing divides a task into small pieces that are carried out by the crowd. In Software Engineering, crowdsourcing divides the software development tasks of to be carried out online by the crow and is simply called Software Crowdsourcing (SW CS). Most SW CS platforms support this emerging software development strategy and operate within a framework of competition among the crowd. Competitive SW CS platforms intentionally minimize communication and collaboration among the parties involved (customer, platform, and crowd) while they compete in the software development tasks. The goal of this paper is to investigate platform moderators in SW CS challenges. Platform moderators are individuals who work for the SW CS platforms to mediate customer and crowd. A qualitative analysis of the content of the communication forums hosted on the TopCoder platform was performed to analyze the messages exchanged by the platform moderators and the crowd. Our empirical results indicate that co-pilots enforce and, at the same time, extend the limitations of the documentation associated with the tasks to support crowd members, provide technical help to crowd members during the competitions, and engage the crowd in the challenges. Co-pilots are organized, work diligently, worrying about being fair, and, at the same time, seeking to find a balance between autonomy and dependency on the customer. We conclude by providing insights to improve the design of software crowdsourcing platforms.

References

[1]

Nikolay Archak. 2010. Money, Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com. 21--30. https://doi.org/10.1145/1772690.1772694

[2]

Joost Bianchi, Yuri Knopper, Ozgur Eris, Petra Badke-Schaub, and Lampros Roussos. 2015. Online Ways of Sharedness: A Syntactic Analysis of Design Collaboration in OpenIDEO.

[3]

Kevin Boudreau, Patrick Gaulé, Karim Lakhani, Christoph Riedl, and AnitaWoolley. 2014. From Crowds to Collaborators: Initiating Effort Catalyzing Interactions Among Online Creative Workers. SSRN Electronic Journal (01 2014). https://doi.org/10.2139/ssrn.2384068

[4]

Brabham and Daren C. 2008. Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. The International Journal of Research into New Media Technologies 14 (1) (01 2008), 75--90.

[5]

Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology 3 (01 2006), 77--101. https://doi.org/10.1191/1478088706qp063oa

[6]

Anhai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. 2011. Crowdsourcing Systems on the World-Wide Web. Commun. ACM 54, 4 (April 2011), 86--96. https://doi.org/10.1145/1924421.1924442

Digital Library

[7]

Anhai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. 2011. Crowdsourcing Systems on the World-Wide Web. Commun. ACM 54, 4 (April 2011), 86--96. https://doi.org/10.1145/1924421.1924442

Digital Library

[8]

Alpana Dubey, Kumar Abhinav, Sakshi Taneja, Gurdeep Virdi, Anurag Dwarakanath, Alex Kass, and Suma Mani. 2016. Dynamics of Software Development Crowdsourcing. https://doi.org/10.1109/ICGSE.2016.13

[9]

Mark Fuge, Kevin Tee, Alice Agogino, and Nathan Maton. 2014. Analysis of Collaborative Design Networks: A Case Study of OpenIDEO. Journal of Computing and Information Science in Engineering 14 (06 2014), 021009. https://doi.org/10.1115/1.4026510

[10]

Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The Crowd is a Collaborative Network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 134--147. https://doi.org/10.1145/2818048.2819942

Digital Library

[11]

Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.). Crown Publishing Group, New York, NY, USA.

Digital Library

[12]

Katja Hutter, Julia Hautz, Johann Füller, Julia Mueller-Seeger, and Kurt Matzler. 2011. Communitition: The Tension Between Competition and Collaboration in Community Based Design Contests. Creativity and Innovation Management 20 (02 2011), 3--21. https://doi.org/10.1111/j.1467--8691.2011.00589.x

[13]

Joshua Introne, Robert Laubacher, Gary Olson, and Thomas Malone. 2011. The Climate CoLab: Large scale model-based collaborative planning. Proceedings of the 2011 International Conference on Collaboration Technologies and Systems, CTS 2011 (03 2011). https://doi.org/10.1109/CTS.2011.5928663

[14]

Evgeny Kaganer, Erran Carmel, Rudy Hirschheim, and Olsen Timothy. 2013. Managing the Human Cloud. MIT Sloan Management Review 54 (12 2013), 23.

[15]

Aniket Kittur and Robert E. Kraut. 2008. Harnessing the Wisdom of Crowds in Wikipedia: Quality Through Coordination. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08). ACM, New York, NY, USA, 37--46. https://doi.org/10.1145/1460563.1460572

Digital Library

[16]

Aniket Kittur and Robert E. Kraut. 2010. Beyond Wikipedia: Coordination and Conflict in Online Production Groups. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 215--224. https://doi.org/10.1145/1718918.1718959

[17]

Aniket Kittur, Jeffrey Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 1301--1318. https://doi.org/10.1145/2441776.2441923

Digital Library

[18]

Travis Kriplean, Ivan Beschastnikh, and David W. McDonald. 2008. Articulations of Wikiwork: Uncovering Valued Work in Wikipedia Through Barnstars. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08). ACM, New York, NY, USA, 47--56. https://doi.org/10.1145/1460563.1460573

[19]

Karim Lakhani, David Garvin, and Eric Lonstein. 2010. TopCoder (A): Developing Software through Crowdsourcing. (01 2010).

[20]

Thomas LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and Andre van der Hoek. 2015. Borrowing from the Crowd: A Study of Recombination in Software Design Competitions. https://doi.org/10.1109/ICSE.2015.72

[21]

Leticia Machado, Alexandre Zanatta, Sabrina Marczak, and Rafael Prikladnicki. 2017. The Good, the Bad and the Ugly: An Onboard Journey in Software Crowdsourcing Competitive Model. https://doi.org/10.1109/CSI-SE.2017.6

[22]

Thomas Malone, Robert Laubacher, and Chrysanthos Dellarocas. 2010. The Collective Intelligence Genome. IEEE Engineering Management Review 38 (03 2010), 38--52. https://doi.org/10.1109/EMR.2010.5559142

[23]

Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2016. A Survey of the Use of Crowdsourcing in Software Engineering. Journal of Systems and Software 126 (09 2016). https://doi.org/10.1016/j.jss.2016.09.015

[24]

Tara Matthews, Steve Whittaker, Hernan Badenes, Barton A. Smith, Michael Muller, Kate Ehrlich, Michelle X. Zhou, and Tessa Lau. 2013. Community Insights: Helping Community Leaders Enhance the Value of Enterprise Online Communities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 513--522. https://doi.org/10.1145/2470654.2470728

Digital Library

[25]

Ricardo Melo, Leticia Machado, Rafael Prikladnicki, and Cleidson de Souza. 2018. Um Estudo Qualitativo sobre o Crowdsourcing: Análise da Colaboração na plataforma TopCoder. http://toc.proceedings.com/40795webtoc.pdf

[26]

Sreeja Nag, Ira Heffan, Alvar Saenz-Otero, and Mike Lydon. 2012. SPHERES Zero Robotics Software Development: Lessons on Crowdsourcing and Collaborative Competition. https://doi.org/10.1109/AERO.2012.6187452

[27]

Wanda J. Orlikowski. 1992. Learning from Notes: Organizational Issues in Groupware Implementation. In Proceedings of the 1992 ACM Conference on Computer-supported Cooperative Work (CSCW '92). ACM, New York, NY, USA, 362--369. https://doi.org/10.1145/143457.143549

Digital Library

[28]

Razieh Saremi, Ye Yang, Guenther Ruhe, and David Messinger. 2017. Leveraging Crowdsourcing For Team Elasticity: An Empirical Evaluation at TopCoder. ICSE 2017 SEIP (05 2017).

Digital Library

[29]

Shilad Sen, Shyong K. Lam, Al Mamunur Rashid, Dan Cosley, Dan Frankowski, Jeremy Osterhouse, F. Maxwell Harper, and John Riedl. 2006. Tagging, Communities, Vocabulary, Evolution. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (CSCW '06). ACM, New York, NY, USA, 181--190. https://doi.org/10.1145/1180875.1180904

Digital Library

[30]

Klaas-Jan Stol, Bora Caglayan, and Brian Fitzgerald. 2017. Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective. IEEE Transactions on Software Engineering PP (11 2017). https://doi.org/10.1109/TSE.2017.2774297

[31]

Klaas-Jan Stol and Brian Fitzgerald. 2014. Two's Company, Three's a Crowd: A Case Study of Crowdsourcing Software Development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 187--198. https://doi.org/10.1145/2568225.2568249

Digital Library

[32]

A. Strauss and J. Corbin. 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications, Newbury Park, California.

[33]

Yla Tausczik and Ping Wang. 2017. To Share, or Not to Share?: Community-Level Collaboration in Open Innovation Contests. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 100 (Dec. 2017), 23 pages. https://doi.org/10.1145/3134735

Digital Library

[34]

TopCoder. 2008. TopCoder. Retrieved March 2, 2008 from https://www.topcoder.com

[35]

Luis Vaz, Sabrina Marczak, and Igor Steinmacher. 2018. An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform. 62--71. https://doi.org/10.1145/3266237.3266265

[36]

Martin Wattenberg, Fernanda Viégas, and Katherine Hollenbach. 2007. Visualizing Activity on Wikipedia with Chromograms, Vol. 4663. 272--287. https://doi.org/10.1007/978--3--540--74800--7_23

[37]

Ye Yang, Muhammad Rezaul Karim, Razieh Saremi, and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. 1--10. https://doi.org/10.1145/2961111.2962594

[38]

Ye Yang and Razieh Saremi. 2015. Award vs. Worker Behaviors in Competitive Crowdsourcing Tasks. Empirical Software Engineering and Measurement (ESEM), 2015 ACM/IEEE International Symposium on (10 2015).

[39]

Alexandre Zanatta, Igor Steinmacher, Leticia Machado, Cleidson Souza, and Rafael Prikladnicki. 2017. Barriers Faced by Newcomers to Software-Crowdsourcing Projects. IEEE Software 34 (03 2017), 37--43. https://doi.org/10.1109/MS.2017.32

[40]

Pär Ågerfalk, Brian Fitzgerald, and Klaas-Jan Stol. 2015. Software Sourcing in the Age of Open Leveraging the Unknown Workforce. https://doi.org/10.1007/978--3--319--17266--8

Cited By

Cheng KCuvin POlechowski AZhou S(2023)User Perspectives on Branching in Computer-Aided DesignProceedings of the ACM on Human-Computer Interaction10.1145/36102207:CSCW2(1-30)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610220
Shahin MHussain WNurwidyantoro APerera HShams RGrundy JWhittle J(2022)Operationalizing Human Values in Software Engineering: A SurveyIEEE Access10.1109/ACCESS.2022.319097510(75269-75295)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3190975
Machado LMelo Rde Souza CPrikladnicki R(2021)Collaborative Behavior and Winning Challenges in Competitive Software CrowdsourcingProceedings of the ACM on Human-Computer Interaction10.1145/34639325:GROUP(1-25)Online publication date: 13-Jul-2021
https://dl.acm.org/doi/10.1145/3463932
Show More Cited By

Index Terms

On Moderating Software Crowdsourcing Challenges

Recommendations

Collaborative Behavior and Winning Challenges in Competitive Software Crowdsourcing
GROUP

Software Crowdsourcing (SW CS) allows a requester to increase the speed of its software development efforts by submitting a task to be performed by the crowd. SW CS is usually structured around software platforms, which are used by crowd members to ...
Strategies for Crowdworkers to Overcome Barriers in Competition-based Software Crowdsourcing Development
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Crowdsourcing in software development uses a large pool of developers on-demand to outsource parts or the entire software project to a crowd. To succeed, this requires a continuous influx of developers, or simply crowdworkers. However, crowdworkers face ...
The role of platform moderators in software crowdsourcing projects
CHASE '19: Proceedings of the 12th International Workshop on Cooperative and Human Aspects of Software Engineering

Crowdsourcing divides a task into small pieces that are carried out by the crowd. In Software Engineering, crowdsourcing divides the software development tasks of to be carried out online by the crowd and is simply called Software Crowdsourcing (SW CS). ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 4, Issue GROUP

GROUP

January 2020

405 pages

EISSN:2573-0142

DOI:10.1145/3378060

Editors:
Cliff Lampe
University of Michigan
,
Jeff Nichols
Google
,
Karrie Karahalios
University of Illinois Urbana-Champaign
,
Geraldine Fitzpatrick
Vienna University of Technology
,
Uichin Lee
KAIST
,
Andres Monroy-Hernandez
Microsoft Research
,
Wolfgang Stuerzlinger
Simon Fraser University, Canada

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 January 2020

Published in PACMHCI Volume 4, Issue GROUP

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conselho Nacional de Desenvolvimento Científico e Tecnológico

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng KCuvin POlechowski AZhou S(2023)User Perspectives on Branching in Computer-Aided DesignProceedings of the ACM on Human-Computer Interaction10.1145/36102207:CSCW2(1-30)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610220
Shahin MHussain WNurwidyantoro APerera HShams RGrundy JWhittle J(2022)Operationalizing Human Values in Software Engineering: A SurveyIEEE Access10.1109/ACCESS.2022.319097510(75269-75295)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3190975
Machado LMelo Rde Souza CPrikladnicki R(2021)Collaborative Behavior and Winning Challenges in Competitive Software CrowdsourcingProceedings of the ACM on Human-Computer Interaction10.1145/34639325:GROUP(1-25)Online publication date: 13-Jul-2021
https://dl.acm.org/doi/10.1145/3463932
Jain SDeodhar S(2021)Social mechanisms in crowdsourcing contests: a literature reviewBehaviour & Information Technology10.1080/0144929X.2021.188063841:5(1080-1114)Online publication date: 28-Jan-2021
https://doi.org/10.1080/0144929X.2021.1880638
Cheng RZachry M(2020)Building Community Knowledge In Online CompetitionsProceedings of the ACM on Human-Computer Interaction10.1145/34152504:CSCW2(1-22)Online publication date: 15-Oct-2020
https://dl.acm.org/doi/10.1145/3415250

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents