short-paper

Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks

Authors:

Ujwal Gadiraju,

Patrick Siehndel,

Besnik Fetahu, and

Ricardo KawaseAuthors Info & Claims

HT '15: Proceedings of the 26th ACM Conference on Hypertext & Social Media

August 2015

Pages 33 - 38

https://doi.org/10.1145/2700171.2791053

Published: 24 August 2015 Publication History

Get Access

Abstract

Crowdsourcing systems are being widely used to overcome several challenges that require human intervention. While there is an increase in the adoption of the crowdsourcing paradigm as a solution, there are no established guidelines or tangible recommendations for task design with respect to key parameters such as task length, monetary incentive and time required for task completion. In this paper, we propose the tuning of these parameters based on our findings from extensive experiments and analysis of categorization tasks. We delve into the behavior of workers that consume categorization tasks to determine measures that can make task design more effective.

References

[1]

C. Eickhoff and A. de Vries. How crowdsourcable is your task. In Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM), pages 11--14, 2011.

Google Scholar

[2]

C. Eickhoff and A. P. de Vries. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, 16(2):121--137, 2013.

Digital Library

Google Scholar

[3]

U. Gadiraju, R. Kawase, and S. Dietze. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media, pages 218--223. ACM, 2014.

Digital Library

Google Scholar

[4]

U. Gadiraju, R. Kawase, S. Dietze, and G. Demartini. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of CHI'15, CHI Conference on Human Factors in Computing Systems, 2015.

Digital Library

Google Scholar

[5]

P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64--67. ACM, 2010.

Digital Library

Google Scholar

[6]

N. Kaufmann, T. Schulze, and D. Veit. More than fun and money. worker motivation in crowdsourcing - a study on mechanical turk. In AMCIS, 2011.

Google Scholar

[7]

G. Kazai, J. Kamps, and N. Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1941--1944. ACM, 2011.

Digital Library

Google Scholar

[8]

C. C. Marshall and F. M. Shipman. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In Proceedings of the 5th Annual ACM Web Science Conference, WebSci '13, pages 234--243, New York, NY, USA, 2013. ACM.

Digital Library

Google Scholar

[9]

W. Mason and D. J. Watts. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100--108, 2010.

Digital Library

Google Scholar

[10]

D. Oleson, A. Sorokin, G. P. Laughlin, V. Hester, J. Le, and L. Biewald. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human computation, 11:11, 2011.

Digital Library

Google Scholar

[11]

J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM, 2011.

Google Scholar

[12]

J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 2863--2872. ACM, 2010.

Digital Library

Google Scholar

Cited By

View all

Kölle MWalter VSchmohl SSoergel U(2021)Remembering Both the Machine and the Crowd When Sampling Points: Active Learning for Semantic Segmentation of ALS Point CloudsPattern Recognition. ICPR International Workshops and Challenges10.1007/978-3-030-68787-8_37(505-520)Online publication date: 21-Feb-2021
https://doi.org/10.1007/978-3-030-68787-8_37
Qiu SGadiraju UBozzon A(2020)Just the Right Mood for HIT!Web Engineering10.1007/978-3-030-50578-3_26(381-396)Online publication date: 10-Jun-2020
https://doi.org/10.1007/978-3-030-50578-3_26
Göritz ABorchert KHirth M(2019)Using Attention Testing to Select Crowdsourced Workers and Research ParticipantsSocial Science Computer Review10.1177/0894439319848726(089443931984872)Online publication date: 11-Jun-2019
https://doi.org/10.1177/0894439319848726
Show More Cited By

Index Terms

Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks
1. Information systems

Recommendations

Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Crowdsourcing is increasingly being used as a means to tackle problems requiring human intelligence. With the ever-growing worker base that aims to complete microtasks on crowdsourcing platforms in exchange for financial gains, there is a need for ...
Read More
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Web

Within the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Read More
Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
Abstract
The suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a ...
Read More

Comments

Information & Contributors

Information

Published In

HT '15: Proceedings of the 26th ACM Conference on Hypertext & Social Media

August 2015

360 pages

ISBN:9781450333955

DOI:10.1145/2700171

General Chair:
Yeliz Yesilada
Middle East Technical University Northern Cyprus Campus
,
Program Chairs:
Rosta Farzan
University of Pittsburgh, USA
,
Geert-Jan Houben
Delft University of Technology, The Netherlands

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

European Commission

Conference

HT '15

Sponsor:

SIGWEB

HT '15: 26th ACM Conference on Hypertext and Social Media

September 1 - 4, 2015

Guzelyurt, Northern Cyprus

Acceptance Rates

HT '15 Paper Acceptance Rate 24 of 60 submissions, 40%;

Overall Acceptance Rate 378 of 1,158 submissions, 33%

Upcoming Conference

HT '24

Sponsor:
sigweb

35th ACM Conference on Hypertext and Social Media

September 10 - 13, 2024

Poznan , Poland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
190
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kölle MWalter VSchmohl SSoergel U(2021)Remembering Both the Machine and the Crowd When Sampling Points: Active Learning for Semantic Segmentation of ALS Point CloudsPattern Recognition. ICPR International Workshops and Challenges10.1007/978-3-030-68787-8_37(505-520)Online publication date: 21-Feb-2021
https://doi.org/10.1007/978-3-030-68787-8_37
Qiu SGadiraju UBozzon A(2020)Just the Right Mood for HIT!Web Engineering10.1007/978-3-030-50578-3_26(381-396)Online publication date: 10-Jun-2020
https://doi.org/10.1007/978-3-030-50578-3_26
Göritz ABorchert KHirth M(2019)Using Attention Testing to Select Crowdsourced Workers and Research ParticipantsSocial Science Computer Review10.1177/0894439319848726(089443931984872)Online publication date: 11-Jun-2019
https://doi.org/10.1177/0894439319848726
Barbosa NChen MBrewster SFitzpatrick GCox AKostakos V(2019)Rehumanized CrowdsourcingProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300773(1-12)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300773
Gadiraju UMöller SNöllenburg MSaupe DEgger-Lampl SArchambault DFisher B(2017)Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the CrowdEvaluation in the Crowd. Crowdsourcing and Human-Centered Experiments10.1007/978-3-319-66435-4_2(6-26)Online publication date: 28-Sep-2017
https://doi.org/10.1007/978-3-319-66435-4_2

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys

Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web

Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys

Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web

Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations