research-article

Keep it simple: reward and task design in crowdsourcing

Authors:

Ailbhe Finnerty,

Pavel Kucherbaev,

Stefano Tranquillini,

Gregorio ConvertinoAuthors Info & Claims

CHItaly '13: Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI

Article No.: 14, Pages 1 - 4

https://doi.org/10.1145/2499149.2499168

Published: 16 September 2013 Publication History

Get Access

Abstract

Crowdsourcing is emerging as an effective method for performing tasks that require human abilities, such as tagging photos, transcribing handwriting and categorising data. Crowd workers perform small chunks of larger tasks in return for a reward, which is generally monetary. Reward can be one factor for motivating workers to produce higher quality results. Yet, as highlighted by previous research, the task design, in terms of its instructions and user interface, can also affect the workers' perception of the task, thus affecting the quality of results. In this study we investigate both factors, reward and task design, to better understand their role in relation to the quality of work in crowdsourcing. In Experiment 1 we test a variety of reward schemas while in Experiment 2 we measure the effects of the complexity of tasks and interface on attention. The long-term goal is to establish guidelines for designing tasks with the aim to maximize workers' performance.

References

[1]

Chipman, S. F., Schraagen, J. M., & Shalin, V. L. Introduction to cognitive task analysis. In J. M. Schraagen et.al. (Eds.), Cognitive task analysis (pp. 3--23). Mahwah, NJ: Lawrence Erlbaum Associates (2000).

Google Scholar

[2]

Crump MJC, McDonnell JV, Gureckis TM Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8 (3). (2013).

Google Scholar

[3]

Dow, S., Kulkarni, A. P., Bunge, B., Nguyen, T., Klemmer, S. R., and Hartmann, B. Shepherding the crowd: managing and providing feedback to crowd workers. In CHI Extend. Abstracts (2011), 1669--1674.

Digital Library

Google Scholar

[4]

Heer, J. and Bostock, M. Crowdsourcing AMT graphical perception: using mechanical turk to assess visualization design. In CHI (2010), 203--212.

Digital Library

Google Scholar

[5]

Howe, J. The rise of crowdsourcing Wired Magazine 14 (6). (2006)

Google Scholar

[6]

Kaufmann, N., and Schulze, T. Worker motivation in crowdsourcing and human computation. Education 17, (2011).

Google Scholar

[7]

Kieras, D. and Polson P. G., An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies, 22 (1985), 365--394.

Crossref

Google Scholar

[8]

Kittur, A. Crowdsourcing, Collaboration and Creativity. XRDS, 17 (2010), 22--26.

Digital Library

Google Scholar

[9]

Kittur, A. Nickerson, J V. Bernstein, Ml Gerber, E. Shaw, A. Zimmerman, J. Lease, M. and Horton, J. The future of crowd work. In CSCW (2013)., 1301--1318.

Digital Library

Google Scholar

[10]

Komarov, S. Reinecke, K. and Gajos K. Z. Crowdsourcing Performance Evaluations of User Interfaces. In CHI (2013), 207--216.

Digital Library

Google Scholar

[11]

Mason, W. A., and Watts, D. J. Financial incentives and the "performance of crowds". In KDD Workshop on Human Computation (2009), 77--85.

Digital Library

Google Scholar

[12]

Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., and Vukovic, M. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM (2011).

Google Scholar

[13]

Shaw, A. D., Horton, J. J., and Chen, D. L. Designing incentives for inexpert human raters. In CSCW (2011), 275--284.

Digital Library

Google Scholar

[14]

Sweller, J. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12 (1988) 257--85.

Crossref

Google Scholar

Cited By

View all

Tocchetti ACorti LBalayn AYurrita MLippmann PBrambilla MYang J(2024)A.I. Robustness: a Human-Centered Perspective on Technological Challenges and OpportunitiesACM Computing Surveys10.1145/3665926Online publication date: 27-May-2024
https://doi.org/10.1145/3665926
He ZHuang CDing CRohatgi SHuang T(2024)If in a Crowdsourced Data Annotation Pipeline, a GPT-4Proceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642834(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642834
Narayanan SYu GHo CYin M(2023)How does Value Similarity affect Human Reliance in AI-Assisted Ethical Decision Making?Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3600211.3604709(49-57)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.1145/3600211.3604709
Show More Cited By

Index Terms

Keep it simple: reward and task design in crowdsourcing
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Web-based interaction

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Curiosity Killed the Cat, but Makes Crowdwork Better
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Crowdsourcing systems are designed to elicit help from humans to accomplish tasks that are still difficult for computers. How to motivate workers to stay longer and/or perform better in crowdsourcing systems is a critical question for designers. ...
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Diversifying participation in crowd work can benefit the worker and requester. Increasing numbers of older adults are online, but little is known about their awareness of or how they engage in mainstream crowd work. Through an online survey with 505 ...

Comments

Information & Contributors

Information

Published In

CHItaly '13: Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI

September 2013

215 pages

ISBN:9781450320610

DOI:10.1145/2499149

Conference Chairs:
Franca Garzotto,
Massimo Zancanaro,
Program Chairs:
Antonella De Angeli,
Fabio Paternò

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 September 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHItaly '13

Sponsor:

CHItaly '13: Biannual conference of the Italian chapter of SIGCHI

September 16 - 20, 2013

Trento, Italy

Acceptance Rates

Overall Acceptance Rate 109 of 242 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
804
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Tocchetti ACorti LBalayn AYurrita MLippmann PBrambilla MYang J(2024)A.I. Robustness: a Human-Centered Perspective on Technological Challenges and OpportunitiesACM Computing Surveys10.1145/3665926Online publication date: 27-May-2024
https://doi.org/10.1145/3665926
He ZHuang CDing CRohatgi SHuang T(2024)If in a Crowdsourced Data Annotation Pipeline, a GPT-4Proceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642834(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642834
Narayanan SYu GHo CYin M(2023)How does Value Similarity affect Human Reliance in AI-Assisted Ethical Decision Making?Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3600211.3604709(49-57)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.1145/3600211.3604709
Thebault-Spieker JVenkatagiri SMine NLuther K(2023)Diverse Perspectives Can Mitigate Political Bias in Crowdsourced Content ModerationProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594080(1280-1291)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594080
Nouri ZPrakash NGadiraju UWachsmuth H(2023)Supporting Requesters in Writing Clear Crowdsourcing Task Descriptions Through Computational Flaw AssessmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584039(737-749)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584039
Paulino DCorreia ABarroso JParedes H(2023)Cognitive personalization for online microtask labor platforms: A systematic literature reviewUser Modeling and User-Adapted Interaction10.1007/s11257-023-09383-w34:3(617-658)Online publication date: 19-Sep-2023
https://doi.org/10.1007/s11257-023-09383-w
Xie HMaddalena EQarout RChecco A(2023)The Dark Side of Recruitment in Crowdsourcing: Ethics and Transparency in Micro-Task MarketplacesComputer Supported Cooperative Work (CSCW)10.1007/s10606-023-09464-932:3(439-474)Online publication date: 28-Jul-2023
https://doi.org/10.1007/s10606-023-09464-9
Duan XHo CYin M(2022)The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk EvaluationProceedings of the ACM Web Conference 202210.1145/3485447.3512239(1685-1696)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512239
Hall MAfzali MKrause MCaton S(2022)What Quality Control Mechanisms do We Need for High-Quality Crowd Work?IEEE Access10.1109/ACCESS.2022.320729210(99709-99723)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3207292
Brokman-Meltzer MPerez DGelbard R(2021)Perceived Complexity of a Project’s Optimal Work Plan Influences Its Likelihood of Adoption by Project ManagersProject Management Journal10.1177/87569728211026509(875697282110265)Online publication date: 16-Jul-2021
https://doi.org/10.1177/87569728211026509
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

Curiosity Killed the Cat, but Makes Crowdwork Better

"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work