research-article

Quality management on Amazon Mechanical Turk

Authors:

Panagiotis G. Ipeirotis,

Foster Provost, and

Jing WangAuthors Info & Claims

HCOMP '10: Proceedings of the ACM SIGKDD Workshop on Human Computation

July 2010

Pages 64 - 67

https://doi.org/10.1145/1837885.1837906

Published: 25 July 2010 Publication History

Get Access

Abstract

Crowdsourcing services, such as Amazon Mechanical Turk, allow for easy distribution of small tasks to a large number of workers. Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take advantage of the verification difficulty and submit answers of low quality. Currently, most requesters rely on redundancy to identify the correct answers. However, redundancy is not a panacea. Massive redundancy is expensive, increasing significantly the cost of crowdsourced solutions. Therefore, we need techniques that will accurately estimate the quality of the workers, allowing for the rejection and blocking of the low-performing workers and spammers.

However, existing techniques cannot separate the true (unrecoverable) error rate from the (recoverable) biases that some workers exhibit. This lack of separation leads to incorrect assessments of a worker's quality. We present algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error. Our algorithm generates a scalar score representing the inherent quality of each worker. We illustrate how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers. We present experimental results demonstrating the performance of the proposed algorithm under a variety of settings.

References

[1]

Carpenter, B. Multilevel bayesian models of categorical data annotation. Available at http://lingpipe-blog.com/lingpipe-white-papers/, 2008.

Google Scholar

[2]

Dawid, A. P., and Skene, A. M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics 28, 1 (Sept. 1979), 20--28.

Crossref

Google Scholar

[3]

Malone, T. W., Laubacher, R., and Dellarocas, C. Harnessing crowds: Mapping the genome of collective intelligence. Available at http://ssrn.com/abstract=1381502, 2010.

Google Scholar

[4]

Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. Journal of Machine Learning Research 11 (Apr. 2010), 1297--1322.

Digital Library

Google Scholar

Cited By

View all

Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Shrestha AFlood ASohrawardi SWright MAl-Ameen M(2024)A First Look into Targeted Clickbait and its Countermeasures: The Power of StorytellingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642301(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642301
Kim DLee JChung H(2024)A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental LimitsIEEE Transactions on Information Theory10.1109/TIT.2023.331572070:3(2076-2117)Online publication date: Mar-2024
https://doi.org/10.1109/TIT.2023.3315720
Show More Cited By

Index Terms

Quality management on Amazon Mechanical Turk
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools
  2. Human computer interaction (HCI)
2. Information systems

Recommendations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Read More
Evaluating the accessibility of crowdsourcing tasks on Amazon's mechanical turk
ASSETS '14: Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility

Crowd work web sites such as Amazon Mechanical Turk enable individuals to work from home, which may be useful for people with disabilities. However, the web sites for finding and performing crowd work tasks must be accessible if people with disabilities ...
Read More
Crowdsourcing Quality Concerns: An Examination of Amazon’s Mechanical Turk
SIGITE '22: Proceedings of the 23rd Annual Conference on Information Technology Education

The use of crowdsourcing platforms, such as Amazon’s Mechanical Turk (MTurk), have been an effective and frequent tool for researchers to gather data from participants for a study. It provides a fast, efficient, and cost-effective method for acquiring ...
Read More

Comments

Information & Contributors

Information

Published In

HCOMP '10: Proceedings of the ACM SIGKDD Workshop on Human Computation

July 2010

95 pages

ISBN:9781450302227

DOI:10.1145/1837885

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Division of Information and Intelligent Systems

Conference

KDD '10

Sponsor:

KDD '10: The 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

July 25, 2010

Washington DC

Upcoming Conference

KDD '24

Sponsor:
sigkdd
sigkdd

The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

604
Total Citations
View Citations
3,868
Total Downloads

Downloads (Last 12 months)179
Downloads (Last 6 weeks)27

Other Metrics

View Author Metrics

Citations

Cited By

View all

Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Shrestha AFlood ASohrawardi SWright MAl-Ameen M(2024)A First Look into Targeted Clickbait and its Countermeasures: The Power of StorytellingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642301(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642301
Kim DLee JChung H(2024)A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental LimitsIEEE Transactions on Information Theory10.1109/TIT.2023.331572070:3(2076-2117)Online publication date: Mar-2024
https://doi.org/10.1109/TIT.2023.3315720
Anderson BHerman PHays R(2024)Predictors of Pain Management Strategies in Adults with Low-Back Pain: A Secondary Analysis of Amazon Mechanical Turk Survey DataJournal of Integrative and Complementary Medicine10.1089/jicm.2023.023330:3(297-305)Online publication date: 1-Mar-2024
https://doi.org/10.1089/jicm.2023.0233
Shrestha ABehfar AAl-Ameen M(2024) “It is Luring You to Click on the Link With False Advertising” - Mental Models of Clickbait and Its Impact on User’s Perceptions and Behavior Towards Clickbait Warnings International Journal of Human–Computer Interaction10.1080/10447318.2024.2323248(1-19)Online publication date: 8-Mar-2024
https://doi.org/10.1080/10447318.2024.2323248
Chang CHe YDu XYang XXie H(2024)Dynamic Labeling: A Control System for Labeling Styles in Image Annotation TasksHuman Interface and the Management of Information10.1007/978-3-031-60107-1_8(99-118)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60107-1_8
Jeong HChung HKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Recovering top-two answers and confusion probability in multi-choice crowdsourcingProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619013(14836-14868)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619013
Chen PYang YYang DSun HChen ZLin PElkind E(2023)Black-box data poisoning attacks on crowdsourcingProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/332(2975-2983)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/332
Washington P(2023)Crowdsourcing and Human-In-The-Loop Workflows in Precision Health: Perspective (Preprint)Journal of Medical Internet Research10.2196/51138Online publication date: 22-Jul-2023
https://doi.org/10.2196/51138
Zhang HSong HLi SZhou MSong D(2023)A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language ModelsACM Computing Surveys10.1145/361768056:3(1-37)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3617680
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Evaluating the accessibility of crowdsourcing tasks on Amazon's mechanical turk

Crowdsourcing Quality Concerns: An Examination of Amazon’s Mechanical Turk

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Evaluating the accessibility of crowdsourcing tasks on Amazon's mechanical turk

Crowdsourcing Quality Concerns: An Examination of Amazon’s Mechanical Turk

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations