Cited By
View all- Lian RZheng YWang C(2024)PrivRo: A Privacy-Preserving Crowdsourcing Service With Robust Quality AwarenessIEEE Transactions on Services Computing10.1109/TSC.2024.337715817:4(1682-1697)Online publication date: Jul-2024
Quest for the Gold Par: Minimizing the Number of Gold Questions to Distinguish between the Good and the Bad
Pages 185 - 194
Abstract
The benefits of crowdsourcing for data science have furthered its widespread use over the past decade. Yet fraudulent workers undermine the emerging crowdsourcing economy: requestors face the choice of either risking low quality results or having to pay extra money for quality safeguards like e.g., gold questions or majority voting. Obviously, the more safeguards injected into the workload, the lower the risks imposed by fraudulent workers, yet the higher the costs are. So, how many of them are really needed? Is there such a 'one size fits all' number? The aim of this paper is to identify custom-tailored numbers of gold questions per worker for managing the cost/quality balance. Our new method follows real life experiences: the more we know about workers before assigning a task, the clearer our belief or disbelief in this worker's reliability gets. Employing probabilistic models, namely Bayesian belief networks and certainty factor models, our method creates worker profiles reflecting different a-priori belief values, and we prove that the actual number of gold questions per worker can indeed be assessed. Our evaluation on real-world crowdsourcing datasets demonstrates our method's efficiency in saving money while maintaining high quality results. Moreover, our methods performs well despite the quite limited information known about workers in today's platforms.
References
[1]
Aberer, K., Despotovic, Z., "Managing trust in a peer-2-peer information system," Procs. of Int. Conf. on Information and Knowledge Management (CIKM), Atlanta, GA, USA, 2001.
[2]
Daltayanni, M. de Alfaro, L and Papadimitriou, P., "Worker-Rank: Using Employer Implicit Judgements to Infer Worker Reputation," Procs. of ACM Int. Conf. on Web Search and Data Mining (WSDM), Shanghai, China, 2015.
[3]
Davidson, S.B., Khanna, S., Milo, T. and Roy, S., "Using the Crowd for Top-k and Group-by Queries," Procs. of the Int. Conf. on Database Theory (ICDT), Genoa, Italy, 2013.
[4]
Dawid, A.P., Skene, A.M., "Maximum likelihood estimation of observer error-rates using the EM algorithm," Jour. of the Royal Statistical Society (JSTOR), vol. 28, no. 1, 1979.
[5]
Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., Xin, R., "CrowdDB: Answering queries with crowdsourcing," Pro-ceedings of the 2011 ACM SIGMOD Int. Conf. on Manage-ment of Data (SIGMOD), Athens, Greece, 2011.
[6]
Hossain, M., "Users' motivation to participate in online crowdsourcing platforms," Procs. of Int. Conf. on Innovation, Management and Technology Research (ICIMTR), Malacca, Malaysia, 2012.
[7]
Ignjatovic, A., Foo, N., Lee, C.T., "An Analytic Approach to Reputation Ranking of Participants in Online Transactions," Procs. of Int. Conf. on Web Intelligence and Intelligent Agent Technology (WI-IAT), Sydney, Australia, 2008.
[8]
Kamvar, S.D., Schlosser, M.T., Garcia-Molina, H., "The Eigentrust algorithm for reputation management in P2P net-works," Procs. of Int. Conf. on World Wide Web (WWW), Budapest, Hungary, 2003.
[9]
Kazai, G., "In Search of Quality in Crowdsourcing for Search Engine Evaluation," Proc. of European Conf. on Advances in Information Retrieval (ECIR), Dublin, Ireland, 2011.
[10]
Krieg, M.L,"A tutorial on Bayesian belief networks," Tech-nical report in the Defense Science and Technology Organiza-tion publications (DSTO), 2001.
[11]
Lofi, C., El Maarry, K., Balke, W.-T., "Skyline Queries in Crowd-Enabled Databases," Procs. of Int. Conf. on Extending Database Technology (EDBT), Genoa, Italy, 2013.
[12]
El Maarry, K., Güntzer, U., Balke, W.-T., "Realizing Impact Sourcing by Adaptive Gold Questions: A Socially Responsi-ble Measure for Workers' Trustworthiness," Procs. of Int. Conf. on Web-Age Inf. Mgmt (WAIM), Qingdao, China, 2015.
[13]
El Maarry, K., Balke, W.-T., "Retaining Rough Diamonds: Towards a Fairer Elimination of Low-skilled Workers," Procs. of Int. Conf. on Database Systems for Advanced Appli-cations (DASFAA), Hanoi, Vietnam, 2015.
[14]
El Maarry, K., Balke, W.-T., Cho, H., Hwang, S., Baba, Y., "Skill ontology-based model for Quality Assurance in Crowdsourcing," in Procs. of UnCrowd Workshop (at DASFAA'14), Bali, Indonesia, 2014
[15]
Liu, A.I.Q., Steyvers, M., "Scoring Workers in Crowdsourc-ing: How Many Control Questions are enough?" Procs. of Adv. in Neural Inf. Proc. Systems (NIPS), volume 26, State-line, NV, USA, 2013.
[16]
Mellouli, T. "Complex Certainty Factors for Rule Based Sys-tems -- Detecting Inconsistent Argumentations," Procs. of Int. Workshop on Functional and (Constraint) Logic Program-ming, Wittenberg, Germany, 2014.
[17]
Nieke, C., Güntzer, U., Balke, W.-T, "TopCrowd: Efficient Crowd-enabled Top-k Retrieval on Incomplete Data", Procs. of Int. Conf. on Conceptual Modeling (ER), Atlanta, GA, USA, 2014.
[18]
Noorian, Z., Ulieru, M., "The State of the Art in Trust and Reputation Systems: A Framework for Comparison," Jour. of Theor. and Appl. Electr. Commerce Research, vol. 5(2), 2010.
[19]
Pearl, J., "Bayesian Networks: A model of self-activated memory for evidential reasoning," Procs. of Conf. of the Cog-nitive Science Society, Irvine, CA, USA, 1985.
[20]
Pearl, J., "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference", Morgan Kaufmann Pub-lishers Inc., San Francisco, CA, USA, 1988.
[21]
Raykar, V.C., Yu, S., Zhao. L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy,L., " Learning From Crowds," Jour. of Ma-chine Learning Research (JMLR), vol. 11, 2010.
[22]
Ross, J., Zaldivar, A., Irani, L., Tomlinson, B., "Who are the Turkers" Worker Demographics in Amazon Mechanical Turk," Tech. Report, Dept. of Information, UC Irvine, USA, 2009.
[23]
Shortliffe, E.H., Buchanan, B.G., "A model of inexact reason-ing in medicine," Jour. of Mathematical Biosciences, vol. 23(3--4), 1975.
[24]
Stadler, F., "Induction and Deduction in the Sciences," Springer Netherlands, 2011.
[25]
Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J., "Whose Vote Should Count More: Optimal Integration of La-bels from Labelers of Unknown Expertise," Procs. of Adv. in Neural Inf. Proc. Systems (NIPS), Vancouver, Canada, 2009.
[26]
Yu, B., Singh, M., "Detecting Deception in Reputation Man-agement," Procs. of Int. Joint Conf. on Autonomous Agents and Multiagent Systems, Melbourne, Australia, 2003.
[27]
Zheng, Y., Wang, J., Li, G., Cheng, R., Feng, J., "QASCA": A Quality-Aware Task Assignment System for Crowdsourc-ing Applications," Procs. of ACM Int. Conf. on Management of Data (SIGMOD), Melbourne, Australia, 2015.
Index Terms
- Quest for the Gold Par: Minimizing the Number of Gold Questions to Distinguish between the Good and the Bad
Recommendations
CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems
SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of DataIn this paper, we present CrowdDQS, a system that uses the most recent set of crowdsourced voting evidence to dynamically issue questions to workers on Amazon Mechanical Turk (AMT). CrowdDQS posts all questions to AMT in a single batch, but delays the ...
Preserving worker privacy in crowdsourcing
This paper proposes a crowdsourcing quality control method with worker-privacy preservation. Crowdsourcing allows us to outsource tasks to a number of workers. The results of tasks obtained in crowdsourcing are often low-quality due to the difference in ...
Integrating change management and change auditing into information technology consulting practice
IT consulting firms develop consulting methodology to standardise their consulting practice and to ensure the quality of their consulting work. Consulting firms may implement their consulting methodologies through either procedures, methods or ...
Comments
Information & Contributors
Information
Published In
May 2018
399 pages
ISBN:9781450355636
DOI:10.1145/3201064
- General Chairs:
- Hans Akkermans,
- Kathy Fontaine,
- Ivar Vermeulen,
- Program Chairs:
- Geert-Jan Houben,
- Matthew S. Weber
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 May 2018
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
WebSci '18
Sponsor:
Acceptance Rates
WebSci '18 Paper Acceptance Rate 30 of 113 submissions, 27%;
Overall Acceptance Rate 245 of 933 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 54Total Downloads
- Downloads (Last 12 months)3
- Downloads (Last 6 weeks)0
Reflects downloads up to 24 Dec 2024
Other Metrics
Citations
Cited By
View all- Lian RZheng YWang C(2024)PrivRo: A Privacy-Preserving Crowdsourcing Service With Robust Quality AwarenessIEEE Transactions on Services Computing10.1109/TSC.2024.337715817:4(1682-1697)Online publication date: Jul-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in