Abstract
Supervised machine learning tasks require human-labeled data. Crowdsourcing allows scaling up the labeling process, but the quality of the labels obtained can vary. To address this limitation, we propose methods for predicting label quality based on worker trajectories, i.e., on the sequence of documents workers explore during their crowdsourcing tasks. Trajectories represent a lightweight and non-intrusive form of worker behavior signal. We base our analysis on previously collected datasets composed of thousands of assessment data records including information such as workers’ trajectories, workers’ assessments, and experts’ assessments. We model such behavior sequences as embeddings, to facilitate their management. Then, we: (1) use supervised methods to predict worker performance using a given ground truth; (2) perform an unsupervised analysis to provide insight into crowdsourcing quality when no gold standard is available. We test several supervised approaches which all beat the baseline we propose. Also, we identify significant differences between trajectory clusters in terms of assessments and worker performance. The trajectory-based analysis is a promising direction for non-intrusive worker performance evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
References
ABC and RMIT University: RMIT ABC Fact Check (2021). https://apo.org.au/collection/302996/rmit-abc-fact-check. Accessed 10 June 2021
Agichtein, E., Brill, E., Dumais, S., Ragno, R.: Learning user interaction models for predicting web search result preferences. In: SIGIR, p. 3–10. ACM (2006)
Benham, R., Mackenzie, J., Culpepper, J.S., Moffat, A.: Different keystrokes for different folks: visualizing crowdworker querying behavior. In: CHIIR, pp. 331–335. ACM (2021)
Bhuiyan, M.M., Zhang, A.X., Sehat, C.M., Mitra, T.: Investigating differences in crowdsourced news credibility assessment: raters, tasks, and expert criteria. PACMHCI 4, 1–26 (2020)
Callison-Burch, C.: Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk. In: EMNLP 2009, pp. 286–295. ACM (2009)
Ceolin, D., Noordegraaf, J., Aroyo, L.: Capturing the ineffable: collecting, analysing, and automating web document quality assessments. In: Blomqvist, E., Ciancarini, P., Poggi, F., Vitali, F. (eds.) EKAW 2016. LNCS (LNAI), vol. 10024, pp. 83–97. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49004-5_6
Ceolin, D., Primiero, G., Wielemaker, J., Soprano, M.: Assessing the quality of online reviews using formal argumentation theory. In: Brambilla, M., Chbeir, R., Frasincar, F., Manolescu, I. (eds.) ICWE 2021. LNCS, vol. 12706, pp. 71–87. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-74296-6_6
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. JAIR 16, 321–357 (2002)
Chen, C., et al.: Tensorflow official model garden (2020). https://github.com/tensorflow/models
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP, pp. 1724–1734. ACL (2014)
Dekel, O., Shamir, O.: Vox Populi: collecting high-quality labels from a crowd. In: COLT (2009)
Demartini, G., Mizzaro, S., Spina, D.: Human-in-the-loop artificial intelligence for fighting online misinformation: challenges and opportunities. IEEE Bull. 43(3), 65–74 (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
Eady, G., Nagler, J., Guess, A., Zilinsky, J., Tucker, J.A.: How many people live in political bubbles on social media? evidence from linked survey and Twitter data. SAGE Open 9(1), 2158244019832705 (2019)
Epstein, Z., Pennycook, G., Rand, D.: Will the crowd game the algorithm?: using layperson judgments to combat misinformation on social media by downranking distrusted sources. In: CHI, pp. 1–11. ACM (2020)
Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, pp. 226–231. AAAI Press (1996)
Fang, H., Zhang, D., Shu, Y., Guo, G.: Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations (2020)
Flaxman, S., Goel, S., Rao, J.M.: Filter bubbles, echo chambers, and online news consumption. Public Opin. Q. 80(S1), 298–320 (2016)
Guo, F.: Analyzing Workers Trajectories for Performance Evaluation - An Information Quality Assessment case (2021). Master’s Thesis
Han, L., Checco, A., Difallah, D., Demartini, G., Sadiq, S.: Modelling user behavior dynamics with embeddings. In: CIKM 2020, pp. 445–454 (2020)
Han, L., et al.: All Those Wasted Hours: On Task Abandonment in Crowdsourcing. In: WSDM, pp. 321–329. ACM (2019)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neur. Comp. 9(8), 1735–1780 (1997)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188 (2014)
La Barbera, D., Roitero, K., Demartini, G., Mizzaro, S., Spina, D.: Crowdsourcing truthfulness: the impact of judgment scale and assessor bias. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 207–214. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_26
Li, G., et al.: Misinformation-oriented expert finding in social networks. WWW 23(2), 693–714 (2020)
Lim, S., Jatowt, A., Färber, M., Yoshikawa, M.: Annotating and analyzing biased sentences in news articles using crowdsourcing. In: LREC, pp. 1478–1484. ELRA (2020)
Liu, Y., Wu, Y.F.B.: FNED: a deep network for fake news early detection on social media. TIST 38(3), 1–33 (2020)
Macqueen, J.: Some methods for classification and analysis of multivariate observations. In: Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297 (1967)
Maddalena, E., Ceolin, D., Mizzaro, S.: Multidimensional news quality: a comparison of crowdsourcing and nichesourcing. In: INRA, vol. 2482. CEUR-WS.org (2018)
Nguyen, C.T.: Echo chambers and epistemic bubbles. Episteme 17(2), 141–161 (2020)
Olejnik, S.F., Algina, J.: Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psych. Methods 8, 434–47 (2004)
PolitiFact: The Principles of the Truth-O-Meter: PolitiFact’s methodology for independent fact-checking (2020). https://www.politifact.com/article/2018/feb/12/principles-truth-o-meter-politifacts-methodology-i/
Redi, J.A., Hoßfeld, T., Korshunov, P., Mazza, F., Povoa, I., Keimel, C.: Crowdsourcing-based multimedia subjective evaluations: A case study on image recognizability and aesthetic appeal. In: CrowdMM, pp. 29–34. ACM (2013)
Roitero, K., et al.: The COVID-19 infodemic: can the crowd judge recent misinformation objectively? In: CIKM, pp. 1305–1314. ACM (2020)
Schuster, M., Paliwal, K.: Bidirectional recurrent neural networks. TSP 45(11), 2673–2681 (1997)
Soprano, M., et al.: The many dimensions of truthfulness: crowdsourcing misinformation assessments on a multidimensional scale. IP &M 58(6), 102710 (2021)
Sun, F., et al.: Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In: CIKM, pp. 1441–1450. Springer (2019)
Uprety, S., et al.: Quantum-like structure in multidimensional relevance judgements. In: Jose, M., et al. (eds.) ECIR 2020. LNCS, vol. 12035, pp. 728–742. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45439-5_48
Wang, W.Y.: “Liar, Liar Pants on Fire”: a new benchmark dataset for fake news detection. In: ACL 2017, pp. 422–426. ACL (2017)
Zampieri, F., Roitero, K., Culpepper, J.S., Kurland, O., Mizzaro, S.: On topic difficulty in IR evaluation: the effect of systems, corpora, and system components. In: SIGIR, pp. 909–912. ACM (2019)
Zhang, Y., Zhang, J., Lease, M., Gwizdka, J.: Multidimensional relevance modeling via psychometrics and crowdsourcing. In: SIGIR, pp. 435–444 (2014)
Zheng, H., Li, D., Hou, W.: Task design, motivation, and participation in crowdsourcing contests. Int. J. Electron. Commer. 15, 57–88 (2011)
Zuccon, G., Leelanupab, T., Whiting, S., Yilmaz, E., Jose, J.M., Azzopardi, L.: Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Inf. Retr. 16(2), 267–305 (2013)
Acknowledgments
This research has been partly supported by the Netherlands eScience Center project “The Eye of the Beholder” (project nr. 027.020.G15).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ceolin, D., Roitero, K., Guo, F. (2023). Predicting Crowd Workers Performance: An Information Quality Case. In: Garrigós, I., Murillo Rodríguez, J.M., Wimmer, M. (eds) Web Engineering. ICWE 2023. Lecture Notes in Computer Science, vol 13893. Springer, Cham. https://doi.org/10.1007/978-3-031-34444-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-34444-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34443-5
Online ISBN: 978-3-031-34444-2
eBook Packages: Computer ScienceComputer Science (R0)