Quantifying and Leveraging Uncertain and Imprecise Answers in Multiple Choice Questionnaires for Crowdsourcing: Quantifier et exploiter des réponses incertaines et imprécises dans des questionnaires à choix multiples pour le crowdsourcing
Authors: 
Constance Thierry, 
Géry Casiez, Jean-Christophe Dubois, Yolande Le-Gall, 
Sylvain Malacria, 
Thomas Pietrzak, Arnaud Martin, Pierrick UroAuthors Info & Claims
Constance Thierry, Géry Casiez, Jean-Christophe Dubois, Yolande Le-Gall, Sylvain Malacria, Thomas Pietrzak, Arnaud Martin, Pierrick UroAuthors Info & ClaimsArticle No.: 14, Pages 1 - 10
Abstract
Questionnaires are efficient for collecting numerous user feedback. However, the reliability of results is a major issue, even with honest participants. Indeed, they face situations of doubt, and usually do not have the option to express their hesitations. We describe a user study in which we provide participants with the possibility to give 1) a certitude rate 2) imprecise answers, 3) both a certitude rate and imprecise answers. Firstly, we observe that contributors express their hesitations consistently: there is a correlation between the task difficulty on the one hand, and the uncertainty and imprecision of the answer, on the other hand. Secondly, our results demonstrate the effectiveness of the decision-making process by using this additional information with the belief functions theory. Indeed, this process helps to reduce the error rate and fewer participants are required to reach a satisfactory correct answers rate.
References
[1]
Lina Abassi and Imen Boukhris. 2019. A Worker Clustering-Based Approach of Label Aggregation under the Belief Function Theory. Applied Intelligence 49, 1 (Jan. 2019), 53–62. https://doi.org/10.1007/s10489-018-1209-z
[2]
Yael Amsterdamer, Yael Grossman, Tova Milo, and Pierre Senellart. 2013. Crowd Mining. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (New York, New York, USA) (SIGMOD ’13). Association for Computing Machinery, New York, NY, USA, 241–252. https://doi.org/10.1145/2463676.2465318
[3]
AMT [n.d.]. Amazon Mechanical Turk. https://docs.aws.amazon.com/AWSMechTurk/latest/ RequesterUI/CreatingYourBatchofHITs.htmlAccessed: 17/12/2020.
[4]
Yoram Bachrach, Thore Graepel, Gjergji Kasneci, Michal Kosinski, and Jurgen Van Gael. 2012. Crowd IQ: Aggregating Opinions to Boost Performance. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (Valencia, Spain) (AAMAS ’12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 535–542.
[5]
Rubi Boim, Ohad Greenshpan, Tova Milo, Slava Novgorodov, Neoklis Polyzotis, and Wang-Chiew Tan. 2012. Asking the right questions in crowd data sourcing. In 2012 IEEE 28th International Conference on Data Engineering. IEEE, 1261–1264.
[6]
Géry Casiez and Nicolas Roussel. 2011. No More Bricolage!: Methods and Tools to Characterize, Replicate and Compare Pointing Transfer Functions. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). ACM, New York, NY, USA, 603–614. https://doi.org/10.1145/2047196.2047276
[7]
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979), 20–28.
[8]
Arthur P. Dempster. 2008. Upper and Lower Probabilities Induced by a Multivalued Mapping. Springer Berlin Heidelberg, Berlin, Heidelberg, 57–72. https://doi.org/10.1007/978-3-540-44792-4_3
[9]
Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological) (1977), 1–38.
[10]
Javier Diaz, Maria Rifqi, Bernadette Bouchon-Meunier, Sandra Jhean-Larose, and Guy Denhiére. 2008. Imperfect Answers in Multiple Choice Questionnaires. In Times of Convergence. Technologies Across Learning Contexts, Pierre Dillenbourg and Marcus Specht (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 144–154.
[11]
Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the Crowd Yields Better Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1013–1022. https://doi.org/10.1145/2145204.2145355
[12]
Graham Farrell. 2006. A comparison of an innovative web-based assessment tool utilizing confidence measurement to the traditional multiple choice, short answer and problem solving questions. https://hdl.handle.net/2134/4587
[13]
Olusegun Folorunso and Olusegun Afeez Mustapha. 2015. A fuzzy expert system to Trust-Based Access Control in crowdsourcing environments. Applied Computing and Informatics 11, 2 (2015), 116 – 129. https://doi.org/10.1016/j.aci.2014.07.001
[14]
G. Kazai, J. Kamps, and N. Milic-Frayling. 2013. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Inf Retrieval 16 (2013), 138–178. https://doi.org/10.1007/s10791-012-9205-0
[15]
K.S. Khan, D.A. Davies, and J.K. Gupta. 2001. Formative self-assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Med Teach. 23, 2 (2001), 158–163. https://doi.org/10.1080/01421590031075
[16]
Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work. Association for Computing Machinery, New York, NY, USA, 1301–1318. https://doi.org/10.1145/2441776.2441923
[17]
Dalila Koulougli, Allel HadjAli, and Idir Rassoul. 2016. Handling query answering in crowdsourcing systems: A belief function-based approach. In 2016 Annual Conference of the North American Fuzzy Information Processing Society, NAFIPS 2016. IEEE, El Paso, TX, USA, 1–6. https://doi.org/10.1109/NAFIPS.2016.7851590
[18]
Adam Marcus and Aditya Parameswaran. 2015. Crowdsourced data management: Industry and academic perspectives. Foundations and Trends in Databases 6, 1-2 (2015), 1–161.
[19]
Arnaud Martin. 2019. Conflict Management in Information Fusion with Belief Functions. In Information quality in information fusion and decision making. Springer, 79–97.
[20]
Amal Ben Rjab; Mouloud Kharoune; Zoltan Miklos; Arnaud Martin. 2016. Characterization of experts in crowdsourcing platforms. Belief Functions: Theory and Applications. 9861 (2016).
[21]
A. T. Nguyen, Byron C. Wallace, and Matthew Lease. 2015. Combining Crowd and Expert Labels Using Decision Theoretic Active Learning. In Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP15/paper/view/11567
[22]
Bahareh Rahmanian and Joseph G. Davis. 2014. User Interface Design for Crowdsourcing Systems. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces (Como, Italy) (AVI ’14). Association for Computing Machinery, New York, NY, USA, 405–408. https://doi.org/10.1145/2598153.2602248
[23]
Vikas C Raykar and Shipeng Yu. 2012. Annotation models for crowdsourced ordinal data. Journal of Machine Learning Research 13 (2012).
[24]
Glenn Shafer. 1976. A mathematical theory of evidence. Vol. 42. Princeton university press.
[25]
Aaron D. Shaw, John J. Horton, and Daniel L. Chen. 2011. Designing Incentives for Inexpert Human Raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (Hangzhou, China) (CSCW ’11). Association for Computing Machinery, New York, NY, USA, 275–284. https://doi.org/10.1145/1958824.1958865
[26]
C. Thierry, J.-C. Dubois, Y. Le Gall, and A. Martin. 2019. Modeling Uncertainty and Inaccuracy on Data from Crowdsourcing Platforms: MONITOR. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). 776–783. https://doi.org/10.1109/ICTAI.2019.00112
[27]
Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. 2010. The Multidimensional Wisdom of Crowds. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 2 (Vancouver, British Columbia, Canada) (NIPS’10). Curran Associates Inc., Red Hook, NY, USA, 2424–2432. https://doi.org/10.5555/2997046.2997166
[28]
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.). Vol. 22. Curran Associates, Inc., 2035–2043. https://proceedings.neurips.cc/paper/2009/file/f899139df5e1059396431415e770c6dd-Paper.pdf
Index Terms
- Quantifying and Leveraging Uncertain and Imprecise Answers in Multiple Choice Questionnaires for Crowdsourcing: Quantifier et exploiter des réponses incertaines et imprécises dans des questionnaires à choix multiples pour le crowdsourcing
Recommendations
OLAP over uncertain and imprecise data
We extend the OLAP data model to represent data ambiguity, specifically imprecision and uncertainty, and introduce an allocation-based approach to the semantics of aggregation queries over such data. We identify three natural query properties and use ...
Naive possibilistic classifiers for imprecise or uncertain numerical data
In real-world problems, input data may be pervaded with uncertainty. In this paper, we investigate the behavior of naive possibilistic classifiers, as a counterpart to naive Bayesian ones, for dealing with classification tasks in the presence of ...
Database aggregation of imprecise and uncertain evidence
special issue: Knowledge discovery from distributed information sourcesInformation from which knowledge can be discovered is frequently distributed due to having been recorded at different times or to having arisen from different sources. Such information is often subject to both imprecision and uncertainty. The Dempster-...
Comments
Information & Contributors
Information
Published In
March 2024
209 pages
ISBN:9798400718113
DOI:10.1145/3649792
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
IHM '24
IHM '24: 35th International Francophone Conference on Human-Computer Interaction
March 25 - 29, 2024
Paris, France
Acceptance Rates
Overall Acceptance Rate 103 of 199 submissions, 52%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 19Total Downloads
- Downloads (Last 12 months)19
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format