research-article

Supervised learning from multiple experts: whom to trust when everyone lies a bit

Authors:

Gerardo Hermosillo Valadez,

Luca Bogoni,

Linda MoyAuthors Info & Claims

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

Pages 889 - 896

https://doi.org/10.1145/1553374.1553488

Published: 14 June 2009 Publication History

Get Access

Abstract

We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.

References

[1]

Dawid, A. P., & Skeene, A. M. (1979). Maximum likei-hood estimation of observed error-rates using the EM algorithm. Applied Statistics, 28, 20--28.

Crossref

Google Scholar

[2]

Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39, 1--38.

Crossref

Google Scholar

[3]

Frank, E., & Hall, M. (2001). A simple approach to ordinal classification. Lecture Notes in Computer Science, 145--156.

Google Scholar

[4]

Hui, S. L., & Zhou, X. H. (1998). Evaluation of diagnostic tests without a gold standard. Statistical Methods in Medical Research, 7, 354--370.

Crossref

Google Scholar

[5]

Lugosi, G. (1992). Learning with an unreliable teacher. Pattern Recognition, 25, 79--87.

Digital Library

Google Scholar

[6]

Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models (pp. 355--368). Kluwer Academic Publishers.

Digital Library

Google Scholar

[7]

Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? Improving data quality and data mining using multiple, noisy labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 614--622).

Digital Library

Google Scholar

[8]

Smyth, P. (1995). Learning with probabilistic supervision. In Computational learning theory and natural learning systems 3, 163--182. MIT Press.

Google Scholar

[9]

Smyth, P., Fayyad, U., Burl, M., Perona, P., & Baldi, P. (1995). Inferring ground truth from subjective labelling of venus images. In Advances in neural information processing systems 7, 1085--1092.

Google Scholar

[10]

Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254--263).

Digital Library

Google Scholar

[11]

Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. Proceedings of the First IEEE Workshop on Internet Vision at CVPR 08 (pp. 1--8).

Crossref

Google Scholar

Cited By

View all

Zhang JWu MSun ZZhou C(2025)Learning From Crowds Using Graph Neural Networks With Attention MechanismIEEE Transactions on Big Data10.1109/TBDATA.2024.337810011:1(86-98)Online publication date: Feb-2025
https://doi.org/10.1109/TBDATA.2024.3378100
Xia MHuang ZWu RLyu GZhao JChen GWang HSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Unbiased multi-label learning from crowdsourced annotationsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694289(54064-54081)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694289
Sun YZhang YHan BLi LZhou JSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Self-cognitive denoising in the presence of multiple noisy label sourcesProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693994(47261-47279)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693994
Show More Cited By

Index Terms

Supervised learning from multiple experts: whom to trust when everyone lies a bit
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Model development and analysis
      1. Model verification and validation
      2. Modeling methodologies
2. Mathematics of computing
  1. Probability and statistics

Recommendations

Multiple kernel-based dictionary learning for weakly supervised classification

In this paper, we develop a multiple instance learning (MIL) algorithm using the dictionary learning framework where the labels are given in the form of positive and negative bags, with each bag containing multiple samples. A positive bag is guaranteed ...
Inductive Semi-supervised Multi-Label Learning with Co-Training
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

In multi-label learning, each training example is associated with multiple class labels and the task is to learn a mapping from the feature space to the power set of label space. It is generally demanding and time-consuming to obtain labels for training ...
Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning

Object category localization is a challenging problem in computer vision. Standard supervised training requires bounding box annotations of object instances. This time-consuming annotation process is sidestepped in weakly supervised learning. In this ...

Comments

Information & Contributors

Information

Published In

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

June 2009

1331 pages

ISBN:9781605585161

DOI:10.1145/1553374

General Chair:
Andrea Danyluk
Williams College
,
Program Chairs:
Léon Bottou
NEC Laboratories America
,
Michael Littman
Rutgers University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICML '09

Sponsor:

Microsoft Research

ICML '09: The 26th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 14 - 18, 2009

Quebec, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

199
Total Citations
View Citations
1,302
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhang JWu MSun ZZhou C(2025)Learning From Crowds Using Graph Neural Networks With Attention MechanismIEEE Transactions on Big Data10.1109/TBDATA.2024.337810011:1(86-98)Online publication date: Feb-2025
https://doi.org/10.1109/TBDATA.2024.3378100
Xia MHuang ZWu RLyu GZhao JChen GWang HSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Unbiased multi-label learning from crowdsourced annotationsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694289(54064-54081)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694289
Sun YZhang YHan BLi LZhou JSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Self-cognitive denoising in the presence of multiple noisy label sourcesProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693994(47261-47279)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693994
Tang RZhang LWang L(2024)DualPSNet: A Discrepant-Annotations-Driven Framework for Medical Image Segmentation from Dual-Polygon Supervision2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650187(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650187
Ballester RLabeyrie YMulayim MFernandez-Marquez JCerquides J(2024)Crowdsourced geolocationCognitive Systems Research10.1016/j.cogsys.2024.10126688:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.cogsys.2024.101266
Tschirschwitz DBenz CFlorek MNorderhus HStein BRodehorst V(2024)Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object Detection with Repeated LabelsPattern Recognition10.1007/978-3-031-54605-1_39(605-623)Online publication date: 8-Mar-2024
https://doi.org/10.1007/978-3-031-54605-1_39
Wu GZhou LXia JLi LBao XWu X(2023)Crowdsourcing Truth Inference Based on Label Confidence ClusteringACM Transactions on Knowledge Discovery from Data10.1145/355654517:4(1-20)Online publication date: 24-Feb-2023
https://dl.acm.org/doi/10.1145/3556545
Anton SMartínez-Ojeda RHristu RStanciu GToma ABanica CFernández EHuttunen MBueno JStanciu S(2023)Automated Detection of Corneal Edema With Deep Learning-Assisted Second Harmonic Generation MicroscopyIEEE Journal of Selected Topics in Quantum Electronics10.1109/JSTQE.2023.325868729:6: Photonic Signal Processing(1-10)Online publication date: Nov-2023
https://doi.org/10.1109/JSTQE.2023.3258687
Wang CCai ZSeo DLi Y(2023)TMETA: Trust Management for the Cold Start of IoT Services With Digital-Twin-Aided BlockchainIEEE Internet of Things Journal10.1109/JIOT.2023.328510810:24(21337-21348)Online publication date: 15-Dec-2023
https://doi.org/10.1109/JIOT.2023.3285108
Terbouche HMorel MRodriguez MOthmani A(2023)Multi-Annotation Attention Model for Video Summarization2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00316(3143-3152)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00316
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Multiple kernel-based dictionary learning for weakly supervised classification

Inductive Semi-supervised Multi-Label Learning with Co-Training

Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations