Abstract
Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multi-rater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bouguelia, M.R., Nowaczyk, S., Santosh, K.C., Verikas, A.: Agreeing to disagree: active learning with noisy labels without crowdsourcing. Int. J. Mach. Learn. Cybern. 9(8), 1307–1319 (2018)
Wang, H., Suh, J.W., Das, S.R., Pluta, J.B., Craige, C., Yushkevich, P.A.: Multi-atlas segmentation with joint label fusion. IEEE TPAMI 35(3), 611–623 (2013). https://doi.org/10.1109/TPAMI.2012.143
Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.: MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels, December 2017. http://arxiv.org/abs/1712.05055
Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., Li, L.J.: Learning From Noisy Labels With Distillation (2017)
Ramirez, J., Berezuk, C., McNeely, A.A., Gao, F., McLaurin, J., Black, S.E.: Imaging the perivascular space as a potential biomarker of neurovascular and neurodegenerative diseases. Cell. Mol. Neurobiol. 36(2), 289–299 (2016)
Sudre, C., et al.: 3D multirater RCNN for multimodal multiclass detection and characterization of extremely small objects. In: Proceedings of the 2nd International MIDL Conference on Proceedings of Machine Learning Research, vol. 102, pp. 447–456. PMLR, London, United Kingdom, 08–10 Jul 2019
Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., Silberman, N.: Learning from noisy labels by regularized estimation of annotator confusion. In: Conference on Computer Vision and Pattern Recognition (2019)
Tillin, T., Forouhi, N.G., McKeigue, P.M., Chatuverdi, N.F.T.S., Chaturvedi, N.: Southall and Brent REvisited: cohort profile of SABRE, a UK population-based comparison of cardiovascular disease and diabetes in people of European, Indian Asian and African Caribbean origins. Int. J. Epidemiol. 41(1), 33–42 (2012). https://doi.org/10.1093/ije/dyq175
Wardlaw, J.M., et al.: Neuroimaging standards for research into small vessel disease and its contribution to ageing and neurodegeneration. Lancet Neurol. 12, 822–838 (2013)
Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE TMI 23(7), 903–921 (2004)
Acknowledgments
We are extremely grateful to all the participants of the SABRE study, and past and present members of the SABRE team. This work was supported by an Alzheimer’s Society Junior Fellowship (AS-JF-17-011), the Wellcome/EPSRC Centre for Medical Engineering [WT 203148/Z/16/Z], IMI2 grant AMYPAD [115952], the MSCA-ITN-Demo [721820], and the Wellcome Flagship Programme in High-Dimensional Neurology. The SABRE study was funded at baseline by the Medical Research Council, Diabetes UK, and the British Heart Foundation. At follow-up, the study was funded by the Wellcome Trust (067100, 37055891 and 086676/7/08/Z), the British Heart Foundation (PG/06/145, PG/08/103/26133, PG/12/29/29497 and CS/13/1/30327) and Diabetes UK (13/0004774). We gratefully acknowledge NVIDIA corporation for the donation of a GPU Tesla K40 that was used in the preparation of this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Sudre, C.H. et al. (2019). Let’s Agree to Disagree: Learning Highly Debatable Multirater Labelling. In: Shen, D., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science(), vol 11767. Springer, Cham. https://doi.org/10.1007/978-3-030-32251-9_73
Download citation
DOI: https://doi.org/10.1007/978-3-030-32251-9_73
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32250-2
Online ISBN: 978-3-030-32251-9
eBook Packages: Computer ScienceComputer Science (R0)
