Article

Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation

Authors:

Bernhard Kainz,

Daniel RueckertAuthors Info & Claims

Uncertainty for Safe Utilization of Machine Learning in Medical Imaging: 4th International Workshop, UNSURE 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings

Pages 59 - 69

https://doi.org/10.1007/978-3-031-16749-2_6

Published: 18 September 2022 Publication History

Abstract

Probability calibration for deep models is highly desirable in safety-critical applications such as medical imaging. It makes output probabilities of deep networks interpretable, by aligning prediction probability with the actual accuracy in test data. In image segmentation, well-calibrated probabilities allow radiologists to identify regions where model-predicted segmentations are unreliable. These unreliable predictions often occur to out-of-domain (OOD) images that are caused by imaging artifacts or unseen imaging protocols. Unfortunately, most previous calibration methods for image segmentation perform sub-optimally on OOD images. To reduce the calibration error when confronted with OOD images, we propose a novel post-hoc calibration model. Our model leverages the pixel susceptibility against perturbations at the local level, and the shape prior information at the global level. The model is tested on cardiac MRI segmentation datasets that contain unseen imaging artifacts and images from an unseen imaging protocol. We demonstrate reduced calibration errors compared with the state-of-the-art calibration algorithm.

References

[1]

Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE CVPR, pp. 427–436 (2015)

[2]

Gonzalez C, Gotkowski K, Bucher A, Fischbach R, Kaltenborn I, Mukhopadhyay A, et al. de Bruijne M et al. Detecting when pre-trained nnU-Net models fail silently for Covid-19 lung lesion segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 2021 Cham Springer 304-314

[3]

Ding, Z., Han, X., Liu, P., Niethammer, M.: Local temperature scaling for probability calibration. In: Proceedings of the IEEE/CVF ICCV, pp. 6889–6899 (2021)

[4]

Platt, J., et al.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, vol. 10, no. 3, pp. 61–74 (1999)

[5]

Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In: ICML, vol. 1, pp. 609–616. Citeseer (2001)

[6]

Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML, pp. 1321–1330. PMLR (2017)

[7]

Tomani, C., Buettner, F.: Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In: Proceedings of the AAAI Conference, vol. 35, pp. 9886–9896 (2021)

[8]

Ji, B., Jung, H., Yoon, J., Kim, K., et al.: Bin-wise temperature scaling (BTS): improvement in confidence calibration performance through simple scaling techniques. In: IEEE/CVF ICCV Workshop, pp. 4190–4196. IEEE (2019)

[9]

Ovadia, Y., et al.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In: Advances in NeurIPS, vol. 32 (2019)

[10]

Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., Dokania, P.: Calibrating deep neural networks using focal loss. In: Advances in NeurIPS, vol. 33, pp. 15288–15299 (2020)

[11]

Karimi, D., Gholipour, A.: Improving calibration and out-of-distribution detection in deep models for medical image segmentation. IEEE Trans. Artif. Intell., 1 (2022, early access). https://ieeexplore.ieee.org/document/9735278

[12]

Kireev, K., Andriushchenko, M., Flammarion, N.: On the effectiveness of adversarial training against common corruptions. arXiv preprint arXiv:2103.02325 (2021)

[13]

Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: ICML, pp. 1050–1059. PMLR (2016)

[14]

Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in NIPS, vol. 30 (2017)

[15]

Wang G, Li W, Aertsen M, Deprest J, Ourselin S, and Vercauteren T Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks Neurocomputing 2019 338 34-45

[16]

Mehrtash A, Wells WM, Tempany CM, Abolmaesumi P, and Kapur T Confidence calibration and predictive uncertainty estimation for deep medical image segmentation IEEE Trans. Med. Imaging 2020 39 12 3868-3878

[17]

Baumgartner CF, et al., et al. Shen D, et al., et al. PHiSeg: capturing uncertainty in medical image segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 2019 Cham Springer 119-127

[18]

Zhang L et al. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation IEEE Trans. Med. Imaging 2020 39 7 2531-2540

[19]

Chen C, et al., et al. Martel AL, et al., et al. Realistic adversarial data augmentation for MR image segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 2020 Cham Springer 667-677

[20]

Ouyang, C., et al.: Causality-inspired single-source domain generalization for medical image segmentation. arXiv preprint arXiv:2111.12525 (2021)

[21]

Larrazabal AJ, Martínez C, Glocker B, and Ferrante E Post-DAE: anatomically plausible segmentation via post-processing with denoising autoencoders IEEE Trans. Med. Imaging 2020 39 12 3813-3820

[22]

Liu, Q., Chen, C., Dou, Q., Heng, P.A.: Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary (2022)

[23]

Chen C, Hammernik K, Ouyang C, Qin C, Bai W, Rueckert D, et al. de Bruijne M et al. Cooperative training and latent space data augmentation for robust medical image segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 2021 Cham Springer 149-159

[24]

Robinson R et al. Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S, et al. Automatic quality control of cardiac MRI segmentation in large-scale population imaging Medical Image Computing and Computer Assisted Intervention – MICCAI 2017 2017 Cham Springer 720-727

[25]

Li K, Yu L, and Heng PA Towards reliable cardiac image segmentation: assessing image-level and pixel-level segmentation quality via self-reflective references Med. Image Anal. 2022 78 102426

[26]

Wang S, et al., et al. Martel AL, et al., et al. Deep generative model-based quality control for cardiac MRI segmentation Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 2020 Cham Springer 88-97

[27]

Nixon, J., Dusenberry, M.W., Zhang, L., Jerfel, G., Tran, D.: Measuring calibration in deep learning. In: CVPR Workshops, vol. 2 (2019)

[28]

Raju, A., et al.: Deep implicit statistical shape models for 3D medical image delineation. arXiv (2021)

[29]

Bernard O et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans. Med. Imaging 2018 37 11 2514-2525

[30]

Pérez-García F, Sparks R, and Ourselin S TorchIO: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning Comput. Methods Programs Biomed. 2021 208

[31]

Zhuang X et al. Cardiac segmentation on late gadolinium enhancement MRI: a benchmark study from multi-sequence cardiac MR segmentation challenge Med. Image Anal. 2022 81

[32]

Ronneberger O, Fischer P, and Brox T Navab N, Hornegger J, Wells WM, and Frangi AF U-Net: convolutional networks for biomedical image segmentation Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 2015 Cham Springer 234-241

[33]

Naeini, M.P., Cooper, G., Hauskrecht, M.: Obtaining well calibrated probabilities using Bayesian binning. In: Twenty-Ninth AAAI Conference (2015)

Cited By

Philps BValdes Hernandez MBernabeu Llinares M(2023)Proper Scoring Loss Functions Are Simple and Effective for Uncertainty Quantification of White Matter HyperintensitiesUncertainty for Safe Utilization of Machine Learning in Medical Imaging10.1007/978-3-031-44336-7_21(208-218)Online publication date: 12-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-44336-7_21

Index Terms

Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
1. Applied computing
  1. Life and medical sciences
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Image segmentation
  2. Machine learning
    1. Learning paradigms
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Review of brain MRI image segmentation methods

Brain image segmentation is one of the most important parts of clinical diagnostic tools. Brain images mostly contain noise, inhomogeneity and sometimes deviation. Therefore, accurate segmentation of brain images is a very difficult task. However, the ...
Liver segmentation in MRI

There are few fully automated methods for liver segmentation in magnetic resonance images (MRI) despite the benefits of this type of acquisition in comparison to other radiology techniques such as computed tomography (CT). Motivated by medical ...
An Improved Image Denoising and Segmentation Approach for Detecting Tumor from 2-D MRI Brain Images
ACSAT '12: Proceedings of the 2012 International Conference on Advanced Computer Science Applications and Technologies

Image denoising and segmentation are the two most challenging fields in medical image processing particularly when it is application specific. The presence of noise not only degrades the visual quality but also immensely affects the accuracies of ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Uncertainty for Safe Utilization of Machine Learning in Medical Imaging: 4th International Workshop, UNSURE 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings

Sep 2022

151 pages

ISBN:978-3-031-16748-5

DOI:10.1007/978-3-031-16749-2

Editors:
Carole H. Sudre
University College London, London, UK
,
Christian F. Baumgartner
University of Tübingen, Tübingen, Germany
,
Adrian Dalca
Massachusetts General Hospital, Charlestown, MA, USA
,
Chen Qin
Imperial College London, London, UK
,
Ryutaro Tanno
Google DeepMind, London, UK
,
Koen Van Leemput
Technical University of Denmark, Kongens Lyngby, Denmark
,
William M. Wells III
Harvard Medical School, Boston, MA, USA

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 18 September 2022

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Philps BValdes Hernandez MBernabeu Llinares M(2023)Proper Scoring Loss Functions Are Simple and Effective for Uncertainty Quantification of White Matter HyperintensitiesUncertainty for Safe Utilization of Machine Learning in Medical Imaging10.1007/978-3-031-44336-7_21(208-218)Online publication date: 12-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-44336-7_21

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents