Abstract
H &E images can be utilized to predict genetic mutations as biomarkers to potentially substitute many molecular biomarker assays in order to aid patients. Having a single model built by conducting prediction tasks simultaneously can save computation resources and provide a more generalizable model for future usage. A basic technique for generating such a comprehensive and efficient model is to employ a multi-task learning approach. However, overfitting the model to the trivial answers can occur in training for multiple tasks with extremely imbalanced class labels where resampling and rebalancing for all minor classes simultaneously are prohibited. Herein we propose a sequential multi-task learning approach to train a single model capable of predicting multiple genetic mutations while avoiding overfitting to trivial answers for imbalanced classes. We compared our strategy to the baseline multi-task training, as well as two more advanced approaches: (1) using weighted loss and (2) using self-supervised pre-training. We also used a trimming method to deal with noisy labels. To assess our methods, we trained models to predict 10 genetic mutations on the H &E images of the TCGA-LUAD dataset. AUROC and F1 score are reported, while we demonstrate that F1 score may be a more suitable metric for multi-task learning with imbalanced labels. It is shown that our proposed trimming strategy combined with sequential learning could improve the predictions on all of the mutations compared with other multi-task learning approaches. Also, we investigated the application of continual learning.
H. Akrami—Work done as intern at Merck & Co., Inc., Rahway, NJ, USA.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arpit, D., et al.: A closer look at memorization in deep networks. In: International Conference on Machine Learning, pp. 233–242. PMLR (2017)
Ciga, O., Xu, T., Martel, A.L.: Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022)
Coudray, N., et al.: Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24(10), 1559–1567 (2018)
Douillard, A., Chen, Y., Dapogny, A., Cord, M.: PLOP: learning without forgetting for continual semantic segmentation. arXiv preprint arXiv:2011.11390 (2020)
Fu, Y., et al.: Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat. Cancer 1(8), 800–810 (2020)
Graham, S., Vu, Q.D., Jahanifar, M., Minhas, F., Snead, D., Rajpoot, N.: One model is all you need: multi-task learning enables simultaneous histology image segmentation and classification. arXiv preprint arXiv:2203.00077 (2022)
Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21271–21284 (2020)
Jung, H., Ju, J., Jung, M., Kim, J.: Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122 (2016)
Kather, J.N., et al.: Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Cancer 1(8), 789–799 (2020)
Kim, Y., Kim, J.M., Akata, Z., Lee, J.: Large loss matters in weakly supervised multi-label classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14156–14165 (2022)
Li, J., et al.: A multi-resolution model for histopathology image classification and localization with multiple instance learning. Comput. Biol. Med. 131, 104253 (2021)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. arXiv preprint arXiv:1706.08840 (2017)
Mai, Z., Li, R., Kim, H., Sanner, S.: Supervised contrastive replay: revisiting the nearest class mean classifier in online class-incremental continual learning. arXiv preprint arXiv:2103.13885 (2021)
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)
Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong learning of human actions with deep neural network self-organization. Neural Netw. 96, 137–149 (2017)
Rusu, A.A., et al.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
Sener, O., Koltun, V.: Multi-task learning as multi-objective optimization. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 (2017)
Soltoggio, A.: Short-term plasticity as cause-effect hypothesis testing in distal reward learning. Biol. Cybern. 109(1), 75–94 (2015)
Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS ONE 15(6) (2020)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Akrami, H. et al. (2022). Sequential Multi-task Learning for Histopathology-Based Prediction of Genetic Mutations with Extremely Imbalanced Labels. In: Huo, Y., Millis, B.A., Zhou, Y., Wang, X., Harrison, A.P., Xu, Z. (eds) Medical Optical Imaging and Virtual Microscopy Image Analysis. MOVI 2022. Lecture Notes in Computer Science, vol 13578. Springer, Cham. https://doi.org/10.1007/978-3-031-16961-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-16961-8_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16960-1
Online ISBN: 978-3-031-16961-8
eBook Packages: Computer ScienceComputer Science (R0)
