Abstract
Selecting regions of interest (ROIs) in whole-slide histology images (WSIs) is a crucial step for spatial molecular profiling. As a general practice, pathologists manually select ROIs within each WSI based on morphological tumor markers to guide spatial profiling, which can be inconsistent and subjective. To enhance reproducibility and avoid inter-pathologist variability, we introduce a novel immune-guided end-to-end pipeline to automate the ROI selection in multiplex immunofluorescence (mIF) WSIs stained with three cell markers (Syto13, CD45, PanCK). First, we estimate immune infiltration (CD45\(^+\) expression) scores at the grid level in each WSI. Then, we incorporate the Pathology Language and Image Pre-Training (PLIP) foundational model to extract features from each grid and further select a subset of grids representative of the whole slide that comparatively matches pathologists’ assessment. Further, we implement state-of-the-art detection models for ROI detection in each grid, incorporating learning from pathologists’ ROI selection. Our study shows a significant correlation between our automated method and pathologists’ ROI selection across five different types of carcinomas, as evidenced by a significant Spearman’s correlation coefficient (> 0.785, p < 0.001), substantial inter-rater agreement (Cohen’s \(\kappa >\) 0.671), and the ability to replicate the ROI selection made by independent pathologists with excellent average performance (0.968 precision and 0.991 mean average precision at a 0.5 intersection-over-union). By minimizing manual intervention, our solution provides a flexible framework that potentially adapts to various markers, thus enhancing the efficiency and accuracy of digital pathology analyses.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, M., Zhang, B., Topatana, W., Cao, J., Zhu, H., Juengpanich, S., Mao, Q., Yu, H., Cai, X.: Classification and mutation prediction based on histopathology H &E images in liver cancer using deep learning. npj Precision Oncology 4(1), 14 (6 2020)
Chollet, F.: Xception: Deep Learning with Depthwise Separable Convolutions (10 2016)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (10 2020)
Gooden, M.J., de Bock, G.H., Leffers, N., Daemen, T., Nijman, H.W.: The prognostic influence of tumour-infiltrating lymphocytes in cancer: a systematic review with meta-analysis. British journal of cancer 105(1), 93–103 (2011)
He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778. IEEE (6 2016). https://doi.org/10.1109/CVPR.2016.90
Hossain, M.S., Shahriar, G.M., Syeed, M.M., Uddin, M.F., Hasan, M., Shivam, S., Advani, S.: Region of interest (ROI) selection using vision transformer for automatic analysis using whole slide images. Scientific Reports 13(1) (12 2023). https://doi.org/10.1038/s41598-023-38109-6
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual-language foundation model for pathology image analysis using medical Twitter. Nature medicine 29(9), 2307–2316 (9 2023). https://doi.org/10.1038/s41591-023-02504-3
Jocher, G.: Ultralytics yolov5 (2020). https://doi.org/10.5281/zenodo.3908559, https://github.com/ultralytics/yolov5
Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics yolov8 (2023), https://github.com/ultralytics/ultralytics
Korkut, S., Erkan, C., Aksoy, S.: On the benefits of region of interest detection for whole slide image classification. In: Tomaszewski, J.E., Ward, A.D. (eds.) Medical Imaging 2023: Digital and Computational Pathology. p. 23. SPIE (4 2023). https://doi.org/10.1117/12.2654193
Li, R., Huang, J.: Fast Regions-of-Interest Detection in Whole Slide Histopathology Images. pp. 120–127 (2015). https://doi.org/10.1007/978-3-319-28194-0_15
Lin, J.R., Wang, S., Coy, S., Chen, Y.A., Yapp, C., Tyler, M., Nariya, M.K., Heiser, C.N., Lau, K.S., Santagata, S., et al.: Multiplexed 3d atlas of state transitions and immune interaction in colorectal cancer. Cell 186(2), 363–381 (2023)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object Detection (8 2017)
Nagarkar, D.B., Mercan, E., Weaver, D.L., Brunyé, T.T., Carney, P.A., Rendi, M.H., Beck, A.H., Frederick, P.D., Shapiro, L.G., Elmore, J.G.: Region of interest identification and diagnostic agreement in breast pathology (9 2016). https://doi.org/10.1038/modpathol.2016.85
Nugaliyadde, A., Wong, K.W., Parry, J., Sohel, F., Laga, H., Somaratne, U.V., Yeomans, C., Foster, O.: RCNN for Region of Interest Detection in Whole Slide Images. pp. 625–632 (2020). https://doi.org/10.1007/978-3-030-63823-8_71
Qaiser, T., Lee, C.Y., Vandenberghe, M., Yeh, J., Gavrielides, M.A., Hipp, J., Scott, M., Reischl, J.: Usability of deep learning and H &E images predict disease outcome-emerging tool to optimize clinical trials. npj Precision Oncology 6(1), 37 (6 2022). https://doi.org/10.1038/s41698-022-00275-7
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6), 1137–1149 (6 2017). https://doi.org/10.1109/TPAMI.2016.2577031
Salgado, R., Denkert, C., Demaria, S., Sirtaine, N., Klauschen, F., Pruneri, G., Wienert, S., Van den Eynden, G., Baehner, F.L., Pénault-Llorca, F., et al.: The evaluation of tumor-infiltrating lymphocytes (tils) in breast cancer: recommendations by an international tils working group 2014. Annals of oncology 26(2), 259–271 (2015)
Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition (9 2014)
Yu, J., Wu, Z., Mayer, A., Trevino, A., Zou, J.: A multi-granularity approach to similarity search in multiplexed immunofluorescence images (05 2024)
Acknowledgments
This work was supported by the Patient Mosaic Project, team members listed in the Supplementary, at The University of Texas MD Anderson Cancer Center; we thank Drs. Sabitha Prabhakaran and Neus Bota for their assistance. Patient Mosaic is supported by generous philanthropic contributions from the Albert and Margaret Alkek Foundation, among others. This project is funded by the generous support of Lyda-Hill Philanthropies.
Author information
Authors and Affiliations
Consortia
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gautam, T. et al. (2024). Immune-Guided AI for Reproducible Regions of Interest Selection in Multiplex Immunofluorescence Pathology Imaging. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15004. Springer, Cham. https://doi.org/10.1007/978-3-031-72083-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-72083-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72082-6
Online ISBN: 978-3-031-72083-3
eBook Packages: Computer ScienceComputer Science (R0)
