Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

CAF-AHGCN: context-aware attention fusion adaptive hypergraph convolutional network for human-interpretable prediction of gigapixel whole-slide image

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Predicting labels of gigapixel whole-slide images (WSIs) and localizing regions of interest (ROIs) with high precision are of great interest in computational pathology. The existing methods are mainly based on multi-instance learning (MIL) approach and its variants. However, such algorithms primarily treat the instances of slides as independent samples, which cannot effectively describe the relationship between instances in WSI. This would cause only a subset of high-score instances can be located, which is not applicable in clinical scenarios. Context-aware attention fusion adaptive hypergraph convolution network (CAF-AHGCN) is proposed to adaptively establish the local and global topology of cropped image patches spatially arranged in the slides. The framework combines hypergraph embedding representation with attention-based MIL pooling aggregation in a hierarchical feature fusion manner, which fully preserves high-order spatial structural correlations of the patches to make a slide-level prediction. Here, an adaptive hypergraph convolutional network is designed to dynamically adjust the correlation strength between the hypergraph nodes as the model goes deeper. Poly-1 loss and residual connection are also applied to prevent over-smoothing and improve the generalization ability of deep CAF-AHGCN model. We verified the superiority of CAF-AHGCN on two datasets, CAMELYON16 and TCGA-NSCLC. The results showed that the ACC, AUC and F1 score predicted by our model outperform other state-of-the-art algorithms on both datasets. The heat maps obtained by CAF-AHGCN are highly consistent with the pixel-wise annotated label of the WSI. The results show that CAF-AHGCN not only achieves high-accuracy label prediction for WSI, but also provides patch-wise human-interpretable features in ROI localization heatmaps. The outstanding performance of CAF-AHGCN framework provides a new perspective for future clinical applications of computer-aided diagnosis and intelligent systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Metastatic lymph node data are publicly available from the CAMELYON16 (https://camelyon17.grand-challenge.org/Data). The TCGA diagnostic whole-slide data (TCGA-NSCLC) is available from the NIH genomic data commons (https://portal.gdc.cancer.gov).

References

  1. Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009)

    Article  Google Scholar 

  2. Marini, N., Marchesin, S., Otálora, S., Wodzinski, M., Caputo, A., Van Rijthoven, M., Aswolinskiy, W., Bokhorst, J.-M., Podareanu, D., Petters, E., et al.: Unleashing the potential of digital pathology data by training computer-aided diagnosis models without human annotations. NPJ Digital Med 5, 1–18 (2022)

    Article  Google Scholar 

  3. Wu, J., Zhang, R., Gong, T., Bao, X., Gao, Z., Zhang, H., Wang, C., Li, C.: A precision diagnostic framework of renal cell carcinoma on whole-slide images using deep learning. In: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, pp 2104–2111 (2021).

  4. Sankarapandian, S., Kohn, S., Spurrier, V., Grullon, S., Soans, R.E., Ayyagari, K.D., Chamarthi, R.V., Motaparthi, K., Lee, J.B., Shon, W., et al.: A pathology deep learning system capable of triage of melanoma specimens utilizing dermatopathologist consensus as ground truth. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 629–638 (2021).

  5. Hsu, W.-W., Wu, Y., Hao, C., Hou, Y.-L., Gao, X., Shao, Y., Zhang, X., He, T., Tai, Y.: A computer-aided diagnosis system for breast pathology: a deep learning approach with model interpretability from pathological perspective. arXiv preprint arXiv:210802656 (2021).

  6. Li, X., Yang, B., Chen, T., Gao, Z., & Li, H.: Multiple instance learning-based two-stage metric learning network for whole slide image classification. Visual Comp., pp 1–16 (2023). https://doi.org/10.1007/s00371-023-03131-2.

  7. Kanavati, F., Ichihara, S., Rambeau, M., Iizuka, O., Arihiro, K., Tsuneki, M.: Deep learning models for gastric signet ring cell carcinoma classification in whole slide images. Technol. Cancer Res. Treat. 20, 15330338211027900 (2021)

    Article  Google Scholar 

  8. Li, Z., Zhang, J., Tan, T., Teng, X., Sun, X., Zhao, H., Liu, L., Xiao, Y., Lee, B., Li, Y., et al.: Deep learning methods for lung cancer segmentation in whole-slide histopathology images—the acdc@ lunghp challenge 2019. IEEE J. Biomed. Health Inform. 25, 429–440 (2020)

    Article  Google Scholar 

  9. Wang, S., Zhu, Y., Yu, L., Chen, H., Lin, H., Wan, X., Fan, X., Heng, P.-A.: RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification. Med. Image Anal. 58, 101549 (2019)

    Article  Google Scholar 

  10. Xu, G., Song, Z., Sun, Z., Ku, C., Yang, Z., Liu, C., Wang, S., Ma, J., Xu, W.: Camel: A weakly supervised learning framework for histopathology image segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 10682–10691 (2019).

  11. Lerousseau, M., Vakalopoulou, M., Classe, M., Adam, J., Battistella, E., Carré, A., Estienne, T., Henry, T., Deutsch, E., Paragios, N.: Weakly supervised multiple instance learning histopathological tumor segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, pp 470–479 (2020).

  12. Thandiackal, K., Chen, B., Pati, P., Jaume, G., Williamson, D.F., Gabrani, M., Goksel, O.: Differentiable Zooming for Multiple Instance Learning on Whole-Slide Images. arXiv preprint arXiv:220412454 (2022).

  13. Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 14318–14328 (2021).

  14. Deng, R., Cui, C., Remedios, L.W., Bao, S., Womick, R.M., Chiron, S., Li, J., Roland, J.T., Lau, K.S., Liu, Q., et al.: Cross-scale attention guided multi-instance learning for Crohn’s disease diagnosis with pathological images. In: International Workshop on Multiscale Multimodal Medical Imaging. Springer, Cham, pp 24–33 (2022).

  15. Ahmad, N., Asghar, S., Gillani, S.A.: Transfer learning-assisted multi-resolution breast cancer histopathological images classification. Vis. Comput. 38(8), 2751–2770 (2022)

    Article  Google Scholar 

  16. Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019)

    Article  Google Scholar 

  17. Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coupland, S.E., Zheng, Y.: DTFD-MIL: double-tier featuredistillation multiple instance learning for histopathology whole slide image classification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 18802–18812 (2022).

  18. Wang, X., Chen, H., Gan, C., Lin, H., Dou, Q., Huang, Q., Cai, M., Heng, P.-A.: Weakly supervised learning for whole slide lung cancer image classification. IEEE Trans. Cybern. 50(9), 3950–3962 (2020)

    Article  Google Scholar 

  19. Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et al.: Transmil: transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural. Inf. Process. Syst. 34, 2136–2147 (2021)

    Google Scholar 

  20. Li, W., Nguyen, V.-D., Liao, H., Wilder, M., Cheng, K., Luo, J.: Patch transformer for multi-tagging whole slide histopathology images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp 532–540 (2019).

  21. Zheng, Y., Li, J., Shi, J., Xie, F., Jiang, Z.: Kernel Attention Transformer (KAT) for Histopathology Whole Slide Image Classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, pp 283–292 (2022).

  22. Kipf, T.N., Welling, M,: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:160902907 (2016).

  23. Lu, W., Graham, S., Bilal, M., Rajpoot, N., Minhas, F.: Capturing cellular topology in multi-gigapixel pathology images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp 260–261 (2020).

  24. Chen, R.J., Lu, M.Y., Shaban, M., Chen, C., Chen, T.Y., Williamson, D.F., Mahmood, F.: Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 339–349 (2021).

  25. Guan, Y., Zhang, J., Tian, K., Yang, S., Dong, P., Xiang, J., Yang, W., Huang. J., Zhang, Y., Han, X.: Node-aligned graph convolutional network for whole-slide image representation and classification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 18813–18823 (2022).

  26. Ahmedt-Aristizabal, D., Armin, M.A., Denman, S., Fookes, C.,Petersson, L.: A survey on graph-based deep learning for computational histopathology. Comp. Med. Imaging Graphics. 95, 102027 (2021).

  27. Abel, R., Benami, I., Louzoun, Y.: Topological based classification using graph convolutional networks. arXiv preprint arXiv:191106892 (2019).

  28. Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomed. Eng. 5, 555–570 (2021)

    Article  Google Scholar 

  29. Cheng, S., Liu, S., Yu, J., Rao, G., Xiao, Y., Han, W., Zhu, W., Lv, X., Li, N., Cai, J., et al.: Robust whole slide image analysis for cervical cancer screening using deep learning. Nat. Commun. 12, 1–10 (2021)

    Article  Google Scholar 

  30. Kanavati, F., Toyokawa, G., Momosaki, S., Rambeau, M., Kozuma, Y., Shoji, F., Yamazaki, K., Takeo, S., Iizuka, O., Tsuneki, M.: Weakly-supervised learning for lung carcinoma classification using deep learning. Sci. Rep. 10, 1–11 (2020)

    Article  Google Scholar 

  31. Ianni, J.D., Soans, R.E., Sankarapandian, S., Chamarthi, R.V., Ayyagari, D., Olsen, T.G., Bonham, M.J., Stavish, C.C., Motaparthi, K., Cockerell, C.J., et al.: Tailored for real-world: a whole slide image classification system validated on uncurated multi-site data emulating the prospective pathology workload. Sci. Rep. 10, 1–12 (2020)

    Article  Google Scholar 

  32. Xiang, T., Song, Y., Zhang, C., Liu, D., Chen, M., Zhang, F.,Huang, H., O’ Donnell, L., Cai, W.: DSNet: A dual-stream framework for weakly-supervised gigapixel pathology image analysis. IEEE Trans. Med. Imaging. 41(8), 2180–2190 (2022).

  33. Marini, N., Otálora, S., Ciompi, F., Silvello, G., Marchesin, S., Vatrano, S., Buttafuoco, G., Atzori, M., Müller, H.: Multi-scale task multiple instance learning for the classification of digital pathology images with global annotations. In: MICCAI Workshop on computational pathology. PMLR, pp 170–181 (2021)

  34. Hou, W., Yu, L., Lin, C., Huang, H., Yu, R., Qin, J., Wang, L.: H2-MIL: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In: Proceedings of the AAAI conference on artificial intelligence. pp 933–941 (2022).

  35. Chen, R.J., Lu, M.Y., Weng, W.-H., Chen, T.Y., Williamson, D.F., Manz,T., Shady, M., Mahmood, F.: Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4015–4025 (2021).

  36. Li, H., Yang, F., Zhao, Y., Xing, X., Zhang, J., Gao, M., Huang, J., Wang, L., Yao, J.: DT-MIL: deformable transformer for multi-instance learning on histopathological image. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, pp 206–216 (2021)

  37. Zhao, Y., Lin, Z., Sun, K., Zhang, Y., Huang, J., Wang, L., Yao, J.: SETMIL: Spatial encoding transformer-based multiple instance learning for pathological image analysis. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 66–76 (2022).

  38. Sureka, M., Patil, A., Anand, D., Sethi, A.: Visualization for histopathology images using graph convolutional neural networks. In: 2020 IEEE 20th International conference on bioinformatics and bioengineering (BIBE). IEEE, pp 331–335 (2020).

  39. Jaume, G., Pati, P., Bozorgtabar, B., Foncubierta, A., Anniciello, A.M., Feroce, F., Rau, T., Thiran, J.-P., Gabrani, M., Goksel, O.: Quantifying explainers of graph neural networks in computational pathology. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8106–8116 (2021).

  40. Zhou, Y., Graham, S., Alemi Koohbanani, N., Shaban, M., Heng, P-A, Rajpoot, N. Cgc-net: Cell graph convolutional network for grading of colorectal cancer histology images. In: Proceedings of the IEEE/CVF international conference on computer vision workshops (2019).

  41. Liang, M., Chen, Q., Li, B., Wang, L., Wang, Y., Zhang, Y., Wang, R., Jiang, X., Zhang, C.: Interpretable classification of pathology whole-slide images using attention based context-aware graph convolutional neural network. Comput. Methods Programs Biomed. 229, 107268 (2023)

    Article  Google Scholar 

  42. Zheng, Y., Gindra, R.H., Green, E.J., Burks, E.J., Betke, M., Beane, J.E., Kolachalama, V.B.: A graph-transformer for whole slide image classification. arXiv preprint arXiv:220509671 (2022).

  43. Zhao, Y., Yang, F., Fang, Y., Liu, H., Zhou, N., Zhang, J., Sun, J., Yang, S., Menze, B., Fan, X., et al.: Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4837–4846 (2020).

  44. Gao, Y., Zhang, Z., Lin, H., et al.: Hypergraph learning: methods and practices. IEEE Trans. Pattern Anal. Mach. Intell. 44(5), 2548–2566 (2020)

    Google Scholar 

  45. Giraldo, J. H., Scarrica, V., Staiano, A., et al.: Hypergraph convolutional networks for weakly-supervised semantic segmentation//2022 IEEE International Conference on Image Processing (ICIP). IEEE, pp 16–20 (2020).

  46. Bai, S., Zhang, F., Torr, P.H.S.: Hypergraph convolution and hypergraph attention. Pattern Recogn. 110, 107637 (2021)

    Article  Google Scholar 

  47. Huang, J., Huang, X., Yang. J.: Residual enhanced multi-hypergraph neural network. In: 2021 IEEE international conference on image processing (ICIP). IEEE, pp 3657–3661 (2021).

  48. Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International conference on machine learning. PMLR, pp 2127–2136 (2018).

  49. Leng, Z., Tan, M., Liu, C., Cubuk, E.D., Shi, X., Cheng, S., Anguelov, D.: PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions. arXiv preprint arXiv:2204.12511 (2022).

  50. Qu, L., Luo, X., Liu, S., Wang, M., Song, Z.: Dgmil: Distribution guided multiple instance learning for whole slide image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 24–34). Cham: Springer Nature Switzerland (2022).

  51. Bontempo, G., Lumetti, L., Porrello, A., Bolelli, F., Calderara, S., Ficarra, E.: Buffer-MIL: Robust Multi-instance Learning with a Buffer-based Approach. In: International Conference on Image Analysis and Processing (pp. 1–12). Cham: Springer Nature Switzerland (2023).

Download references

Acknowledgements

This work is supported in part by Natural Science Foundation of Shanxi Province under Grant 202303021211014 and 20210302123411, National Natural Science Foundation of China under Grant 11804209, Research Project Supported by Shanxi Scholarship Council of China 2023-010, the Natural Science Foundation of Shanxi Province under Grant 202203021222015.

Author information

Author notes

  1. Meiyan Liang and Xing Jiang have contributed equally.

Authors and Affiliations

  1. School of Physics and Electronic Engineering, Shanxi University, Taiyuan, 030006, China

    Meiyan Liang, Xing Jiang & Qinghui Chen

  2. ShanxiBeike Biotechnology Co., Ltd., Taiyuan, 030032, China

    Meiyan Liang

  3. School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China

    Jie Cao & Yuejin Zhao

  4. Department of Pathology, Shanxi Bethune Hospital, Shanxi Academy of Medical Sciences, Tongji Shanxi Hospital, Third Hospital of Shanxi Medical University, Taiyuan, 030032, China

    Lin Wang

  5. Department of Rehabilitation Treatment, Shanxi Rongjun Hospital, Taiyuan, 030000, China

    Bo Li

  6. Beijing Key Laboratory for Terahertz Spectroscopy and Imaging, Ministry of Education, Key Laboratory of Terahertz, Optoelectronics, Capital Normal University, Beijing, 100048, China

    Cunlin Zhang

Authors
  1. Meiyan Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xing Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qinghui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cunlin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yuejin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ML analyzed conceptualization, methodology, software, visualization, writing—review & editing and writing—original draft. Xing Jiang performed conceptualization, methodology, software and writing—original draft. JC provided conceptualization and writing—review &editing. LW: developed visualization, investigation and interpretation. QC approved visualization and software. BL carried out visualization, investigation and interpretation. CZ prepared writing—review & editing. YZ conducted conceptualization, funding and supervision.

Corresponding authors

Correspondence to Meiyan Liang, Jie Cao or Lin Wang.

Ethics declarations

Conflict of interest

The authors declared that they have no conflicts of interest in this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, M., Jiang, X., Cao, J. et al. CAF-AHGCN: context-aware attention fusion adaptive hypergraph convolutional network for human-interpretable prediction of gigapixel whole-slide image. Vis Comput 40, 8747–8765 (2024). https://doi.org/10.1007/s00371-024-03269-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-024-03269-7

Keywords

Search

Navigation