Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

A spatial squeeze and multimodal feature fusion attention network for multiple tumor segmentation from PET–CT Volumes

Published: 01 May 2023 Publication History
  • Get Citation Alerts
  • Abstract

    Tumor segmentation is a key step in computer-aided diagnosis. The PET–CT co-segmentation method combines the high sensitivity of PET images and the anatomical information of CT images. For whole-body multiple tumors, such as soft tissue sarcoma, lymphoma, etc., due to the different lesion location and size, it is necessary to segment the tumor area according to the whole body anatomical information. How to effectively leverage whole-body contextual information and the fusion of multimodal information is the key to the problem. To address this issue, we propose a spatial squeeze and multimodal feature fusion attention network for whole-body multiple tumors segmentation based on PET–CT volumes. Our proposed method consists of two parts, a Coronal-Spatial Squeeze Attention Extraction Network (CSAE-Net) and a Precise PET–CT Fusion Attention Segmentation Network (PFAS-Net), respectively. In CSAE-Net, we squeeze a 3D PET–CT volume along the coronal plane into m 2D images, and obtain 3D Coronal Spatial Squeeze Attention Volume based on these 2D images. In PFAS-Net, the input is a 2D axial PET–CT slice, and the previously obtained coronal spatial squeeze attention map is used to guide the segmentation. Moreover, a Multimodal Fusion Attention (MFA) module is proposed to fuse the metabolic information of PET and the anatomical information of CT. We perform experiments on PET–CT datasets of two whole-body multiple tumors, Soft Tissue Sarcoma (STS) and Lymphoma. The results show that our proposed method improved Dice values by 8.03% in STS and 1.74% in Lymphoma. Also the visualization results show that our proposed method is able to suppress high-uptake regions of normal tissues.

    References

    [1]
    Bi L., Fulham M., Li N., Liu Q., Song S., Dagan Feng D., Kim J., Recurrent feature fusion learning for multi-modality pet-ct tumor segmentation, Comput. Methods Programs Biomed. 203 (2021),.
    [2]
    Bi L., Kim J., Feng D., Fulham M., Multi-stage thresholded region classification for whole-body PET-CT lymphoma studies, in: Golland P., Hata N., Barillot C., Hornegger J., Howe R. (Eds.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014, Springer International Publishing, Cham, 2014, pp. 569–576,.
    [3]
    Blodgett T.M., Meltzer C.C., Townsend D.W., PET/CT: Form and function, Radiology 242 (2) (2007) 360–385,.
    [4]
    Cao Y., Xu J., Lin S., Wei F., Hu H., GCNet: Non-local networks meet squeeze-excitation networks and beyond, in: 2019 IEEE/CVF International Conference on Computer Vision Workshop, ICCVW, 2019, pp. 1971–1980,.
    [5]
    Chen C., Dou Q., Jin Y., Chen H., Qin J., Heng P.-A., Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion, in: Shen D., Liu T., Peters T.M., Staib L.H., Essert C., Zhou S., Yap P.-T., Khan A. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer International Publishing, Cham, 2019, pp. 447–456,.
    [6]
    Foster B., Bagci U., Mansoor A., Xu Z., Mollura D.J., A review on segmentation of positron emission tomography images, Comput. Biol. Med. 50 (2014) 76–96,.
    [7]
    Fu X., Bi L., Kumar A., Fulham M., Kim J., Multimodal spatial attention module for targeting multimodal PET-CT lung tumor segmentation, IEEE J. Biomed. Health Inf. (2021) 1,.
    [8]
    Gallamini A., Zwarthoed C., Borra A., Positron emission tomography (PET) in oncology, Cancers 6 (4) (2014) 1821–1889,.
    [9]
    Guo Z., Li X., Huang H., Guo N., Li Q., Deep learning-based image segmentation on multimodal medical imaging, IEEE Trans. Radiat. Plasma Med. Sci. 3 (2) (2019) 162–169,.
    [10]
    Hou Q., Zhou D., Feng J., Coordinate attention for efficient mobile network design, 2021, arXiv:2103.02907.
    [11]
    Hu X., Guo R., Chen J., Li H., Waldmannstetter D., Zhao Y., Li B., Shi K., Menze B., Coarse-to-fine adversarial networks and zone-based uncertainty analysis for NK/T-Cell lymphoma segmentation in CT/PET images, IEEE J. Biomed. Health Inf. 24 (9) (2020) 2599–2608,.
    [12]
    Hui C., Xiuying W., Jianlong Z., Stefan E., Yong Y., Dagan F., Michael F., Topology polymorphism graph for lung tumor segmentation in PET-CT images, Phys. Med. Biol. 60 (12) (2015) 4893–4914,.
    [13]
    Kingma D.P., Ba J., Adam: A method for stochastic optimization, in: 2015 International Conference for Learning Representations, ICLR, 2014, arXiv:1412.6980.
    [14]
    Kumar A., Fulham M., Feng D., Kim J., Co-learning feature fusion maps from PET-CT images of lung cancer, IEEE Trans. Med. Imaging 39 (1) (2020) 204–217,.
    [15]
    Li H., Jiang H., Li S., Wang M., Wang Z., Lu G., Guo J., Wang Y., DenseX-Net: An end-to-end model for lymphoma segmentation in whole-body PET/CT images, IEEE Access 8 (2020) 8004–8018,.
    [16]
    Lian C., Ruan S., Denœux T., Li H., Vera P., Spatial evidential clustering with adaptive distance metric for tumor segmentation in images, IEEE Trans. Biomed. Eng. 65 (1) (2018) 21–30,.
    [17]
    Lian C., Ruan S., Denoeux T., Li H., Vera P., Joint tumor segmentation in PET-CT images using co-clustering and fusion based on belief functions, IEEE Trans. Image Process. 28 (2) (2019) 755–766,.
    [18]
    Milletari F., Navab N., Ahmadi S.-A., V-Net: Fully convolutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference on 3D Vision (3DV), 2016, pp. 565–571,.
    [19]
    Nehmeh S., El-Zeftawy H., Greco C., Schwartz J., Erdi Y., Kirov A., Schmidtlein C., Gyau A., Larson S., Humm J., An iterative technique to segment PET lesions using a Monte Carlo based mathematical model, Med. Phys. 36 (10) (2009) 4803–4809.
    [20]
    Ronneberger O., Fischer P., Brox T., U-Net: Convolutional networks for biomedical image segmentation, in: Navab N., Hornegger J., Wells W.M., Frangi A.F. (Eds.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Springer International Publishing, Cham, 2015, pp. 234–241,.
    [21]
    Vallières M., Freeman C.R., Skamene S.R., Naqa I.E., A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities, Phys. Med. Biol. 60 (14) (2015) 5471–5496,.
    [22]
    Vinogradova K., Dibrov A., Myers G., Towards interpretable semantic segmentation via gradient-weighted class activation mapping, 2020, arXiv:2002.11434.
    [23]
    Wahl R.L., Jacene H., Kasamon Y., Lodge M.A., From RECIST to PERCIST: Evolving considerations for PET response criteria in solid tumors, J. Nucl. Med. 50 (Suppl 1) (2009) 122S–150S,.
    [24]
    Xu G., Cao H., Udupa J.K., Tong Y., Torigian D.A., DiSegNet: A deep dilated convolutional encoder-decoder architecture for lymph node segmentation on PET/CT images, Comput. Med. Imaging Graph. 88 (2021),.
    [25]
    Zhao X., Li L., Lu W., Tan S., Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network, Phys. Med. Biol. 64 (1) (2018),.
    [26]
    Zhong Z., Kim Y., Buatti J., Wu X., 3D alpha matting based co-segmentation of tumors on PET-CT images, in: Cardoso M.J., Arbel T., Gao F., Kainz B., van Walsum T., Shi K., Bhatia K.K., Peter R., Vercauteren T., Reyes M., Dalca A., Wiest R., Niessen W., Emmer B.J. (Eds.), Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment, Springer International Publishing, Cham, 2017, pp. 31–42,.
    [27]
    Zhong Z., Kim Y., Plichta K.A., Allen B.G., Zhou L., Buatti J.M., Wu X., Simultaneous cosegmentation of tumors in PET-CT images using deep fully convolutional networks, Med. Phys. 46 (2) (2018) 619–633,.
    [28]
    Zhong Z., Kim Y., Zhou L., Plichta K., Allen B., Buatti J., Wu X., 3D fully convolutional networks for co-segmentation of tumors on PET-CT images, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 228–231,.
    [29]
    Zhou T., Ruan S., Canu S., A review: Deep learning for medical image segmentation using multi-modality fusion, Array 3–4 (2019),.

    Index Terms

    1. A spatial squeeze and multimodal feature fusion attention network for multiple tumor segmentation from PET–CT Volumes
            Index terms have been assigned to the content through auto-classification.

            Recommendations

            Comments

            Information & Contributors

            Information

            Published In

            cover image Engineering Applications of Artificial Intelligence
            Engineering Applications of Artificial Intelligence  Volume 121, Issue C
            May 2023
            1624 pages

            Publisher

            Pergamon Press, Inc.

            United States

            Publication History

            Published: 01 May 2023

            Author Tags

            1. Biomedical segmentation
            2. PET–CT
            3. Deep learning
            4. Attention

            Qualifiers

            • Research-article

            Contributors

            Other Metrics

            Bibliometrics & Citations

            Bibliometrics

            Article Metrics

            • 0
              Total Citations
            • 0
              Total Downloads
            • Downloads (Last 12 months)0
            • Downloads (Last 6 weeks)0
            Reflects downloads up to 09 Aug 2024

            Other Metrics

            Citations

            View Options

            View options

            Get Access

            Login options

            Media

            Figures

            Other

            Tables

            Share

            Share

            Share this Publication link

            Share on social media