Tumor Segmentation in Weakly Paired Anatomical and Functional MRI images with Multimodal Information Fusion

Multimodal magnetic resonance imaging (MRI) contains complementary information in anatomical and functional images that help the accurate diagnosis and treatment evaluation of lung cancers. Accurately segmenting tumor regions in each modality can help to obtain comprehensive and precise information. Existing multimodal segmentation methods are mostly used for images with strict registration, but it is difficult to achieve it for lung MRI images. It is challenging to obtain accurate tumor segmentation in lung anatomical and functional MRI images simultaneously. In this paper, we propose a method for lung tumor segmentation in weakly paired anatomical and functional MRI images. Firstly, we use domain adaptation to narrow the gap in features across different modalities, enabling the segmentation network to adapt to different modality data simultaneously. At the same time, we explore a two-stage multimodal co-attention mechanism to help the extraction and effective fusion of multi-modal information. We evaluate the proposed method on lung tumor segmentation with a clinical dataset of 90 chest MRI scans of non-small cell lung cancer (NSCLC). The results show that this method effectively improves the segmentation accuracy of each modality, the DSC of anatomical MRI is increased to 0.81±0.19, and the DSC of functional MRI is increased to 0.77±0.23, which is significantly improved compared with several multimodal tumor segmentation methods (p <0.05).