Abstract
The ability to translate medical images across different modalities is crucial for synthesizing missing data and aiding in clinical diagnosis. However, existing learning-based techniques have limitations when it comes to capturing cross-modal and global features. These techniques are often tailored to specific pairs of modalities, limiting their practical utility, especially considering the variability of missing modalities in different cases. In this study, we introduce MedPrompt, a multi-task framework designed to efficiently translate diverse modalities. Our framework incorporates the Self-adaptive Prompt Block, which dynamically guides the translation network to handle different modalities effectively. To encode the cross-modal prompt efficiently, we introduce the Prompt Extraction Block and the Prompt Fusion Block. Additionally, we leverage the Transformer model to enhance the extraction of global features across various modalities. Through extensive experimentation involving five datasets and four pairs of modalities, we demonstrate that our proposed model achieves state-of-the-art visual quality and exhibits excellent generalization capability. The results highlight the effectiveness and versatility of MedPrompt in addressing the challenges associated with cross-modal medical image translation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brody, H.: Medical imaging. Nature 502(7473), S81–S81 (2013)
Wang, S., Wang, H., Cheung, A.C., Shen, Y., Gan, M.: Ensemble of 3d densely connected convolutional network for diagnosis of mild cognitive impairment and alzheimer’s disease. In: Deep Learning Applications, pp. 53–73 (2020)
Wang, S.Q., He, J.H.: Variational iteration method for a nonlinear reaction-diffusion process. Int. J. Chem. React. Eng. 6(1) (2008)
Zhou, T., Chen, X., Shen, Y., Nieuwoudt, M., Pun, C.M., Wang, S.: Generative ai enables eeg data augmentation for alzheimer’s disease detection via diffusion model. In: ISPCE-ASIA, pp. 1–6 (2023)
Gong, C., Jing, C., Chen, X., Pun, C.M., Huang, G., Saha, A., Nieuwoudt, M., Li, H.X., Hu, Y., Wang, S.: Generative ai for brain image computing and brain network computing: A review. Front. Neurosci. 17, 1203104 (2023)
Lei, B., Zhu, Y., Yu, S., Hu, H., Xu, Y., Yue, G., Wang, T., Zhao, C., Chen, S., Yang, P., et al.: Multi-scale enhanced graph convolutional network for mild cognitive impairment detection. Pattern Recogn. 134, 109106 (2023)
Huang, G., Chen, X., Shen, Y., Wang, S.: Mr image super-resolution using wavelet diffusion for predicting alzheimer’s disease. In: BI, pp. 146–157 (2023)
Chen, X., Lei, B., Pun, C.M., Wang, S.: Brain diffuser: An end-to-end brain image to brain network pipeline. In: PRCV, pp. 16–26 (2023)
You, S., Lei, B., Wang, S., Chui, C.K., Cheung, A.C., Liu, Y., Gan, M., Wu, G., Shen, Y.: Fine perceptive gans for brain mr image super-resolution in wavelet domain. In: IEEE Transactions on Neural Networks and Learning Systems (2022)
Hu, B., Zhan, C., Tang, B., Wang, B., Lei, B., Wang, S.Q.: 3-d brain reconstruction by hierarchical shape-perception network from a single incomplete image. In: IEEE Transactions on Neural Networks and Learning Systems (2023)
Wang, S.Q., Li, H.X.: Bayesian inference based modelling for gene transcriptional dynamics by integrating multiple source of knowledge. BMC Syst. Biol. 6(1), 1–13 (2012)
Lei, B., Zhang, Y., Liu, D., Xu, Y., Yue, G., Cao, J., Hu, H., Yu, S., Yang, P., Wang, T., et al.: Longitudinal study of early mild cognitive impairment via similarity-constrained group learning and self-attention based sbi-lstm. Knowl.-Based Syst. 254, 109466 (2022)
Huang, Y., Shao, L., Frangi, A.F.: Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning. IEEE Trans. Med. Imaging 37(3), 815–827 (2017)
Dong, Y., Chen, X., Shen, Y., Ng, M.K.P., Qian, T., Wang, S.: Multi-modal mood reader: Pre-trained model empowers cross-subject emotion recognition. ArXiv (2024)
Zhou, Z., Huo, Y., Huang, G., Zeng, A., Chen, X., Huang, L., Li, Z.: Qean: quaternion-enhanced attention network for visual dance generation. In: The Visual Computer, pp. 1–13 (2024)
Nie, D., Trullo, R., Lian, J., Petitjean, C., Ruan, S., Wang, Q., Shen, D.: Medical image synthesis with context-aware generative adversarial networks. In: MICCAI, pp. 417–425. Springer (2017)
Dar, S.U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., Cukur, T.: Image synthesis in multi-contrast mri with conditional generative adversarial networks. IEEE Trans. Med. Imaging 38(10), 2375–2388 (2019)
Dalmaz, O., Yurt, M., Çukur, T.: Resvit: Residual vision transformers for multimodal medical image synthesis. IEEE Trans. Med. Imaging 41(10), 2598–2614 (2022)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words. In: Transformers for Image Recognition at Scale. ICLR (2021)
Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: ECCV, pp. 709–727. Springer (2022)
Gao, F., Wu, T., Chu, X., Yoon, H., Xu, Y., Patel, B.: Deep residual inception encoder-decoder network for medical imaging synthesis. IEEE J. Biomed. Health Inform. 24(1), 39–49 (2019)
Hu, S., Lei, B., Wang, S., Wang, Y., Feng, Z., Shen, Y.: Bidirectional mapping generative adversarial networks for brain mr to pet synthesis. IEEE Trans. Med. Imaging 41(1), 145–157 (2021)
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, pp. 2790–2799. PMLR (2019)
Victor, S., Albert, W., Colin, R., Stephen, B., Lintang, S., Zaid, A., Antoine, C., Arnaud, S., Arun, R., Manan, D., et al.: Multitask prompted training enables zero-shot task generalization. In: ICLR (2022)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS 33, 1877–1901 (2020)
Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2021)
Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi-modal prompt learning. In: CVPR, pp. 19113–19122 (2023)
Sohn, K., Chang, H., Lezama, J., Polania, L., Zhang, H., Hao, Y., Essa, I., Jiang, L.: Visual prompt tuning for generative transfer learning. In: CVPR, pp. 19840–19851 (2023)
Chen, X., Cun, X., Pun, C.M., Wang, S.: Shadocnet: Learning spatial-aware tokens in transformer for document shadow removal. In: ICASSP, pp. 1–5 (2023)
Luo, S., Chen, X., Chen, W., Li, Z., Wang, S., Pun, C.M.: Devignet: High-resolution vignetting removal via a dual aggregated fusion transformer with adaptive channel expansion. In: AAAI, pp. 4000–4008 (2024)
Li, Z., Chen, X., Guo, S., Wang, S., Pun, C.M.: Wavenhancer: Unifying wavelet and transformer for image enhancement. J. Comput. Sci. Technol. 39(2), 336–345 (2024)
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: CVPR, pp. 5728–5739 (2022)
Li, Z., Chen, X., Pun, C.M., Cun, X.: High-resolution document shadow removal via a large-scale real-world dataset and a frequency-aware shadow erasing net. In: ICCV, pp. 12449–12458 (2023)
Jiang, Y., Chen, X., Pun, C.M., Wang, S., Feng, W.: Mfdnet: Multi-frequency deflare network for efficient nighttime flare removal. ArXiv (2024)
Zuo, Q., Lei, B., Wang, S., Liu, Y., Wang, B., Shen, Y.: A prior guided adversarial representation learning and hypergraph perceptual network for predicting abnormal connections of alzheimer’s disease. arXiv preprint arXiv:2110.09302 (2021)
Hu, S., Shen, Y., Wang, S., Lei, B.: Brain mr to pet synthesis via bidirectional generative adversarial network. In: MICCAI, pp. 698–707. Springer (2020)
Thummerer, A., van der Bijl, E., Galapon Jr, A., Verhoeff, J.J., Langendijk, J.A., Both, S., van den Berg, C.N.A., Maspero, M.: Synthrad2023 grand challenge dataset: Generating synthetic ct for radiotherapy. In: Medical Physics (2023)
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Biobank, U.: Protocol for a large-scale prospective epidemiological resource (2007)
Acknowledgement
This work was supported in part by the National Natural Science Foundations of China under Grant 62172403 and 12326614, the Distinguished Young Scholars Fund of Guangdong under Grant 2021B1515020019, the Excellent Young Scholars of Shenzhen under Grant RCYX20200714114641211, in part by the Science and Technology Development Fund, Macau SAR, under Grant 0141/2023/RIA2 and 0193/2023/RIA3. This research has been conducted using the UK Biobank Resource under Application Number No.75310. This work was performed at SICC which is supported by SKL-IOTSC, University of Macau.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, X., Luo, S., Pun, CM., Wang, S. (2025). MedPrompt: Cross-modal Prompting for Multi-task Medical Image Translation. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2024. Lecture Notes in Computer Science, vol 15044. Springer, Singapore. https://doi.org/10.1007/978-981-97-8496-7_5
Download citation
DOI: https://doi.org/10.1007/978-981-97-8496-7_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-8495-0
Online ISBN: 978-981-97-8496-7
eBook Packages: Computer ScienceComputer Science (R0)