Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

MIGCL: Fake news detection with multimodal interaction and graph contrastive learning networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Due to the rapid growth of news containing multimedia elements, such as images in social networks, cross-modal learning is crucial for accurate fake news detection. Most previous approaches focus on embedding images and sentences independently into a shared embedding space by developing complex neural networks to coarsely fuse multimodal information. However, these approaches rarely seek fine-grained connections between images and sentences prior to performing multimodal fusion and lack the ability to understand complex intra- and intermodal relationships. In addition, previous studies have primarily concentrated on intra- and intermodal relationships within each sample, but interclass sample dynamics have been neglected. To address these issues, we propose a multimodal interaction and graph contrastive learning network (MIGCL) for fake news detection. The multimodal interaction network consists of cross-modal alignment and filtering mechanisms that take into account both locally fine-grained and comprehensive cross-modal interactions while also adaptively suppressing irrelevant cross-modal interactions. Moreover, we develop a hierarchical graph contrastive learning framework that employs fully and self-supervised contrastive learning methods to investigate the intricate connections of intra- and intermodal representations. More precisely, unimodal graphs are constructed at the intramodal level to explore the authenticity information contained in the intra- and interclass samples of a particular modality. At the intermodal level, multimodal graphs are constructed to capture the correlations between intra- and interclass cross-modal samples. Furthermore, we enhance the robustness of the model feature representation by applying perturbations to the graph structure. The proposed MIGCL achieves superior performance on three benchmark datasets, indicating the efficacy of our approach.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data can be available at https://github.com/wangzhuang1911/Weibo-dataset(Weibo datasets), http://www.multimediaeval.org/mediaeval2016/verifyingmultimediause/(Twitter datasets), https://figshare.com/articles/PHEME_dataset_of_rumours_and_non-rumours/

4010,619(Pheme datasets)

Code availability

The code are available from the corresponding author on reasonable request.

References

  1. Notarmuzi D, Castellano C, Flammini A, Mazzilli D, Radicchi F (2022) Universality, criticality and complexity of information propagation in social media. Nat Commun 13(1):1–8

    Article  MATH  Google Scholar 

  2. Zhang X, Ghorbani AA (2020) An overview of online fake news: Characterization, detection, and discussion. Inf Process Manage 57(2):102025

    Article  MATH  Google Scholar 

  3. Capuano N, Fenza G, Loia V, Nota FD (2023) Content-based fake news detection with machine and deep learning: A systematic review. Neurocomputing 530:91–103

    Article  MATH  Google Scholar 

  4. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimedia 19(3):598–608

    Article  MATH  Google Scholar 

  5. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia, pp 795–816

  6. Singhal S, Shah RR, Chakraborty T, Kumaraguru P, Satoh Si (2019) Spotfake: A multi-modal framework for fake news detection. In: 2019 IEEE fifth International Conference on Multimedia Big Data (BigMM), pp 39–47

  7. Singhal S, Kabra A, Sharma M, Shah RR, Chakraborty T, Kumaraguru P (2020) Spotfake+: A multimodal framework for fake news detection via transfer learning. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence(AAAI-20), pp 13915–13916

  8. Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, et al. (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th Acm Sigkdd International Conference on Knowledge Discovery & Data Mining, pp 849–857

  9. Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference, pp 2915–2921

  10. Zhang H, Fang Q, Qian S, Xu C (2019) Multi-modal knowledge-aware event memory network for social media rumor detection. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 1942–1951

  11. Zhou X, Wu J, Zafarani R (2020) Safe: Similarity-aware multi-modal fake news detection. In: Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, pp 354–367

  12. Wu Y, Zhan P, Zhang Y, Wang L, Xu Z (2021) Multimodal fusion with co-attention networks for fake news detection. In: The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), pp 2560–2569

  13. Qian S, Wang J, Hu J, Fang Q, Xu C (2021) Hierarchical multi-modal contextual attention network for fake news detection. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 153–162

  14. Cui L, Wang S, Lee D (2019) Same: Sentiment-aware multi-modal embedding for detecting fake news. In: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp 41–48

  15. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th International Conference on Learning Representations, pp 1–14

  16. You Y, Chen T, Sui Y, Chen T, Wang Z, Shen Y (2020) Graph contrastive learning with augmentations. Adv Neural Inf Process Syst 33:5812–5823

    MATH  Google Scholar 

  17. Ma J, Gao W, Mitra P, Kwon S, Jansen BJ, Wong K-F, et al. (2016) Detecting rumors from microblogs with recurrent neural networks. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence, pp 3818–3824

  18. Yu F, Liu Q, Wu S, Wang L, Tan T (2017) A convolutional approach for misinformation identification. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp 3901–3907

  19. Alsaif HF, Aldossari HD (2023) Review of stance detection for rumor verification in social media. Eng Appl Artif Intell 119:105801

    Article  MATH  Google Scholar 

  20. Shu K, Zhou X, Wang S, Zafarani R, Liu H (2019) The role of user profiles for fake news detection. In: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp 436–439

  21. Jarrahi A, Safari L (2023) Evaluating the effectiveness of publishers’ features in fake news detection on social media. Multimed Tools Appl 82(2):2913–2939

    Article  MATH  Google Scholar 

  22. Esteban-Bravo M, Vidal-Sanz JM (2024) Predicting the virality of fake news at the early stage of dissemination. Expert Syst Appl 248:123390

    Article  MATH  Google Scholar 

  23. Shahid W, Jamshidi B, Hakak S, Isah H, Khan WZ, Khan MK et al (2022) Detecting and mitigating the dissemination of fake news: Challenges and future research opportunities. IEEE Transact Computat Soc Syst 11(4):4649–4662

    Article  MATH  Google Scholar 

  24. Boididou C, Middleton SE, Jin Z, Papadopoulos S, Dang-Nguyen D-T, Boato G et al (2018) Verifying information with multimedia content on twitter. Multimed Tools Appl 77(12):15545–15571

    Article  Google Scholar 

  25. Qi P, Cao J, Yang T, Guo J, Li J (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp 518–527

  26. Cao J, Qi P, Sheng Q, Yang T, Guo J, Li J (2020) Exploring the role of visual content in fake news detection. Disinformation, In: Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities, New York, pp 1–19

  27. Han B, Han X, Zhang H, Li J, Cao X (2021) Fighting fake news: Two stream network for deepfake detection via learnable srm. IEEE Transact Biom Behav Identity Sci 3(3):320–331

    Article  MATH  Google Scholar 

  28. Li P, Sun X, Yu H, Tian Y, Yao F, Xu G (2021) Entity-oriented multi-modal alignment and fusion network for fake news detection. IEEE Trans Multimed 24:3455–3468

    Article  MATH  Google Scholar 

  29. Xue J, Wang Y, Tian Y, Li Y, Shi L, Wei L (2021) Detecting fake news by exploring the consistency of multimodal data. Inf Process Manage 58(5):102610–102623

    Article  MATH  Google Scholar 

  30. Chen Y, Li D, Zhang P, Sui J, Lv Q, Tun L et al (2022) Cross-modal ambiguity learning for multimodal fake news detection. Proc ACM Web Conf 2022:2897–2905

    Google Scholar 

  31. Long X, Zhang Z, Li Y (2022) Multi-network contrastive learning of visual representations. Knowl-Based Syst 258:109991

    Article  MATH  Google Scholar 

  32. Wang Q, Zhang W, Lei T, Cao Y, Peng D, Wang X (2023) Clsep: Contrastive learning of sentence embedding with prompt. Knowl-Based Syst 266:110381

    Article  MATH  Google Scholar 

  33. Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P et al (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673

    Google Scholar 

  34. Hua J, Cui X, Li X, Tang K, Zhu P (2023) Multimodal fake news detection through data augmentation-based contrastive learning. Appl Soft Comput 136:110125–110134

    Article  Google Scholar 

  35. Wu J, Xu W, Liu Q, Wu S, Wang L (2023) Adversarial contrastive learning for evidence-aware fake news detection with graph neural networks. IEEE Transact Knowl Data Eng 36:5591–5604

    Article  MATH  Google Scholar 

  36. Wang H, Tang P, Kong H, Jin Y, Wu C, Zhou L (2023) Dhcf: Dual disentangled-view hierarchical contrastive learning for fake news detection on social media. Inf Sci 645:119323–119341

    Article  Google Scholar 

  37. Ishmam MF, Shovon MSH, Mridha M, Dey N (2024) From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities. Inform Fusion 106:102270–102310

    Article  Google Scholar 

  38. Xu P, Zhu X, Clifton DA (2023) Multimodal learning with transformers: A survey. IEEE Trans Pattern Anal Mach Intell 45:12113–12132

    Article  MATH  Google Scholar 

  39. Ma J, He Y, Li F, Han L, You C, Wang B (2024) Segment anything in medical images. Nat Commun 15(1):654

    Article  MATH  Google Scholar 

  40. Ma Y, Ji J, Sun X, Zhou Y, Ji R (2023) Towards local visual modeling for image captioning. Pattern Recogn 138:109420

    Article  MATH  Google Scholar 

  41. Moor M, Banerjee O, Abad ZSH, Krumholz HM, Leskovec J, Topol EJ et al (2023) Foundation models for generalist medical artificial intelligence. Nature 616(7956):259–265

    Article  Google Scholar 

  42. Mehrish A, Majumder N, Bharadwaj R, Mihalcea R, Poria S (2023) A review of deep learning techniques for speech processing. Inf Fusion 99:101869

  43. Yang J, Lu J, Lee S, Batra D, Parikh D (2018) Graph r-cnn for scene graph generation. In: Proceedings of the 15th European Conference on Computer Vision (ECCV), pp 670–685

  44. Birjali M, Kasri M, Beni-Hssane A (2021) A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowl-Based Syst 226:107134

    Article  MATH  Google Scholar 

  45. Lee K-H, Chen X, Hua G, Hu H, He X (2018) Stacked cross attention for image-text matching. In: Proceedings of the 15th European Conference on Computer Vision (ECCV), pp 201–216

  46. Cheng G, Yuan X, Yao X, Yan K, Zeng Q, Xie X et al (2023) Towards large-scale small object detection: Survey and benchmarks. IEEE Transact Pattern Anal Mach Intell 45(11):13467–13488

    MATH  Google Scholar 

  47. Norcliffe-Brown W, Vafeias S, Parisot S (2018) Learning conditioned graph structures for interpretable visual question answering. In: Adv Neural Inf Process Syst 31, pp 8344–8353

  48. Maigrot C, Claveau V, Kijak E, Sicre R (2016) Mediaeval 2016: A multimodal system for the verifying multimedia use task. In: MediaEval 2016: “Verfiying Multimedia Use” task

  49. Zubiaga A, Liakata M, Procter R (2017) Exploiting context for rumour detection in social media. In: Social Informatics: 9th International Conference, pp 109–123

  50. Ma J, Gao W, Wei Z, Lu Y, Wong K-F (2015) Detect rumors using time series of social context information on microblogging websites. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp 1751–1754

  51. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), pp 7370–7377

  52. Wu L, Long Y, Gao C, Wang Z, Zhang Y (2023) Mfir: Multimodal fusion and inconsistency reasoning for explainable fake news detection. Inform Fusion 100:101944

    Article  Google Scholar 

  53. Yang H, Zhang J, Zhang L, Cheng X, Hu Z (2024) Mran: Multimodal relationship-aware attention network for fake news detection. Comput Stand Interfaces 89:103822

    Article  MATH  Google Scholar 

  54. Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Transact Neural Netw Learn Syst 32(1):4–24

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work is supported by the Key Cooperation Project of Chongqing Municipal Education Commission under Grant No. HZ2021008. We would like to thank Xuerui Zhang for his guidance and assistance in revising the paper.

Author information

Authors and Affiliations

  1. School of Big Data and Information Industry, Chongqing City Management College, Chongqing, 401331, China

    Wei Cui

  2. Chongqing Key Laboratory of Big Data and Intelligent Computing, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Chongqing, 400714, China

    Mingsheng Shang

  3. College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Wei Cui

Authors
  1. Wei Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingsheng Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingsheng Shang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cui, W., Shang, M. MIGCL: Fake news detection with multimodal interaction and graph contrastive learning networks. Appl Intell 55, 78 (2025). https://doi.org/10.1007/s10489-024-05883-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05883-3

Keywords

Search

Navigation