Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3474085.3481548acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues

Published: 17 October 2021 Publication History

Abstract

Recently, fake news with text and images have achieved more effective diffusion than text-only fake news, raising a severe issue of multimodal fake news detection. Current studies on this issue have made significant contributions to developing multimodal models, but they are defective in modeling the multimodal content sufficiently. Most of them only preliminarily model the basic semantics of the images as a supplement to the text, which limits their performance on detection. In this paper, we find three valuable text-image correlations in multimodal fake news: entity inconsistency, mutual enhancement, and text complementation. To effectively capture these multimodal clues, we innovatively extract visual entities (such as celebrities and landmarks) to understand the news-related high-level semantics of images, and then model the multimodal entity inconsistency and mutual enhancement with the help of visual entities. Moreover, we extract the embedded text in images as the complementation of the original text. All things considered, we propose a novel entity-enhanced multimodal fusion framework, which simultaneously models three cross-modal correlations to detect diverse multimodal fake news. Extensive experiments demonstrate the superiority of our model compared to the state of the art.

References

[1]
Christina Boididou, Katerina Andreadou, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, Michael Riegler, Yiannis Kompatsiaris, et almbox. 2015. Verifying Multimedia Use at MediaEval 2015. In Working Notes Proceedings of the MediaEval 2015 Workshop.
[2]
Christina Boididou, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, Michael Riegler, Stuart E. Middleton, Andreas Petlund, Yiannis Kompatsiaris, et almbox. 2016. Verifying Multimedia Use at MediaEval 2016. In Working Notes Proceedings of the MediaEval 2016 Workshop.
[3]
Juan Cao, Peng Qi, Qiang Sheng, Tianyun Yang, Junbo Guo, and Jintao Li. 2020. Exploring the Role of Visual Content in Fake News Detection. Disinformation, Misinformation, and Fake News in Social Media (2020), 141--161.
[4]
Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information Credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web. 675--684.
[5]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale Hierarchical Image Database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . 248--255.
[6]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171--4186.
[7]
Khattar Dhruv, Goud Jaipal Singh, Gupta Manish, and Varma Vasudeva. 2019. MVAE: Multimodal Variational Autoencoder for Fake News Detection. In The World Wide Web Conference. 2915--2921.
[8]
Bin Guo, Yasan Ding, Lina Yao, Yunji Liang, and Zhiwen Yu. 2020. The Future of False Information Detection on Social Media: New Perspectives and Trends. Comput. Surveys, Vol. 53, 4 (2020), 1--36.
[9]
Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. In Proceedings of the 25th ACM International Conference on Multimedia. 795--816.
[10]
Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations.
[11]
Srijan Kumar and Neil Shah. 2018. False Information on Web and Social Media: A Survey. arXiv preprint arXiv:1804.08559 (2018).
[12]
Peiguang Li, Xian Sun, Hongfeng Yu, Yu Tian, Fanglong Yao, and Guangluan Xu. 2021. Entity-Oriented Multi-Modal Alignment and Fusion Network for Fake News Detection. IEEE Transactions on Multimedia (2021).
[13]
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Advances in Neural Information Processing Systems. 13--23.
[14]
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical Question-Image Co-Attention for Visual Question Answering. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 289--297.
[15]
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting Rumors from Microblogs with Recurrent Neural Networks. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. 3818--3824.
[16]
Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020. Multimodal Analytics for Real-world News Using Measures of Cross-modal Entity Consistency. In Proceedings of the 2020 International Conference on Multimedia Retrieval. 16--25.
[17]
David Nadeau and Satoshi Sekine. 2007. A Survey of Named Entity Recognition and Classification. Lingvisticae Investigationes, Vol. 30, 1 (2007), 3--26.
[18]
Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic Detection of Fake News. In Proceedings of the 27th International Conference on Computational Linguistics. 3391--3401.
[19]
Vahed Qazvinian, Emily Rosengren, Dragomir Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying Misinformation in Microblogs. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. 1589--1599.
[20]
Peng Qi, Juan Cao, Tianyun Yang, Junbo Guo, and Jintao Li. 2019. Exploiting Multi-domain Visual Information for Fake News Detection. In IEEE International Conference on Data Mining. 518--527.
[21]
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter, Vol. 19, 1 (2017), 22--36.
[22]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations .
[23]
Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin'ichi Satoh. 2019. SpotFake: A Multi-modal Framework for Fake News Detection. In Fifth IEEE International Conference on Multimedia Big Data. 39--47.
[24]
Chenguang Song, Nianwen Ning, Yunlei Zhang, and Bin Wu. 2021. A Multimodal Fake News Detection Model Based on Crossmodal Attention Residual and Multichannel Convolutional Neural Networks. Information Processing & Management, Vol. 58, 1 (2021), 102437.
[25]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems. 5998--6008.
[26]
Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 849--857.
[27]
Yaqing Wang, Fenglong Ma, Haoyu Wang, Kishlay Jha, and Jing Gao. 2021. Multi-modal Emergent Fake News Detection via Meta Neural Process Networks. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining .
[28]
Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2020. Fake News Detection via Knowledge-Driven Multimodal Graph Convolutional Networks. In Proceedings of the 2020 International Conference on Multimedia Retrieval . 540--547.
[29]
Junxiao Xue, Yabo Wang, Yichen Tian, Yafei Li, Lei Shi, and Lin Wei. 2021. Detecting Fake News by Exploring the Consistency of Multimodal Data. Information Processing and Management, Vol. 58, 5 (2021), 102610.
[30]
Yang Yang, Lei Zheng, Jiawei Zhang, Qingcai Cui, Zhoujun Li, and Philip S Yu. 2018. TI-CNN: Convolutional Neural Networks for Fake News Detection. arXiv preprint arXiv:1806.00749 (2018).
[31]
Reza Zafarani, Xinyi Zhou, Kai Shu, and Huan Liu. 2019. Fake News Research: Theories, Detection Strategies, and Open Problems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3207--3208.
[32]
Huaiwen Zhang, Quan Fang, Shengsheng Qian, and Changsheng Xu. 2019. Multi-modal Knowledge-aware Event Memory Network for Social Media Rumor Detection. In Proceedings of the 27th ACM International Conference on Multimedia . 1942--1951.
[33]
Xing Zhou, Juan Cao, Zhiwei Jin, Fei Xie, Yu Su, Dafeng Chu, Xuehui Cao, and Junqiang Zhang. 2015. Real-time News Certification System on Sina Weibo. In Proceedings of the 24th International Conference on World Wide Web. 983--988.
[34]
Xinyi Zhou, Jindi Wu, and Reza Zafarani. 2020. SAFE: Similarity-Aware Multi-modal Fake News Detection. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. 354--367.
[35]
Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and Resolution of Rumours in Social Media: A Survey. Comput. Surveys, Vol. 51, 2 (2018), 32.

Cited By

View all

Index Terms

  1. Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MM '21: Proceedings of the 29th ACM International Conference on Multimedia
      October 2021
      5796 pages
      ISBN:9781450386517
      DOI:10.1145/3474085
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 17 October 2021

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. fake news detection
      2. multimodal fusion
      3. social media
      4. visual entity

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      MM '21
      Sponsor:
      MM '21: ACM Multimedia Conference
      October 20 - 24, 2021
      Virtual Event, China

      Acceptance Rates

      Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)271
      • Downloads (Last 6 weeks)21
      Reflects downloads up to 28 Dec 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Do not wait: Preemptive rumor detection with cooperative LLMs and accessible social contextInformation Processing & Management10.1016/j.ipm.2024.10399562:3(103995)Online publication date: May-2025
      • (2025)DPSGInformation Fusion10.1016/j.inffus.2024.102595113:COnline publication date: 1-Jan-2025
      • (2024)Social Event Classification Based on Multimodal Masked Transformer NetworkScientific Insights and Discoveries Review10.59782/sidr.v2i1.1222:1(264-275)Online publication date: 7-Oct-2024
      • (2024)FaKnow: A Unified Library for Fake News DetectionData Intelligence10.3724/2096-7004.di.2024.0026Online publication date: 26-Aug-2024
      • (2024)Ensemble Techniques for Robust Fake News Detection: Integrating Transformers, Natural Language Processing, and Machine LearningSensors10.3390/s2418606224:18(6062)Online publication date: 19-Sep-2024
      • (2024)Fake User Detection Based on Multi-Model Joint RepresentationInformation10.3390/info1505026615:5(266)Online publication date: 9-May-2024
      • (2024)An Environmental Uncertainty Perception Framework for Misinformation Detection and Spread Prediction in the COVID-19 Pandemic: Artificial Intelligence ApproachJMIR AI10.2196/472403(e47240)Online publication date: 29-Jan-2024
      • (2024)Multi-modal Misinformation Detection: Approaches, Challenges and OpportunitiesACM Computing Surveys10.1145/369734957:3(1-29)Online publication date: 22-Nov-2024
      • (2024)Harmfully Manipulated Images Matter in Multimodal Misinformation DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681322(2262-2271)Online publication date: 28-Oct-2024
      • (2024)FakingRecipe: Detecting Fake News on Short Video Platforms from the Perspective of Creative ProcessProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680663(1351-1360)Online publication date: 28-Oct-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media