research-article

Public Access

Deep Multimodal Image-Repurposing Detection

Authors:

Wael AbdAlmageed,

Prem NatarajanAuthors Info & Claims

MM '18: Proceedings of the 26th ACM international conference on Multimedia

Pages 1337 - 1345

https://doi.org/10.1145/3240508.3240707

Published: 15 October 2018 Publication History

Abstract

Nefarious actors on social media and other platforms often spread rumors and falsehoods through images whose metadata (e.g., captions) have been modified to provide visual substantiation of the rumor/falsehood. This type of modification is referred to as image repurposing, in which often an unmanipulated image is published along with incorrect or manipulated metadata to serve the actor's ulterior motives. We present the Multimodal Entity Image Repurposing (MEIR) dataset, a substantially challenging dataset over that which has been previously available to support research into image repurposing detection. The new dataset includes location, person, and organization manipulations on real-world data sourced from Flickr. We also present a novel, end-to-end, deep multimodal learning model for assessing the integrity of an image by combining information extracted from the image with related information from a knowledge base. The proposed method is compared against state-of-the-art techniques on existing datasets as well as MEIR, where it outperforms existing methods across the board, with AUC improvement up to 0.23.

References

[1]

Khurshid Asghar, Zulfiqar Habib, and Muhammad Hussain. 2017. Copy-move and splicing image forgery detection and localization techniques: a review. Australian Journal of Forensic Sciences, Vol. 49, 3 (2017), 281--307.

[2]

Ella Bingham and Heikki Mannila. 2001. Random Projection in Dimensionality Reduction: Applications to Image and Text Data. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '01). ACM, New York, NY, USA, 245--250.

Digital Library

[3]

Christina Boididou, Katerina Andreadou, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, Michael Riegler, and Yiannis Kompatsiaris. 2015. Verifying Multimedia Use at MediaEval 2015. In MediaEval.

[4]

Leo Breiman. 2001. Random Forests . Machine Learning, Vol. 45, 1 (Oct. 2001), 5--32.

Digital Library

[5]

Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and regression trees. http://cds.cern.ch/record/2253780

[6]

Sanjoy Dasgupta. 2000. Experiments with random projection. In Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 143--151.

Digital Library

[7]

Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting on association for computational linguistics. Association for Computational Linguistics, 363--370.

Digital Library

[8]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.

Digital Library

[9]

Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. 2017. Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 1465--1471.

Digital Library

[10]

Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017a. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 795--816.

Digital Library

[11]

Zhiwei Jin, Juan Cao, Yazi Zhang, and Yongdong Zhang. 2015. MCG-ICT at MediaEval 2015: Verifying Multimedia Use with a Two-Level Classification Model. In MediaEval.

[12]

Z. Jin, J. Cao, Y. Zhang, J. Zhou, and Q. Tian. 2017b. Novel Visual and Statistical Image Features for Microblogs News Verification . IEEE Transactions on Multimedia, Vol. 19, 3 (March 2017), 598--608.

Digital Library

[13]

J. R. Kettenring. 1971. Canonical analysis of several sets of variables. Biometrika, Vol. 58, 3 (Dec. 1971), 433--451.

[14]

Regina Marchi. 2012. With Facebook, Blogs, and Fake News, Teens Reject Journalistic "Objectivity". Journal of Communication Inquiry, Vol. 36, 3 (July 2012), 246--262.

[15]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality . In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3111--3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

Digital Library

[16]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python . Journal of Machine Learning Research, Vol. 12 (Oct. 2011), 2825--2830. http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html

Digital Library

[17]

Muhammad Ali Qureshi and Mohamed Deriche. 2015. A bibliography of pixel-based blind image forgery detection techniques. Signal Processing: Image Communication, Vol. 39 (2015), 46--74. http://www.sciencedirect.com/science/article/pii/S0923596515001393

Digital Library

[18]

Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. CSI: A Hybrid Deep Model for Fake News Detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 797--806.

Digital Library

[19]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision, Vol. 115, 3 (Dec. 2015), 211--252.

Digital Library

[20]

Cencheng Shen, Ming Sun, Minh Tang, and Carey E. Priebe. 2014. Generalized canonical correlation analysis for classification. Journal of Multivariate Analysis, Vol. 130, C (2014), 310--322. https://econpapers.repec.org/article/eeejmvana/v_3a130_3ay_3a2014_3ai_3ac_3ap_3a310--322.htm

Digital Library

[21]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition . arXiv:1409.1556 {cs} (Sept. 2014). http://arxiv.org/abs/1409.1556 arXiv: 1409.1556.

[22]

Ratnam Singh and Mandeep Kaur. 2016. Copy Move Tampering Detection Techniques: A Review . International Journal of Applied Engineering Research, Vol. 11, 5 (2016), 3610--3615. https://www.ripublication.com/ijaer16/ijaerv11n5_104.pdf

[23]

Ming Sun, Carey E. Priebe, and Minh Tang. 2013. Generalized canonical correlation analysis for disparate data fusion. Pattern Recognition Letters, Vol. 34, 2 (2013), 194--200.

Digital Library

[24]

Yue Wu, Wael Abd-Almageed, and Prem Natarajan. 2017. Deep Matching and Validation Network: An End-to-End Solution to Constrained Image Splicing Localization and Detection. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 1480--1502.

Digital Library

[25]

Markos Zampoglou, Symeon Papadopoulos, Yiannis Kompatsiaris, Ruben Bouwmeester, and Jochen Spangenberg. 2016. Web and Social Media Image Forensics for News Professionals. In SMN@ ICWSM. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/download/13206/12860

Cited By

Zhang FLiu JXie JZhang QXu YZha ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)ESCNet: Entity-enhanced and Stance Checking Network for Multi-modal Fact-CheckingProceedings of the ACM on Web Conference 202410.1145/3589334.3645455(2429-2440)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645455
Papadopoulos SKoutlis CPapadopoulos SPetrantonakis P(2024)VERITE: a Robust benchmark for multimodal misinformation detection accounting for unimodal biasInternational Journal of Multimedia Information Retrieval10.1007/s13735-023-00312-613:1Online publication date: 8-Jan-2024
https://doi.org/10.1007/s13735-023-00312-6
Xu QDu HChen HLiu BYu X(2024)MMOOC: A Multimodal Misinformation Dataset for Out-of-Context News AnalysisInformation Security and Privacy10.1007/978-981-97-5101-3_24(444-459)Online publication date: 15-Jul-2024
https://doi.org/10.1007/978-981-97-5101-3_24
Show More Cited By

Index Terms

Deep Multimodal Image-Repurposing Detection

Recommendations

Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs
MM '17: Proceedings of the 25th ACM international conference on Multimedia

Microblogs have become popular media for news propagation in recent years. Meanwhile, numerous rumors and fake news also bloom and spread wildly on the open social media platforms. Without verification, they could seriously jeopardize the credibility of ...
Rumor detection in social networks via deep contextual modeling
ASONAM '19: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

Fake news and rumors constitute a major problem in social networks recently. Due to the fast information propagation in social networks, it is inefficient to use human labor to detect suspicious news. Automatic rumor detection is thus necessary to ...
VRoC: Variational Autoencoder-aided Multi-task Rumor Classifier Based on Text
WWW '20: Proceedings of The Web Conference 2020

Social media became popular and percolated almost all aspects of our daily lives. While online posting proves very convenient for individual users, it also fosters fast-spreading of various rumors. The rapid and wide percolation of rumors can cause ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '18: Proceedings of the 26th ACM international conference on Multimedia

October 2018

2167 pages

ISBN:9781450356657

DOI:10.1145/3240508

General Chairs:
Susanne Boll
University of Oldenburg, Germany
,
Kyoung Mu Lee
Seoul National University, Korea
,
Jiebo Luo
University of Rochester, USA
,
Wenwu Zhu
Tsinghua University, China
,
Program Chairs:
Hyeran Byun
Yonsei University, Korea
,
Chang Wen Chen
State Univ. Of New York at Buffalo, USA
,
Rainer Lienhart
University of Augsburg, Germany
,
Tao Mei
JD AI, China

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

DARPA

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22 - 26, 2018

Seoul, Republic of Korea

Acceptance Rates

MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
692
Total Downloads

Downloads (Last 12 months)102
Downloads (Last 6 weeks)9

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang FLiu JXie JZhang QXu YZha ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)ESCNet: Entity-enhanced and Stance Checking Network for Multi-modal Fact-CheckingProceedings of the ACM on Web Conference 202410.1145/3589334.3645455(2429-2440)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645455
Papadopoulos SKoutlis CPapadopoulos SPetrantonakis P(2024)VERITE: a Robust benchmark for multimodal misinformation detection accounting for unimodal biasInternational Journal of Multimedia Information Retrieval10.1007/s13735-023-00312-613:1Online publication date: 8-Jan-2024
https://doi.org/10.1007/s13735-023-00312-6
Xu QDu HChen HLiu BYu X(2024)MMOOC: A Multimodal Misinformation Dataset for Out-of-Context News AnalysisInformation Security and Privacy10.1007/978-981-97-5101-3_24(444-459)Online publication date: 15-Jul-2024
https://doi.org/10.1007/978-981-97-5101-3_24
Pham KNguyen-Nhat MDinh ALe QNguyen MTran ATran MDang-Nguyen D(2024)Ookpik- A Collection of Out-of-Context Image-Caption PairsMultiMedia Modeling10.1007/978-3-031-53302-0_10(132-144)Online publication date: 29-Jan-2024
https://doi.org/10.1007/978-3-031-53302-0_10
Papadopoulos SKoutlis CPapadopoulos SPetrantonakis P(2023)Synthetic Misinformers: Generating and Combating Multimodal MisinformationProceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation10.1145/3592572.3592842(36-44)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592572.3592842
Mu MDas Bhattacharjee SYuan J(2023)Self-Supervised Distilled Learning for Multi-modal Misinformation Identification2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00284(2818-2827)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00284
Shalabi FNguyen HFelouat HChang CEchizen I(2023)Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC58517.2023.10317336(605-612)Online publication date: 31-Oct-2023
https://doi.org/10.1109/APSIPAASC58517.2023.10317336
Wilson AWilkes STeramoto YHale S(2023)Multimodal analysis of disinformation and misinformationRoyal Society Open Science10.1098/rsos.23096410:12Online publication date: 20-Dec-2023
https://doi.org/10.1098/rsos.230964
Sabir ENandi SAbdAlmageed WNatarajan P(2022)MONet: Multi-Scale Overlap Network for Duplication Detection in Biomedical Images2022 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP46576.2022.9897213(3793-3797)Online publication date: 16-Oct-2022
https://doi.org/10.1109/ICIP46576.2022.9897213
Huang MJia SChang MLyu S(2022)Text-Image De-Contextualization Detection Using Vision-Language ModelsICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP43922.2022.9746193(8967-8971)Online publication date: 23-May-2022
https://doi.org/10.1109/ICASSP43922.2022.9746193
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents