short-paper

Open access

Identity-Referenced Deepfake Detection with Contrastive Learning

Authors:

Chengbin QuanAuthors Info & Claims

IH&MMSec '22: Proceedings of the 2022 ACM Workshop on Information Hiding and Multimedia Security

Pages 27 - 32

https://doi.org/10.1145/3531536.3532964

Published: 23 June 2022 Publication History

Abstract

With current advancements in deep learning technology, it is becoming easier to create high-quality face forgery videos, causing concerns about the misuse of deepfake technology. In recent years, research on deepfake detection has become a popular topic. Many detection methods have been proposed, most of which focus on exploiting image artifacts or frequency domain features for detection. In this work, we propose using real images of the same identity as a reference to improve detection performance. Specifically, a real image of the same identity is used as a reference image and input into the model together with the image to be tested to learn the distinguishable identity representation, which is achieved by contrastive learning. Our method achieves superior performance on both FaceForensics++ and Celeb-DF with relatively little training data, and also achieves very competitive results on cross-manipulation and cross-dataset evaluations, demonstrating the effectiveness of our solution.

References

[1]

Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. 2019. Protecting World Leaders Against Deep Fakes. In CVPR Workshops. 38--45.

[2]

Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. 2020. What Makes Fake Images Detectable? Understanding Properties that Generalize. In ECCV (26). 103--120.

[3]

Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. 2020. Gridmask data augmentation. arXiv preprint arXiv:2001.04086 (2020).

[4]

Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. 2018. MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices. In Biometric Recognition, Jie Zhou, Yunhong Wang, Zhenan Sun, Zhenhong Jia, Jianjiang Feng, Shiguang Shan, Kurban Ubul, and Zhenhua Guo (Eds.). Springer International Publishing, Cham, 428--438.

[5]

Francois Chollet. 2017. Xception: Deep Learning With Depthwise Separable Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]

Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, and Luisa Verdoliva. 2021. Id-reveal: Identity-aware deepfake video detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 15108--15117.

[7]

Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).

[8]

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. 2020. Identity-Driven DeepFake Detection. CoRR abs/2012.03930 (2020). arXiv:2012.03930

[9]

Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. 2021. HeadGAN: One-Shot Neural Head Synthesis and Editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 14398--14407.

[10]

Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. 2020. Leveraging Frequency Analysis for Deep Fake Image Recognition. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 3247--3258.

[11]

Shiming Ge, Fanzhao Lin, Chenyu Li, Daichi Zhang, Jiyong Tan, Weiping Wang, and Dan Zeng. 2021. Latent Pattern Sensing: Deepfake Video Detection via Predictive Representation Learning.

[12]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014).

[13]

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 2. IEEE, 1735--1742.

Digital Library

[14]

Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2021. Lips Don't Lie: A Generalisable and Robust Approach To Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5039--5049.

[15]

Javier Hernandez-Ortega, Rubén Tolosana, Julian Fiérrez, and Aythami Morales. 2020. DeepFakesON-Phys: DeepFakes Detection based on Heart Rate Estimation. CoRR abs/2010.00400 (2020). arXiv:2010.00400

[16]

Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang. 2021. Frequency-Aware Discriminative Feature Learning Supervised by Single- Center Loss for Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6458--6467.

[17]

Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang. 2021. Frequency-Aware Discriminative Feature Learning Supervised by Single- Center Loss for Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6458--6467.

[18]

Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. 2019. FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping. CoRR abs/1912.13457 (2019). arXiv:1912.13457

[19]

Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. 2020. Face X-Ray for More General Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]

Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]

Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. 2021. Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 772--781.

[22]

Yu-Cheng Liu, Chia-Ming Chang, I-Hsuan Chen, Yu-Ru Ku, and Jun-Cheng Chen. 2021. An Experimental Evaluation of Recent Face Recognition Losses for Deepfake Detection. In 2020 25th International Conference on Pattern Recognition (ICPR). 9827--9834. https://doi.org/10.1109/ICPR48806.2021.9413323

[23]

Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. FSGAN: Subject Agnostic Face Swapping and Reenactment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

[24]

Niels Provos and Peter Honeyman. 2003. Hide and seek: An introduction to steganography. IEEE security & privacy 1, 3 (2003), 32--44.

[25]

Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, and Jianjun Zhao. 2020. Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms. In Proceedings of the 28th ACM International Conference on Multimedia. 4318--4327.

Digital Library

[26]

Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Niessner. 2019. FaceForensics++: Learning to Detect Manipulated Facial Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

[27]

Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. 2019. Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUI) 3, 1 (2019), 80--87.

[28]

Zekun Sun, Yujie Han, Zeyu Hua, Na Ruan, and Weijia Jia. 2021. Improving the Efficiency and Robustness of Deepfakes Detection Through Precise Geometric Features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3609--3618.

[29]

Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105--6114.

[30]

Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.

Digital Library

[31]

Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2387--2395.

Digital Library

[32]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).

[33]

Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. 2020. CNN-Generated Images Are Surprisingly Easy to Spot... for Now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]

Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021. One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10039--10049.

[35]

Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy. 2018. ReenactGAN: Learning to Reenact Faces via Boundary Transfer. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[36]

Daichi Zhang, Chenyu Li, Fanzhao Lin, Dan Zeng, and Shiming Ge. 2021. Detecting deepfake videos with temporal dropout 3DCNN. IJCAI.

[37]

Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. 2021. Multi-Attentional Deepfake Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2185--2194.

[38]

Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021. Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4176--4186.

Cited By

Qiao TXie SChen YRetraint FShi RLuo X(2024)Deepfake Detection Fighting Against Noisy Label AttackIEEE Transactions on Multimedia10.1109/TMM.2024.338528626(9047-9059)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3385286
Jung JKang CYoon JWoo SHan JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)SAFE: Sequential Attentive Face Embedding with Contrastive Learning for Deepfake Video DetectionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615279(3993-3997)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615279

Index Terms

Identity-Referenced Deepfake Detection with Contrastive Learning
1. Computing methodologies
  1. Artificial intelligence
  2. Computer graphics
    1. Image manipulation
      1. Image processing
2. Security and privacy
  1. Human and societal aspects of security and privacy
    1. Social aspects of security and privacy

Recommendations

AI-assisted deepfake detection using adaptive blind image watermarking
Highlights
- A new adaptive blind image watermarking technology, utilizing artificial intelligence (AI) and named AwDD, has been proposed for detecting color image deepfakes.
- The AI technology used includes face detection, denoising autoencoder (...
Abstract
This paper proposes a new adaptive blind watermarking technology for deepfake detection, which can embed deepfake detection information into the image and verify the image's authenticity without requiring additional information. The proposed ...
PRNU-based Deepfake Detection
IH&MMSec '21: Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security

As deepfakes become harder to detect by humans, more reliable detection methods are required to fight the spread of fake images and videos. In our work, we focus on PRNU-based detection methods, which, while popular in the image forensics scene, have ...
Deepfake Detection for Fake Images with Facemasks
WDC '22: Proceedings of the 1st Workshop on Security Implications of Deepfakes and Cheapfakes

Hyper-realistic face image generation and manipulation have given rise to numerous unethical social issues, e.g., invasion of privacy, threat of security, and malicious political maneuvering, which resulted in the development of recent deepfake ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IH&MMSec '22: Proceedings of the 2022 ACM Workshop on Information Hiding and Multimedia Security

June 2022

177 pages

ISBN:9781450393553

DOI:10.1145/3531536

General Chair:
B.S. Manjunath
UC Santa Barbara, California, USA
,
Program Chairs:
Jan Butora
University of Lille, France
,
Benedetta Tondi
University of Siena, Italy
,
Claus Vielhauer
Brandenburg University of Applied Sciences, Germany

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 June 2022

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

IH&MMSec '22

Sponsor:

SIGMM

IH&MMSec '22: ACM Workshop on Information Hiding and Multimedia Security

June 27 - 28, 2022

CA, Santa Barbara, USA

Acceptance Rates

Overall Acceptance Rate 128 of 318 submissions, 40%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
643
Total Downloads

Downloads (Last 12 months)250
Downloads (Last 6 weeks)35

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qiao TXie SChen YRetraint FShi RLuo X(2024)Deepfake Detection Fighting Against Noisy Label AttackIEEE Transactions on Multimedia10.1109/TMM.2024.338528626(9047-9059)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3385286
Jung JKang CYoon JWoo SHan JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)SAFE: Sequential Attentive Face Embedding with Contrastive Learning for Deepfake Video DetectionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615279(3993-3997)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615279

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents