COMBINING COMPUTER VISION TECHNIQUES AND INTRAFRAME NOISE

METHODS TO DETECT A DEEPFAKE

Ajinkya Kharade1, Pranav Gaikwad1, Shubham Arakh1, Analp Kalore1, Maya P Shelke1, Nihar Ranjan1

1. Department of Information Technology, JSPM’s Rajarshi Shahu College Of Engineering, Pune, Maharashtra, India.
Email: ajinkyakharade373@gmail.com, pranavg225@gmail.com, shubhamarakh4@gmail.com,
analpkalore310@gmail.com, mayabembde07@gmail.com, nihar.pune@gmail.com

Abstract – Deep fakes are synthetic videos created by deep upper shoulders. And the synthesized target appears to
learning algorithms that can convincingly depict individuals behave same way as the source. Face Swapping – it involves
saying or doing things they never did. With the proliferation of swapping of face of targeted person of a video with source
deepfake videos in social media and the potential for them to
video while keeping the same facial expressions. Lip
cause significant harm, deep fake detection has become a
pressing issue. In this research paper, we propose a novel
Syncing – particularly targets on the lip region and syncs it
approach to detect deepfakes by combining computer vision with audio that appears to speak something which he/she
methods with intraframe noise. The proposed approach doesn’t speak in reality [8]. In order to create Deep Fake the
involves extracting features from the video frames, including Deep neural networks are used. Various recent technological
texture, color, and edges, and then adding a layer of intraframe advancements with deep learning techniques including auto
noise to the video frames. We evaluated the proposed approach encoders and GAN [12][13](Generative Adversarial
on several benchmark datasets and found that it achieved high Networks) are used to create fake faces which are applied
accuracy in detecting deepfake videos. mainly in the computer vision. Deep Fake detection method
using the Haar wavelet transforms. The method aims to
Keywords: Deep Fake, intraframe noise, extracting features,
high accuracy. distinguish between original videos and Deep Fake videos by
exploiting the limitations and artifacts introduced during the
Deep Fake generation process.
The proposed method takes advantage of the fact that Deep
1. INTRODUCTION Fake algorithms have specific constraints, such as generating
fake faces with a particular size and resolution. To match the
Deep Fake is a technology that can superimpose the face of a source face to the target in the original video, an affine
person with another person’s face, alter facial expressions or transformation and blur function are applied to the
gestures, appearing as in the original video using artificial synthesized faces. This transformation leaves traces of
intelligence concepts. It manipulates the whole activity of the resolution and blurs inconsistency between the transformed
targeted person which synthesizes audio and visual aspects. region of interest (ROI) and the surrounding area.
Deepfakes are often created with the intention to deceive or
mislead viewers into believing that the manipulated content
is genuine. Some social media users posted a doctored video
to promote the myth that After meeting with President
Donald Trump, House Speaker Nancy Pelosi fumbled over
her speech [1]. The deep fake technology has created and
developed videos that cannot be identified by the normal
human being eyes [2]. It can detect the real and fake images
by using advanced network architecture.

To identify the forged videos and images created by using

deep learning methods can be achieved [3].
The fake images can be present in three forms [7].
Any AI-generated mimicking videos are now referred to as
"Deep Fake," which is a more general term. Basically, there
are three major types of Deep Fake videos. Head puppetry -
synthesizes a video of a targeted person’s whole head and
 switched in the original movie, In the generated DeepFake
Figure: 1[17]. Original vs Fake movies, this additive blur and ROI transformations leave
unique artefacts that may be efficiently recorded by spotting
discrepancies between the ROI and the rest of the image
The detection method utilizes the Haar wavelet transform to using Haar Wavelet transformation.
analyze the blur inconsistency and detect Deep Fake forgery.
Using the Haar wavelet transform function, it compares the Dolhansky, Brian and Howes, Russ and Pflaum, Ben and
blurred synthesised ROI with the surrounding environment. Baram, Nicole and Ferrer, Cristian (2019) in “The Deepfake
By analyzing edge types and sharpness, the method Detection Challenge (DFDC)Preview Dataset.” [7]
determines whether a face image has been blurred and to introduced a preview of the DFDC dataset that will be made
what extent [10]. Major tech companies are actively available later this year with the goal of encouraging
researching methods to detect deepfakes to fight the researchers to familiarise themselves with the data, providing
increasing number of deepfakes on the internet. Recently, preliminary findings, and comparing those findings to
Facebook, Amazon, Microsoft, and the Partnership on AI’s suggested baselines.
Media Integrity Steering Committee have come together and R. Saravana Ram, M. Vinoth Kumar, Tareq M. Al-shami,
launched the Deepfake Detection Challenge to encourage Mehedi Masud, Hanan Aljuaid and Mohamed Abouhawwash
more research and development in detecting and preventing in "Deep Fake Detection Using Computer Vision-Based
deepfakes [6]. Also, Google has released a free dataset for Deep Neural Network with Pairwise Learning" [8] suggested
the public as a contribution to the deepfake detection. The extracting features from the input deepfake image using
Deepfake Detection Challenge initiated by major tech fuzzy clustering.
companies and the release of a free dataset by Google is
mentioned as examples of industry efforts in addressing the In "Deep fake detection using a sparse auto encoder with a
deepfake issue [7], combines deep learning techniques like Graph capsule dual graph CNN" [9], Kandasamy V,
recurrent neural networks (RNN), convolutional neural Hubálovsk, and Trojovsk revealed the two-level deep
networks (CNN), and long short-term memory to provide a learning approach for the detection of deepfake images and
thorough research for deepfake identification. [10] [11] [16]. videos. The proposed SAE approach is used to extract the
features from the face photos.

2. LITERATURE SURVEY Abdulqader M. Almars [10gave a thorough explanation of

the architecture, tools, and performance of the existing
deepfake approaches. It also highlighted the publicly
"Deepfake detection using the rate of change between frames available datasets utilised by the scientific community and
based on computer vision," by G. Lee and M. Kim. [2] sorted them by source, method, and dataset.
proposed a technique for determining the rate of change of
computer vision features using a DNN based on the The proposed technique by Aarti Karandikar, Vedita
difference between a certain number of frames and frames. Deshpande, Sanjana Singh, Sayali Nagbhidkar, and Saurabh
Agrawal [11] focuses on facial manipulation for forgery
The method is the first real-time facial reproduction system detection and employs transfer learning on the VGG-16
that only needs monocular RGB input, according to J. Thies, model to train the dataset.
"Face2face: Real-time face capture and reenactment of RGB
videos," by M. Zollhofer, M. Stamminger, C. Theobalt, and Saravana Balaji Balasu bramanian, Jagadeesh Kannan,
M. Nießner [3]. Prabu, Venkata chalam and Pavel Trojovský [12] better
accuracy.
According to S. Lyu's article "Deepfake detection: Current
challenges and next steps," [4] Predict that a number of Suganthi ST. Ayoobkhan MUA, VKK,
upcoming technology advancements will enhance the phoney BacaninN ,KV,ŠtěpánH,PavelT [14] the fisherface Linear
videos' production effectiveness and visual quality. binary pattern histogram using the DBN classifier (FF-LBPH
DBN) technique was implemented as a detection technique
In "Effective and Fast DeepFake Detection Method Based on for deepfake images. The planned work was carried out
Haar Wavelet Transform," Mohammed Akram Younus and much quickly, and it was quite good at distinguishing
Taha Mohammed Hasan. [6] DeepFakes, a novel method for between phoney and real images.
spotting false faces created artificially, have been proposed.
Given that the DeepFake method can only create face images TAO JIANG, JIAN PING LI, AMIN UL HAQ, ABDUS
with fixed sizes and low resolutions, which must then be SABOOR [16] An very accurate bidirectional recurrent
further blurred and changed to match the faces to be neural network model may identify bogus news.
 3. PROPOSED SYSTEM metadata. Advanced techniques such as deep neural
networks or convolutional neural networks (CNNs)
Deepfake detection is used in various real-life applications can be employed to extract discriminative features.
such as social media platforms, news verification, law
enforcement, entertainment industry, political campaigns, 2. Training and Model Development: A machine
online identity verification, and online reputation learning model is trained on a large and diverse
management. It helps identify and flag manipulated content, dataset that contains both real and deepfake
verify the authenticity of videos and images, analyze
samples. The model learns to differentiate between
evidence in legal proceedings, enforce copyright and protect
intellectual property, combat political misinformation, verify genuine and manipulated media by analyzing the
online identities, and manage online reputation. Deepfake extracted features. Various techniques like
detection is a dynamic field that requires continuous research Depending on the situation, supervised learning,
and updates to stay ahead of evolving deepfake techniques. unsupervised learning, or semi-supervised learning
can be used, the availability of labeled data.

3. Detection Algorithm: The trained model is then

used as the core detection algorithm. It takes the
extracted features of an unknown sample as input
and calculates a confidence score or probability
indicating the likelihood of it being a deepfake. The
algorithm should be designed to detect both known
and unknown types of deepfakes.

4. Multimodal Analysis: To enhance the accuracy and

reliability of the detection system, multimodal
analysis can be employed. This involves analyzing
multiple modalities such as video, audio, and
metadata simultaneously. Inconsistencies or
discrepancies across different modalities can
indicate the presence of a deepfake.

5. Post-processing and Fusion: The outputs from

different detection algorithms or modalities can be
fused or combined to generate a final decision.
Fusion techniques can include weighted averaging,
ensemble methods, or decision-level fusion. This
Figure: 2[8]. Prepro System for Deepfake step helps improve the overall accuracy and
reliability of the system.

6. Real-time Implementation: To detect deepfakes in

deepfake detection can consist of several components and real-time scenarios, the system should be optimized
techniques working together. A high-level description of a for efficiency and speed. Techniques like parallel
hypothetical system is given below: computing, hardware acceleration (e.g., GPUs), or
Preprocessing: The system starts by FIG.3 preprocessing the model compression can be utilized to achieve real-
input data, which could be a video, image, or audio file. This time performance.
step may involve resizing, normalizing, or filtering the data 7. Continuous Monitoring and Updates: Deepfake
to ensure consistency and compatibility with the subsequent techniques are evolving rapidly, so the system
analysis steps. should be regularly updated with new training data
1. Feature Extraction: In this stage, relevant features and models. Continuous monito to stay effective in
are extracted from the input data. These features can detecting the latest threats.
include visual cues, audio characteristics, and
 8. User Interface and Reporting: The system can information for differentiating between real and
provide a user-friendly interface for users to interact deepfake content.
with and analyze the results. It can generate detailed
3. Ensemble Methods: Train multiple deepfake
reports highlighting the detected deepfake
detection models with different architectures or
indicators, confidence scores, and any additional
trained on different subsets of the dataset. Combine
information that aids in further investigation.
their predictions through ensemble methods like
9. ring and research on emerging deepfake techniques majority voting or averaging to improve detection
will enable the system to stay effective in detecting accuracy. Experiment with different ensemble
the latest threats. techniques to find the optimal combination.

10. User Interface and Reporting: The system can 4. Hardware Acceleration: Utilize GPUs or other
provide a user-friendly interface for users to interact hardware accelerators to accelerate the execution of
with and analyze the results. It can generate detailed deep learning models. This can be done using deep
reports highlighting the detected deepfake learning frameworks like TensorFlow or PyTorch,
indicators, confidence scores, and any additional which provide GPU support. Ensure that the
information that aids in further investigation. hardware infrastructure is properly set up and
configured for efficient parallel processing.

5. Model Optimization: Apply model optimization

techniques to reduce the computational load and
4. IMPLEMENTATION DETAILS
memory requirements. This may include model
Implementing deepfake detection using computer vision with quantization to reduce the precision of model
improved quality and speed involves several key steps and weights and activations, or model compression
considerations. Consider the following implementation techniques to reduce the number of parameters.
details: Utilize optimized libraries or frameworks like
TensorFlow Lite or TensorRT to improve the speed
Dataset Collection and Preparation: Gather a diverse and of execution.
high-quality dataset consisting of both real and deepfake
samples. Ensure that the dataset covers various deepfake 6. Real-time Implementation: Optimize the system for
generation techniques, resolutions, lighting conditions, and real-time performance by minimizing latency. Use
camera angles. Preprocess the data by resizing, normalizing, efficient data structures and algorithms to optimize
and augmenting the images or videos as needed. the processing of intermediate results. Employ
techniques like batch processing, multi-threading, or
1. Model Selection and Training: Choose a suitable asynchronous processing to parallelize computations
deep learning architecture for deepfake detection, and reduce latency.
such as a convolutional neural network (CNN) or a
combination of CNN and recurrent neural networks 7. Continuous Improvement: Stay updated with the
(RNNs) for temporal analysis. Consider using pre- latest research advancements in deepfake generation
trained models like ResNet, Inception, or and detection. Continuously evaluate and refine the
EfficientNet and fine-tune them on the deepfake deepfake detection system by incorporating new
detection task. Employ transfer learning to leverage techniques, datasets, and model updates. Regularly
features learned from large-scale datasets like monitor performance metrics and incorporate
ImageNet. feedback to enhance the quality and speed of
detection.
2. Feature Extraction: Extract meaningful features
from the input data using computer vision 8. Evaluation and Testing: Evaluate the performance
techniques. This may involve face detection, facial of the deepfake detection system using appropriate
landmark detection, texture analysis, or motion parameters including F1 score, recall, accuracy, and
analysis. Select features that capture relevant precision. Conduct extensive testing on diverse
datasets, including unseen deepfake variations, to
 assess the robustness and generalization capability θ∗¿ argminθ L(θ)
of the system.
The choice of the loss function depends on the specific
9. Deployment and Integration: Integrate the deepfake approach and can vary. Commonly used loss functions for
detection system into the desired application or binary classification include cross-entropy loss or binary
platform. Ensure that the system is scalable, reliable, logistic loss.
and user-friendly. Consider factors like system
requirements, user interfaces, and deployment Once the model is trained, it can be used to classify new,
options (cloud-based, on-premise, or edge unseen data by calculating the probability of being a
deepfake using the learned parameters:
computing) based on the specific use case and
P¿
deployment environment.
The threshold for classifying an input as a deepfake can be
It's important to note that the implementation details can vary adjusted based on the desired balance between false positives
depending on the specific deepfake detection approach, and false negatives.
available resources, and the chosen technologies.
It is important to note that the specific implementation and
Experimentation, fine-tuning, and adapting the mathematical expressions may vary depending on the chosen
implementation to suit the specific requirements and algorithms and techniques used for feature extraction and
constraints of the system are necessary to achieve optimal classification. The above representation provides a general
results. framework for understanding the process of deepfake
detection using computer vision and algorithms.
A mathematical expression representing the process of
deepfake detection using computer vision and algorithms:

Let: 5. RESULT
X be the input data (preprocessed images/videos).
Y be the ground truth labels (0 for real content, 1 for
The result of the deepfake detection system can be a binary
deepfake content).
classification indicating whether the input media (image or
video) is classified as a deepfake or genuine. This result is
f ( x ) be the feature extraction function that maps the input based on the analysis and prediction made by the trained
data to a set of features. deep learning model or ensemble of models.

The system may provide a confidence score or probability

θ be the parameters of the detection model. along with the classification result, indicating the level of
certainty in the prediction. A higher confidence score
The deepfake detection model can be represented as a suggests a higher likelihood of the input being classified
function h ( X ; θ ) that takes the preprocessed data X and the correctly.
model parameters θ as inputs and outputs the probability of
Additionally, the system may generate a detailed report
the input being a deepfake:
highlighting the detected deepfake indicators or features that
contributed to the classification. This information can be
h ( X ; θ ) =P(Y =1∨X ; θ) useful for further investigation or analysis.

The overall result of the system can be evaluated using

To train the model, we use a labeled dataset performance metrics such as accuracy, precision, recall, and
D= { ( X 1 ,Y 1 ) , ( X 2 , Y 2 ) … ( Xn, Yn ) } F1 score. These metrics assess the effectiveness and
reliability of the deepfake detection system in correctly
identifying deepfakes while minimizing false positives or
false negatives.
The purpose is to discover the best parameters θ * for
It is important to note that the accuracy and performance of
minimising classification error. This is accomplished by the system can vary depending on the quality and diversity of
minimising the loss function. L(θ)with respect to θ : the training dataset, the chosen detection techniques, and the
implementation details. Regular monitoring, evaluation, and
continuous improvement are crucial to ensure the system's working on real-time detection systems, countering anti-
effectiveness in detecting evolving deepfake techniques. forensic techniques, raising awareness through education,
and staying ahead of deepfake technology advancements.
Although detecting deepfakes completely may become more
6. CONCLUSION challenging, ongoing research and technological
advancements will help mitigate the risks associated with
Choosing a suitable deep learning architecture, such as CNN deepfake technology.
or a combination of CNN and RNN, is important for
deepfake detection. Leveraging pre-trained models like
ResNet, Inception, or EfficientNet through transfer learning 8. REFERENCES
can provide a good starting point. Feature extraction
techniques, including face detection, facial landmark [1] https://edition.cnn.com/2019/05/23/politics/doctored-
detection, texture analysis, and motion analysis, help capture video-pelosi/index.html.
relevant information for differentiation. [2] Lee, G. and Kim, M., 2021. Deepfake detection using the
rate of change between frames based on computer vision.
Ensemble methods, such as combining predictions from Sensors, 21(21), p.7367.
multiple models through majority voting or averaging, can [3] Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C.
enhance detection accuracy. Leveraging GPUs or other and Nießner, M., 2016. Face2face: Real-time face capture
hardware accelerators, along with model optimization and reenactment of rgb videos. In Proceedings of the IEEE
techniques like quantization and compression, improves the conference on computer vision and pattern recognition (pp.
speed of execution. Real-time implementation can be 2387-2395).
achieved by minimizing latency and utilizing efficient [4] Lyu, S., 2020, July. Deepfake detection: Current
algorithms. challenges and next steps. In 2020 IEEE international
conference on multimedia & expo workshops (ICMEW) (pp.
Continuously updating the system with the latest research 1-6). IEEE.
advancements, evaluating performance metrics, and [5] Younus, M.A. and Hasan, T.M., 2020, April. Effective
incorporating feedback are crucial for continuous and fast deepfake detection method based on haar wavelet
improvement. Evaluating the system using appropriate transform. In 2020 International Conference on Computer
metrics and extensive testing on diverse datasets ensures Science and Software Engineering (CSASE) (pp. 186-190).
robustness and generalization capability. IEEE.
[6] Khalil, H.A. and Maged, S.A., 2021, May. Deepfakes
Integrating the deepfake detection system into the desired creation and detection using deep learning. In 2021
application or platform requires scalability, reliability, and International Mobile, Intelligent, and Ubiquitous Computing
user-friendliness. Consider system requirements, user Conference (MIUCC) (pp. 1-4). IEEE.
interfaces, and deployment options based on the specific use [7] Dolhansky, B., Howes, R., Pflaum, B., Baram, N. and
case and environment. Ferrer, C.C., 2019. The deepfake detection challenge (dfdc)
preview dataset. arXiv preprint arXiv:1910.08854.
Remember that the implementation details provided here [8] Saravana Ram, R., Vinoth Kumar, M., Al-shami, T.M.,
serve as a general guide. Adjustments and experimentation Masud, M., Aljuaid, H. and Abouhawwash, M., 2023. Deep
may be necessary to optimize the deepfake detection system Fake Detection Using Computer Vision-Based Deep Neural
based on your specific needs and available resources. Stay Network with Pairwise Learning. Intelligent Automation &
informed about the latest developments in deepfake Soft Computing, 35(2).
generation and detection to ensure your system remains [9] Kandasamy, V., Hubálovský, Š. and Trojovský, P., 2022.
effective against evolving deepfake techniques. Deep fake detection using a sparse auto encoder with a graph
capsule dual graph CNN. PeerJ Computer Science, 8, p.e953.
[10] Almars, A.M., 2021. Deepfakes detection techniques
7. FUTURE SCOPE using deep learning: a survey. Journal of Computer and
Communications, 9(5), pp.20-35.
The future scope for deepfake detection involves the [11] Karandikar, A., Deshpande, V., Singh, S., Nagbhidkar,
development of advanced algorithms and techniques to S. and Agrawal, S., 2020. Deepfake video detection using
detect increasingly sophisticated deepfake videos, images, convolutional neural network. International Journal of
and audio. This includes improving detection algorithms Advanced Trends in Computer Science and Engineering,
through machine learning and AI, incorporating multimodal 9(2), pp.1311-1315.
analysis across different sources, creating benchmark [12] Balasubramanian, S.B., Prabu, P., Venkatachalam, K.
datasets for evaluation, focusing on explainable AI to and Trojovský, P., 2022. Deep fake detection using cascaded
provide transparency, promoting collaboration and standards,
deep sparse auto-encoder for effective feature selection.
PeerJ Computer Science, 8, p.e1040.
[13] Shad, H.S., Rizvee, M., Roza, N.T., Hoq, S.M.,
Monirujjaman Khan, M., Singh, A., Zaguia, A. and
Bourouis, S., 2021. Comparative analysis of deepfake image
detection method using convolutional neural network.
Computational Intelligence and Neuroscience, 2021.
[14] Suganthi, S.T., Ayoobkhan, M.U.A., Bacanin, N.,
Venkatachalam, K., Štěpán, H. and Pavel, T., 2022. Deep
learning model for deep fake face recognition and detection.
PeerJ Computer Science, 8, p.e881.
[15] Rafique, R., Gantassi, R., Amin, R., Frnda, J.,
Mustapha, A. and Alshehri, A.H., 2023. Deep fake detection
and classification using error-level analysis and deep
learning. Scientific Reports, 13(1), p.7422.
[16] Jiang, T., Li, J.P., Haq, A.U. and Saboor, A., 2020,
December. Fake news detection using deep recurrent neural
networks. In 2020 17th International Computer Conference
on Wavelet Active Media Technology and Information
Processing (ICCWAMTIP) (pp. 205-208). IEEE.
[17]https://www.google.com/url?sa=i&url=https%3A%2F
%2Fwww.trymaverick.com%2Fblog-posts%2Fare-deep-
fakes-all-evil-when-can-they-be-used-for-
good&psig=AOvVaw2iw8xCwMUh9D92KIFneNPs&ust=1
685099255505000&source=images&cd=vfe&ved=0CBQQ3
YkBahcKEwiY5pSJqpD_AhUAAAAAHQAAAAAQBA