Face Emotion Recognition Using Python

A Project report submitted in partial fulfilment of the requirements for the award of Degree of

BACHELOR OF TECHNOLOGY
In
ELECTRONICS AND COMMUNICATION ENGINEERING
Submitted by

S.HEMALATHA A.CHARAN KUMAR

(19A51A04B6) (19A51A0466)
T.SUSMITHA B.RAJ KUMAR
(19A51A04B9) (19A51A074)
Under the esteemed guidance of
Dr D. Yugandhar, M.Tech, Ph.D
Associate Dean & Carrier guidance academics incharge
Department of ECE
AITAM, Tekkali.

DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

ADITYA INSTITUTE OF TECHNOLOGY AND MANAGEMENT
TEKKALI, ANDHRA PRADHESH.
(An Autonomous Institute)
Approved by AICTE, Permanently affiliated to JNTUK, Accredited by NBA (AICTE) & NAAC
(UGC),
Recognized by UGC u/s 2(f) & 12(B), TEQIP (Phase II) Funded College, Recognized as SIRO by
DSIR, DST, and New Delhi.
2021
 DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING
ADITYA INSTITUTE OF TECHNOLOGY AND MANAGEMENT-AITAM
(Permanently affiliated to Jawaharlal Nehru Technological University, Kakinada,AP)
Approved by AICTE, Accredited by NBA,
K. Kotturu, Tekkali, Srikakulam District- 532201.

CERTIFICATE
This is to certify that the Minor Project work entitled
“FACE EMOTION RECOGNITION USING PYTHON” is a bonified work done
by S. Hemalatha (19A51A04B6) , T. Sushmitha (19A51A04B9), A.
Charan kumar (19A51A0466) , B. Raj kumar (19A51A0474) and submitted in
partial fulfilment of the requirement for the award of the degree of BACHELOR OF
TECHNOLOGY in ELECTRONICS AND COMMUNICATION
ENGINEERING.

Head of the Department: Project guide:

Dr B. Rama Rao Dr D. Yugandhar
M.Tech, Ph.D. , SMIEEE M.Tech, Ph.D.
Professor and HOD Associate Dean
Department of ECE (Alumni, C.G., H.E)
 ACKNOWLEDGEMENT
It is indeed with a great sense of pleasure and immense sense of gratitude that we
acknowledge the help of these individuals. We are highly indebted to our college Director
Prof.V.V. Nageswara Rao and Principal Dr. A. Srinivasa Rao for the facilities provided
to accomplish this project.
We would like to thank our Head of the Department Dr B. Rama Rao for his
Construct criticism throughout our project.
We feel elated in manifesting our sense of gratitude to our Guide Dr D. Yugandhar,
sir for his valuable guidance. He has been a constant source of inspiration for us and we are
very deeply thankful to him for his support and invaluable advice.
We are extremely grateful to our Department staff members, lab technicians and Non-
teaching staff members for their extreme help throughout our project.
Finally we express our heart full thanks to all of our friends who helped us in successful
completion of this project.

S. Hemalatha
T. Sushmitha
A. Charan kumar
B. Rajkumar
 DEPARTMENT OF ECE
Vision of the Department:
Create high quality engineering professionals through research, innovation and team work for
a lasting technology development in the area of electronics and communication engineering.

Mission of the Department:

1. To offer a well-balanced program of instruction, lab practices, research & development
activities, product incubation.
2. Develop accomplished technical personnel with a strong background on fundamental and
advanced concepts, have excellent professional conduct.
3. Enhance overall personality development which includes innovative and group work
exercises, entrepreneur skills, communication skills and employability.
4. Ensuring effective teaching-learning process to provide in depth knowledge of principals
and its applications to electronics and communication engineering and interdisciplinary areas.
5. Providing industry and development interactions through consultancy and sponsored
research.

Program Educational Objectives (PEOs) Of B.Tech in ECE:

PEO I: The graduates would be employed as a practicing engineer in fields such as design,
testing and manufacturing.
PEO II: the graduates would be able to imbibe research, development and entrepreneurship
skills.
PEO III: The graduates would be engaged lifelong self-director learning to maintain and
enhance professional skills.
PEO IV: The graduates will be able to exhibit communication skills, team spirit, leader ship
skills and ethics with social responsibility.
PROGRAM OUTCOMES:
Engineering Graduates Will Be Able To:
1. Engineering Knowledge: Apply the knowledge of mathematics, science, engineering
fundamentals, and an engineering specialization to the solution of complex engineering
problems.

2. Problem analysis: Identify, formulate, review research literature, and analyse complex
engineering problems reaching sustained conclusions using first principals of
mathematics, natural sciences and engineering sciences.

3. Design/ development of solutions: Design solution for complex engineering problem s

and design system components or process that meet the specified needs with appropriate
consideration for the public health and safety and the cultural, societal and environment
considerations.

4. Conduct investigations of complex problems: Use research based knowledge and

research methods including design of experiments, analysis and interpretation of data and
synthesis of the information to provided valid conclusions.

5. Modern tool usage: Create, select and apply appropriate techniques, resources, and
modern engineering and it tools including prediction and modelling to complex engineering
activities with an understanding of the limitations

6. The engineer and society: Apply reasoning informal by the contextual knowledge to
access societal, health, safety, legal and cultural issues and the consequent responsibilities
relevant to the professional engineering practice.

7. Environment and sustainability: Understand the impact of professional engineering

solution in societal and environmental contexts, and demonstrate the knowledge of, and
need for sustainable development.

8. Ethics: Apply ethical principles and commit to professional ethics and responsibilities and
norms of the engineering practice.

9. Individual and team work: Function effectively as an individual, and as a member or

leader in diverse teams, and in multidisciplinary settings.

10. Communication: Communicate effectively on complex engineering activities with the

engineering community and with society at large, such as, being able comprehend and
bright effective reports and design documentation, make effective presentations, and
give and receive clear instructions.
11. Project management and finance: Demonstrate knowledge and understanding of the
Engineering and management principals and apply these to ones own work as a
Member leader in a team, to manage projects and in multidisciplinary environments.

12. Lifelong learning: Recognize the need for, and have the preparation and ability to
engage in independent and lifelong learning in the broadest context of technological
change.

PSO –PROGRAM SPECIFIC OUTCOMES:

PSO1: The competency in the application of circuit analysis and design.
PSO II: An ability to solve electronics and communication engineering problems using
latest hardware and software tools, along with analytical skills to arrive cost effective
cost appropriate solutions.
PSO III: The ability to pursue higher studies in either India and abroad and also lead a
successful career with professional ethics.
 CONTENTS

1. ABSTRACT------------------------------------------------------------------------------9

2. CHAPTER-1------------------------------------------------------------------------------10
1. INTRODUTION
1,1 FACIAL EXPRESSIONS
1.2 IMAGE
1.3 IMAGE PROCESSING
1.4 PIXEL
1.5 TYPES OF IMAGES
1.5,1 BINARY IMAGE
1.5.2 BLACK AND WHITE IMAGE
1.5.3 GRAYSCALE IMAGE
1.5.4 COLOR IMAGE

3. CHAPTER-2------------------------------------------------------------------------------16

2.1 LITERATURE SURVEY

2.1.1 WHAT IS FACE DETECTION

2.1.2 HOW DOES FACE DETECTION WOORK

2.1.3 FACE DETECTION METHODS

2.1.4 FACE DETECTION TECHNIQUES

2.1.5 WHAT ARE THE CHALLENGES IN FACE DETECTION

2.1.6 WHY IS FACE DETECTION IMPORTANT TODAY

2.1.7 FACE DETECTION VS FACE RECOGNITION

2.2 MOTIVATION

4. CHAPTER-3--------------------------------------------------------------------------------23

3.1 CONVOLUTIONAL NEURAL NETWORKS(CNN)3.2 WORKING OF CNN

 3.3.1 ADVANTAGES OF CNN

3.2.2 DISADVATAGE OF CNN

5. CHAPTER-4-------------------------------------------------------------------------------27

4. SYSTEM RECQURIMENT SPECIFICATION

4.1 PYTHON SOFTWARE

4.2 FEATURES OF PYTHON

4.3 FER LIBRARY

6. CHAPTER -5-------------------------------------------------------------------------------30
5.IMPLEMENTATION
5.1 BLOCK DIAGRAM
5.2 PROJECT DISGING / CODE
5.3 WORKING
5.4 RESULTS
7. CHAPTER -6 --------------------------------------------------------------------------------34
6.1 APPLICATIONS
6.1.1 FACIAL EMOTION DETECTION IN INTERVIEWS
6.1.2 TESTING FOR VIDEO GAMES
6.1.3 MARKET RESEARCH
6.2 ADVANTAGES
6.3 DISADVANTAGES
6.4 CONCLUSION
6.5 FUTURESCOPE
8. CHAPTER-7-----------------------------------------------------------------------------------38
7.1 REFERENCES
7.2 WEB REFERENCES
 ABSTRACT
One of the ways humans communicate is by using facial expressions. Research on technology
development in artificial intelligence uses deep-learning methods in human and computer
interactions as an effective system application process. One example, if someone does show
and tries to recognize facial expressions when communicating. The prediction of the expression
or emotion of some people who see it sometimes does not understand. In psychology, the
detection of the face emotions or facial expressions requires analysis and assessment of
decision in prediction a person’s emotions or a group of people communicating. This research
proposes the design of a system that can predict and recognize the classification of facial
emotions based Network (CNN) method using Facial Emotion Recognition (FER-2013) were
on feature extraction using the Convolutional Neural Network (CNN) algorithm in real-time
with the Open CV library, namely : Tensor Flow and Keras. The research design implemented
in the Raspberry Pi consists of three main processes namely : Facial Detection, Facial Feature
Extraction and Facial Emotion Classification.
 Chapter-1
INTRODUCTION
1. INTRODUCTION:
Now-a-days, Emotion recognition is a significant area in computer vision. It is computerized
software that helps to identify human feelings such as Happiness, Anger, Sadness, Fear,
Disgust, and Surprise. Moreover, emotion Recognition has its foothold on public places.
Examples of human emotions on public place like severe argument between two persons,
driving vehicle with anger and so on. By understanding emotion of the person, able to prevent
any gruesome act or danger. Human emotion recognition plays an important role in the
interpersonal relationship. Emotions are reflected from speech, hand and gestures of the body
and through facial expressions. Hence extracting and understanding of emotion has a high
importance of the interaction between human and machine communication person's mood can
be easily identified by their emotions rather than their words.

1.1 FACIAL EXPRESSIONS

Facial expressions are the fastest means of communication while conveying any type of
information. These are not only exposes the sensitivity or feelings of any person but can also
be used to judge his/her mental views. This paper includes the introduction of the face
recognition and facial expression recognition and an investigation on the recent previous
researches for extracting the effective and efficient method for facial expression recognition .

Face emotion recognition is a process of detecting human emotions from facial expressions
and this project main theme is the human brain recognizes emotions automatically and then the
software has been developed that can recognize emotions as well.

1.2 IMAGE:
An image is an artifact that depicts visual perception, such as photograph or other two-
dimensional picture, that resembles a subject usually a physical object and thus provides
a depection of it. In the context of signal processing, an image is a distributed amplitude of
colour (s). A pictorial script is a writing system that employs images as symbols for various
semantic entities, rather than the abstract signs used by alphabets. Image speaks louder than
words. An image, be it a digital or a still image, all of it is nothing but a binary representation
of some visual information. The visual information can be a simple drawing, photographed
pictures, and recorded graphs, logos or anything of this sort. All of these digital images have
something in common. They all can be stored and saved for future use electronically in any
storage device. Listeners are presented with a sample image with information inside it. This is
a digital image stored in any digital storage medium.
1.3 IMAGE PROCESSING :
Image processing is a method to perform some operations on an image, in order to get an
enhanced image or to extract some useful information from it. It is a type of signal processing
in which input is an image and output may be image or characteristics/features associated with
that image. Nowadays, image processing is among rapidly growing technologies. It forms core
research area within engineering and computer science disciplines too. Images define the
world, each image has its own story, it contains a lot of crucial information that can be useful
in many ways. This information can be obtained with the help of the technique known as image
processing. It is the core part of computer vision which plays a crucial role in many real-world
examples like robotics, self-driving cars, and object detection. Image processing allows us to
transform and manipulate thousands of images at a time and extract useful insights from them.
It has a wide range of applications in almost every field. Python is one of the widely used
programming languages for this purpose. Its amazing libraries and tools help in achieving the
task of image processing very efficiently. Through this article, you will learn about classical
algorithms, techniques, and tools to process the image and get the desired output. In digital
image processing, the same image shall be fed in as input to the system and the system shall
interpret and understand the content to let the future actions happen. The algorithms developed
shall play a major role in understanding the content with higher accuracy.

1.4 PIXEL:
One should understand that, an image is nothing but, an array or a matrix of a multiple pixels
which are properly arranged in columns and rows. Now, it is good to understand that what a
pixel is. A pixel is a word framed from picture element. A pixel is the smallest unit in a digital
image. Multiple pixels arranged in rows and columns actually form an image. An image is fully
composed of pixels.

Fig.1.1 Pixel of a n image

1.5 Types of images:
1.5.1 Binary image:

Fig.1.2 input image fig.1.3 LBP image fig.1.4 Mapped LBP

The binary image is all about 0’s and 1’s. A binary will contain only two colours. One being
white and other being black. Always the ‘black’ is represented by ‘0’ and ‘white’ is represented
by ‘1’. Each pixel in this type of image will have either of the two- 0 or 1, representing black
or white respectively. The same presented in the figure. In the binary image, each pixel needs
only the one-bit storage space. Be it white or black, what we need is just a bit to store that pixel.
This is important aspect to be remembered and this shall help in distinguish the binary image
from the black and the white image.

1.5.2 Black and white images:

Fig .1.5 black and white image

Most of the beginners shall have confusion with understanding what a binary image is and what
a black and white image is. There is a fundamental difference that differentiates these two.
When it comes to black and white image, each pixel would need 8-bit storage space. Either if
these pixels could have 0 or 1. Again, 0 represents the black and 1 represents white. Multiple
0s and 1s shall be there in an image, but the storage image requirement for the pixels is much
higher. This gives smoothness and enriched quality for the image.

1.5.3 Grayscale image:

Fig .1.6 Grayscale image

Fig.1.7 Conversion of grayscale image into array form

It is a special image which has a range of shades from the black to white colour that is, the
shades will be black and white. The most commonly used format is 8-bit format and it
accommodates 256 different shades in it. The range of this shade varies from 0-255. One should
also understand that 0 stands for black, 255 stands for white and in-between are the different
shades. A sample grayscale image is presented below ready for the reference which the shades
we discussed.

1.5.4 Colour image:

Each pixel in the colour image shall be having colour information. Each pixel in the colour
image can be composed of three channels and they most commonly regarded as R, G and B
widely which represent red, green and blue colours. In this type, one should again visualize the
image as a matrix to again better understanding. Each box (pixel) in the matrix is composed of
three components R, G, B. Each of these channels needs 8 bits for storage and hence totally, it
would become 24 bits for each pixel. Hence, it is understood that RGB, together constitute a
pixel in a colour image. The shade of the pixel would vary based on the intensity of R or G or
B.
 Fig.1.8 matrix is composed of three components R,G,B

An image is an artifact that depicts visual perception, such as photograph or other two-
dimensional picture, that resembles a subject usually a physical object and thus provides
a depection of it. In the context of signal processing, an image is a distributed amplitude of
color(s). A pictorial script is a writing system that employs images as symbols for various
semantic entities, rather than the abstract signs used by alphabets. Image speaks louder than
words. An image, be it a digital or a still image, all of it is nothing but a binary representation
of some visual information. The visual information can be a simple drawing, photographed
pictures, and recorded graphs, logos or anything of this sort. All of these digital images have
something in common. They all can be stored and saved for future use electronically in any
storage device. Listeners are presented with a sample image with information inside it. This is
a digital image stored in any digital storage medium.
 Chapter-2
LITERATURE SURVEY
&
MOTIVATION
2.1 LITERATURE SURVEY:
Face detection technology is used in many fields: entertainment, security, law
enforcement, biometrics, and more. Over the years, it has progressed from clunky
computer vision techniques all the way to advanced artificial neural networks. Face
detection has a key role in face analysis, tracking, and recognition – but the question
is, what exactly is face detection and how does it work? Read on, and we’ll break down
the term for you, provide examples, and show how it is used in today’s society.

2.1.1 WHAT IS FACE DETECTION:

Face detection is AI-based computer technology that is used to extract and identify human
faces from digital images. When integrated with biometric security systems (particularly, facial
recognition ones), this kind of technology is what makes it possible to monitor and track people
in real-time. In applications that use facial tracking, analysis, and recognition, face detection
typically works as the first step and has a significant impact on how sequential operations
within the app will perform.Face detection helps with facial analysis by identifying the parts
of a video or an image that should be focused on when determining gender, age, and emotions.
Similarly, with facial recognition systems (which create “faceprint” maps of facial features),
face detection data is included in the system’s algorithms. And why? Face detection helps
determine which parts of the video or image are needed to produce a faceprint.

2.1.2 HOW DOES FACE DETECTION WORK:

Face detection technology uses machine learning and algorithms in order to extract human
faces from larger images; such images typically contain plenty of non-face objects, such as
buildings, landscapes, and various body parts. Facial detection algorithms usually begin by
seeking out human eyes, which are one of the easiest facial features to detect. Next, the
algorithm might try to find the mouth, nose, eyebrows, and iris. After identifying these facial
features, and the algorithm concludes that it has extracted a face, it then goes through
additional tests to confirm that it is, indeed, a face.To make algorithms as accurate as
possible, they must be trained with huge data sets that contain hundreds of thousands of
images. Some of these images contain faces, while others do not. The training procedures
help the algorithm’s ability to decide whether an image contains faces, and where those facial
regions are located.Also, now would be a good time to give you definitions of the main types
of algorithms – ML, AI, and Deep Learning.

 Machine Learning (ML): ML algorithms use statistics to find patterns in huge

amounts of data. This data can include words, numbers, images, clicks, and more.
ML is the process behind many modern services – voice assistants (Siri and Alexa),
search engines (Google and Baidu), and recommendation systems (Spotify and
Netflix);
 Artificial Intelligence (AI): If an ML solution is programmed to learn how to
perform a task, rather than just simple performance, then it is AI. Systems that use
 AI demonstrate behaviors similar to human intelligence – for instance, problem
solving, planning, learning, perception, manipulation, and reasoning;
 Deep Learning: This algorithm is a subset of machine learning, and it is what
forms deep neural networks; essentially, machines are given a greater ability to find
and amplify tiny patterns. Such networks have any layers of computational nodes
that collaborate to sift through data and deliver predictions.
Now, as for the exact technologies used to develop face detection applications; these include:

 OpenCV;
 Matlab;
 Tensorflow;
 Neural Networks.
All of these follow almost the exact same procedure for face detection.

2.1.3 FACE DETECTION METHODS:

Three researchers from the University of California, David Kriegman, Ming-Hsuan Yang,
and Narendra Ahuja, published a classification of facial detection methods. There are four
classifiable categories, of which face detection algorithms can belong to 2+ groups. Let’s take
a look at each category.

Feature-Based Method:

This method located faces by extracting structural features. First, an algorithm is trained as a
classifier. Next, it is used to sort facial regions from non-facial regions. The general idea is to
move past humans’ instinctive knowledge of faces. When feature-based approaches tackle
photos with many faces, they have a 94% success rate.

Summary: Features such as a person’s nose or eyes are used to detect a face.
Knowledge-Based Method:

A knowledge-based algorithm is dependent upon a set of rules, and it is built on human

knowledge. For instance, “rules” might include that a face should have eyes, a nose, and a
mouth in certain positions relative to each other. However, this kind of method comes with
one huge challenge: it is very difficult to build an appropriate rules set. If the rules are too
general, there may be many false positives – and, conversely, if the rules are too detailed, the
system could generate many false negatives.

Summary: A face is determined based on whether it meets a set of rules made by a human.

Template Matching Method:

With a template matching algorithm, parameterized or pre-defined templates are used to locate
or detect faces – the system measures the correlation between the input photos and the
templates. For instance, the template may show that a human face is divided into nose, mouth,
eyes, and face contour regions. Also, a facial model could be comprised of just edges and use
the edge detection method – implementation of this approach is easy, but it is insufficient for
face detection.

Summary: Images are compared to standard face patterns that have been previously stored.

Appearance-Based Method:

An appearance-based algorithm uses a set of training images to “learn” what a face should look
like. In general, this method relies on machine learning and statistical analysis to determine
relevant facial characteristics. An appearance-based approach is generally considered to be
stronger than the previously mentioned methods.

Summary: Statistical analysis and machine learning are combined to find a face image’s
characteristics.

2.1.4 FACE DETECTION TECHNIQUES:

Some of the more specific facial detection techniques include:

 1. Removing the background. Let’s say an image has a pre-defined, static
background or a plain, single-color background – removing it can help
determine the face’s boundaries;

2. With color images, the color of the skin can sometimes be used to find faces;

3. Motion can be used to detect faces. In a real-time video, a person’s face is

nearly always in motion. However, a drawback of this technique is that a face
could be confused with other moving objects.

When the aforementioned strategies are combined, they can create a comprehensive face
detection approach.

2.1.5 WHAT ARE THE CHALLENGES IN FACE DETECTION :

 Odd expressions. A human face might have an odd expression, making it difficult for
facial detection algorithms to identify it as a face;
 Face occlusion. If a face is hidden by hair, a hat, a hand, glasses, or a scarf, it may result
in a false negative;
 Illuminations. An image might not have uniform lighting effects; part of the image
may be overexposed, while another part is very dark. Again, this can contribute to
false negatives;
 Complex background. When lots of objects are present in an image, face detection’s
accuracy is reduced;
 Too many faces. If there is a large number of human faces in an image, face detection
software may have a hard time distinguishing between some of them;
 Low resolution. If an image’s resolution is poor, it is more difficult to detect faces;
 Skin color. If somebody’s skin color falls outside of the gradient that is recognized by
the algorithm,their face might not be detected.

2.1.6 WHY IS FACE DETECTION IMPORTANT TODAY?

Face detection is the initial step in face analysis, face tracking, and, most importantly, face
recognition. The latter industry is growing by leaps and bounds, and is applied to device
unlocking, banking, hospitality, law enforcement, building security, and more. Face detection
is necessary for facial recognition algorithms to know which parts of an image must be used to
generate faceprints.
2. 1.7 FACE DETECTION VS. FACE RECOGNITION: WHAT’S THE
DIFFERENCE?

Facial recognition is merely one application of face detection. The for mer is used for
biometric verification and device unlocking, whereas the latter can also be applied to facial
analysis and tracking.

Classification of emotions:

Fig.2.1 Universal emotions

2.2 Motivation:

Trying to interpret a person’s emotional state in a non verbal form, usually requires decoding
his or hers facial expression. Many times, body language especially facial expressions, tell us
more than a words about one’s state of mind. For this project we have performed an experiment
which serves multiple purposes. A facial expression is a gesture executed with the facial
muscles, which convey the emotional state of the subject to observers. Am expression sends a
message about a person’s internal feeling. In Hebrew, The word for ‘face’-has the same letters
as the word represents ‘within’ or ‘inside’, that similarity implies about the facial expression
most important role being a channel of non-verbal communication. Facial expressions are
primary means of conveying non-verbal information among humans, though many animal
species display facial expressions too. Although human developed a very wide range and
powerful of verbal languages, facial expression role in interactions remains essential, and
sometimes even critical.
 Chapter-3
METHODOLOGY
3.1 Convolutional Neural Network (CNN):
A convolutional neural network (CNN) is a type of artificial neural network used in image
recognition and processing that is specifically designed to process pixel data. CNNs are
powerful image processing, artificial intelligence (AI) that use deep learning to perform both
generative and descriptive tasks, often using machine vision that includes image and video
recognition, along a recommender system. A neural network is a hardware or/ a software
patterned after the operations of neurons in the human brain. The convolutional neural network
is a class of a deep neural network. They have an application in image classification, image
segmentation, medical image analysis, natural language processing, brain-computer interfaces
and financial time series. CNN is a regularized version of multilayer perception. Multilayer
perception usually mean fully connected networks that is each neuron in one layer is connected
to all neurons in the next layer.

Fig.3.1 Block diagram of CNN

3.2 Working of CNN:

Convolutional neural network is different from multi-layer perception (MLP) as they have
hidden layers called as convolutional layers. The proposed method is based on a two-level
CNN framework. The first level recommended is background removal, used to extract
emotions from an image as shown in figure. Here the convolution neural network module is
used to extract primary expressional vector (EV). The expressional vector (EV) is generated
by tracking down relevant facial points of importance. EV is directly related to changes in
expression. The EV is obtained by using a basic perceptron unit applied on a background-
removed face image.
FIGURE-1:

Fig.3.2 A block diagram of FERC

The input image is (taken from camera or) extracted from the video. The input image is then
passed to the first-part CNN background removal. After background removal, facial
expressional vector (EV) is generated. Another CNN (the second-part CNN) is applied with
the supervisory model obtained from the ground-truth database. Finally, emotion from the
current input image is detected. Facial vectors marked on the background-removed face. Here,
nose (N), lip (P), forehead (F), eyes (Y) are marked using edge detection and nearest cluster
mapping. The position left, right and centre are represented using L, R, and C respectively.

FIGURE-2:

Fig. 3.3 Convolutional filter operation with the 3*3 kernel

Each pixel from the input image and its eight neighbouring pixels are multiplied with the
corresponding value in the kernel matrix, and finally all multiplied values are added together
to achieve the final output value.
FIGURE-3:

Fig.3.4 a .Vertical and horizontal edge detector filter matrix used at layer 1 of background-
removal CNN (first-part of CNN).

b. Sample EV matrix showing all the 24 values in the panel in top and parameter measured
at bottom.

c. Representation of point in Image domain (top panel) to Hough transform domain (bottom
panel) using Hough transform.

3.2.1 Advantages of CNN:

The main advantage of CNN compared to its predecessors is that it automatically detects the
important features without any human supervision. For example, given many pictures of
cats and dogs it learns distinctive features for each class by itself. CNN is also computationally
efficient.

3.2.2 Disadvantage of CNN:

Large training data needed, don't encode the position and orientation of object.
 Chapter-4
SYSTEM REQUIREMENT
SPECIFICATI0N
4.1 Python software:
Python is an interpreted, high-level, general-purpose programming language. Created by Guido
van Rossum and first released in 1991, Python's design philosophy emphasizes code readability
with its notable use of significant whitespace. Its language constructs and object-oriented
approach aim to help programmers write clear, logical code for small and large-scale projects.

Python is dynamically typed and garbage-collected. It supports multiple programming

paradigms, including procedural, object-oriented, and functional programming. Python is often
described as a "batteries included" language due to its comprehensive standard library .Python
was conceived in the late 1980s as a successor to the ABC language. Python 2.0, released in
2000, introduced features like list comprehensions and a garbage collection system capable of
collecting reference cycles. Python 3.0, released in 2008, was a major revision of the language
that is not completely backward-compatible, and much Python 2 code does not run unmodified
on Python .
The Python 2 language, i.e. Python 2.7.x, was officially discontinued on January 1, 2020 (first
planned for 2015) after which security patches and other improvements will not be released for
it. With Python 2's end-of-life, only Python and later are supported.

Python interpreters are available for many operating systems. A global community of
programmers develops and maintains C Python, an open source reference implementation. A
non-profit organization, the Python Software Foundation, manages and directs resources for
Python and C Python development.

4.2 Features of Python:

1. Simple : Python is a simple and minimalistic language. Reading a good
Python program feels almost like reading English, although very strict
English! This pseudo-code nature of Python is one of its greatest strengths
2. Easy to Learn: Python is extremely easy to get started with and has an
extraordinarily simple syntax
3. Free and Open Source : Python is an example of a FLOSS (Free/Libré
and Open Source Software). In simple terms, you can freely distribute
copies of this software, read it's source code, make changes to it, use pieces
of it in new free programs, and that you know you can do these things.
4. High Level: Python is a high level language as it offers high level
datastructures that reduce development time as well as code size, resulting
in more readable code. Useful types like Python's lists (resizeable arrays)
and dictionaries (hash tables) are built into the language.
5. Object Oriented : Python is a full featured object-oriented programming
language with features such as classes,inheritance, objects and overloading.
Actually it is not just OO language like Java or Ruby but a pleasant mix of
multiple programming paradigms Lisp and Haskell.
6. Scalable : Python encourages clean code design, high-level structure, and
"packaging" of multiple components, all of which deliver the flexibility,
 consistency, and faster development time required as projects expand in
breadth and scope.
7. Extensible : Python is referred as a “glue language” meaning that it is
capable to work in mixed language environment.The python interpreter is
easily extended and can add a new built-in function or modules written in
c/c++/Java code. The interface is exactly the same as for pure modules.
8. Portable : Due to its open-source nature, Python has been ported (i.e.
changed to make it work on) to many platforms. All your Python programs
can work on any of these platforms without requiring any changes at all .
Python can be used on Linux, Windows, FreeBSD, Macintosh, Solaris even
PocketPC .
9. Interpreted and (Byte-) Compiled : A program written in a compiled
language like C or C++ is translated from the source language i.e. C/C++
into a language spoken by your computer (binary code i.e. 0s and 1s) using
a compiler with various flags and options. When you run the program, the
linker/loader software just stores the binary code in the computer's memory
and starts executing from the first instruction in the program. When you use
an interpreted language like Python, there is no separate compilation and
execution steps. You just run the program from the source code. Internally,
Python converts the source code into an intermediate form called bytecodes
and then translates this into the native language of your specific computer
and then runs it. All this makes using Python so much easier. You just run
your programs - you never have to worry about linking and loading with
libraries, etc. They are also more portable this way because you can just
copy your Python program into another system of any kind and it just works!

4.3 FER library: The FER-2013 dataset was created by gathering the results of a
Google image search of each emotion and synonymsoftheemotions. The CK+ dataset has a
total of 5,876 labeled images of 123 individuals. ... Each image is labeled with one of seven
emotions: happy, sad, angry, afraid, surprise, disgust, and contempt.
Fer2013 contains approximately 30,000 facial RGB images of different expressions with size
restricted to 48×48, and the main labels of it can be divided into 7 types: 0=Angry, 1=Disgust,
2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral. The Disgust expression has the minimal
number of images – 600, while other labels have nearly 5,000 samples each.
The commonly used dataset for this image classification is FER2013 / Face Expression
Recognition which prepared by Pierre-Luc Carrier and Aaron Courville, as part of an ongoing
research project (Kaggle said).

 Happiness
 Sadness
 Fear
 Anger
 Surprise
 Disgust
Fig.4.1 emotions demo image
 Chapter-5
IMPLEMENTATION
5.1 BLOCK DIAGRAM:

Fig.5.1 Illustration of the FER system based on transfer learning in deep CNN.

Illustrates the proposed FER system with VGG-16, the well-known pre-trained DCNN model.
The available VGG-16 model is trained with the ImageNet dataset to classify 1000 image
objects. The pre-trained model is modified for emotion recognition redefining the dense layers,
and then fine-tuning is performed with emotion data. In defining the architecture, the last dense
layers of the pre-trained model are replaced with the new dense layer(s) to recognize a facial
image into one of seven emotion classes (i.e., afraid, angry, disgusted, sad, happy, surprised,
and neutral). A dense layer is a regular, fully connected, linear layer of a NN that takes some
dimension as input and outputs a vector of the desired dimension. Therefore, the output layer
contains only seven neurons. The fine-tuning is performed on the architecture having the
convolution base of the pre-trained model plus the added dense layer(s). A cleaned emotion
dataset prepared through preprocessing (i.e., resizing, cropping, and other tasks) is used to train
in fine-tuning. In the case of testing, a cropped image is placed to the input of the system, and
the highest output probability of emotion is considered to be the decision. VGG-16 may be
replaced with any other DCNN models, e.g., ResNet, DenseNet, Inception. Therefore, the size
of the proposed model depends on the size of the pre-trained model used and the architecture
of the added dense layer(s).
5.2 PROJECT DESIGNING/CODE:5.3 Working :

Coverts image into an array of pixel values . where the dimentions of array depends on the
resolutions of image
Array automatically generated by computer from input image

5.4 Results:

Emotion : happy
EMOTION: sad

EMOTION: netural
 Chapter:6
CONCLUSTION
6.1 Applications:
Face detection technology is used in many fields: entertainment, security, law enforcement,
biometrics, and more. Over the years, it has progressed from clunky computer vision
techniques all the way to advanced artificial neural networks. Face detection has a key role in
face analysis, tracking, and recognition – but the question is, what exactly is face detection and
how does it work? Read on, and we’ll break down the term for you, provide examples, and
show how it is used in today’s society.

6.1.1 Facial Emotion Detection in Interviews:

A candidate-interviewer interaction is susceptible to many categories of judgment and
subjectivity. Such subjectivity makes it hard to determine whether candidate's personality is a
good fit for the job. Identifying what a candidate is trying to say is out of our hands because of
the multiple layers of language interpretation, cognitive biases, and context that lie in between.
That's where AI comes in, which can measure candidate's facial expressions to capture their
moods and further assess their personality traits.
Notably, Unilever is already starting to incorporate this technology into their recruitment
process. With this technology, a recruiter will be able to know, say, the overall confidence level
of an interviewee and make a decision about whether or not this candidate will be able to
perform well at a client-facing job. Similarly, it will be possible to find whether the candidate
is honestly replying to all the questions by measuring the change in emotions during his
responses and correlating it the vast amount of knowledge available in this area.
Employee morale can also be perceived using this technology by holding and recording
interactions on the job. As an HR tool, it can help not only in devising recruiting strategies but
also in designing HR policies that bring about best performance from employees.

6.1.2 Testing for Video Games:

Video games are designed keeping in mind a specific target audience. Each video game aims
to evoke a particular behavior and set of emotions from the users. During the testing phase,
users are asked to play the game for a given period and their feedback is incorporated to make
the final product. Using facial emotion detection can aid in understanding which emotions a
user is going through in real-time as he is playing without analyzing the complete video
manually.
Such product feedback can be taken by analyzing a live feed of the user and detecting his facial
emotions. While feelings of frustration and anger are commonly experienced in advanced video
games, making use of facial emotion detection will help understand which emotions are
experienced at what points in the game. It is also possible that some unexpected or undesirable
emotions are observed during the game. Taking feedback from the user has experienced the
game can be inefficient. This is because it can often be difficult to put an experience into words.
Moreover, users may be unable to remember what exactly they went through emotionally
across different parts of the game. Facial emotion detection is a practical means of going
beyond the spoken or written feedback and appreciating what the user is experiencing. When
feedback is taken in this format, it becomes genuinely non-intrusive when it comes to user
experience. At the same time, such feedback is more reliable than other forms.
6.1.3 Market Research:
Traditional market research companies have employed verbal methods mostly in the form of
surveys to find the consumers wants and needs. However, such methods assume that consumers
can formulate their preferences verbally and the stated preferences correspond to future actions
which may not always be right.
Another popular approach in market research industry is to employ behavioral methods that let
users act while trying the product. Such methods are considered more objective than verbal
methods. Behavioral methods use video feeds of a user interacting with the product, and the
video is then analyzed manually to observer user's reactions and emotions. This can quickly
become very labor intensive increasing the overall cost. Facial emotion recognition AI can
automatically detect facial expressions on user's faces and automate the video analysis
completely. Market research companies can use this technology to scale the data collection
efforts and rely on the technology to do analysis quickly.
Detecting emotions with technology is a challenging task, yet one where machine learning
algorithms have shown great promise. Using ParallelDots' Facial Emotion Detection API,
customers can process images, and videos in real-time for monitoring video feeds or
automating video analytics, thus saving costs and making life better for their users. The API is
priced on a Pay-As-You-Go model, allowing you to test out the technology before scaling up.
Facial Emotion detection is only a subset of what visual intelligence could do to analyze videos
and images automatically.

6.2 ADVANTAGES:
As a key element in facial imaging applications such as facial recognition, facial analysis and
face detection creates various advantages for users, including:
Improved security: Face detection improves surveillance efforts and helps track down
criminals and terrorists. Personal security is also enhanced since there is nothing for hackers to
steal or change, such as passwords.
Easy to integrate: Face detection and facial recognition technology is easy to integrate and
most solutions are compatible with the majority of the security software.
Automated identification: In the past, identification was manually performed by a person;
this was inefficient and frequently inaccurate. Face detection allows the identification process
to be automated, thus saving time and increasing accuracy.

6.3 DISADVANTAGES:
While face detection provides several large benefits to users, it also holds various
disadvantages, including:
Massive data storage burden: The ML technology used in face detection requires powerful
data storage that may not be available to all users.
Detection is vulnerable: While face detection provides more accurate results than manual
identification processes, it can also be more easily thrown off by changes in appearance or
camera angles.
A potential breach of privacy: Face detection's ability to help the government track down
criminals creates huge benefits; however, the same surveillance can allow the government to
observe private citizens. Strict regulations must be set to ensure the technology is used fairly
and in compliance with human privacy rights.

6.4 CONCLUSION:
Facial expressions can display personal emotions and indicate an individual's intentions within
a social situation. They are extremely important to the social interaction of individuals.
Background scenes in which faces are perceived provide important contextual information for
facial expression processing. Facial expression recognition in the real-world case is a long-
standing problem. The low image quality, partial occlusions, and illumination variation in the
real-word environment make the feature extraction process more challenging. In this paper, we
exploit both texture and geometric features for effective facial expression recognition.

6.5 FUTURE SCOPE:

In future, we can extend the work in developing an FER system to recognise different other
facial or sign expressions rather than the basic expressions. For instance, exhaustion,
frustration, anticipation, aggressiveness and so on helps in real-time applications. The present
system uses available datasets that limit the efficiency when applied the same in real time. It is
a known fact that the available facial expression datasets are not only frontal-view and also
pre-processed under the controlled environment. Moreover, they have still image and image
sequences are of fixed sizes, which enable the feature extraction process simple whereas in the
real-time environment it's not possible. To cope with existing datasets and in a real-time
environment, compatible new techniques should extensively develop from the baseline.
CHAPTER-7
REFERENCES
7.1 REFERENCES:
 A literature survey on Facial Expression Recognition using Global Featuresby
Vaibhavkumar J. Mistry and Mahesh M. Goyani,International Journal of Engineering
and Advanced Technology (IJEAT),April,2013
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.645.5162&rep=rep1&typ
e=pdf]

 Convolutional Neural Networks (CNN) With TensorFlow by Sourav from

Edureka[https://www.youtube.com/watch?v=umGJ30-15_A]
 A Survey on Facial Expression Recognition Techniques by Swati
Mishra(Chhattisgarh Swami Vivekanand Technical University, Department of
Computer Science & Engineering, Raipur Institute of Technology, Mandir Hasaud,
Raipur, Chhattisgarh, India) andAvinash Dhole( Assistant Professor, Department of
Computer Science & Engineering, RaipurInstitute of Technology, Chhattisgarh
Swami Vivekanand andTechnical University, Mandir Hasaud, Raipur, Chhattisgarh,
India) , International Journal of Science and Research (IJSR),2013
[https://pdfs.semanticscholar.org/e241/25e4e9471a33ba2c7f0979541199caa02f8b.pd
f]
 Recognizing Facial Expressions Using Deep Learning by Alexandru Savoiu
Stanford University and James Wong Stanford University
[http://cs231n.stanford.edu/reports/2017/pdfs/224.pdf]

 Deep Learning Simplified by Sourav from Edureka

[https://www.youtube.com/watch?v=dafuAz_CV7Q&list=PL9ooVrP1hQOEX8BKD
plfG86ky8s7Oxbzg]

 Predicting facial expressions with machine learning algorithms by Alex Young ,

Andreas Eliasson , Ara Hayrabedian , Lukas Weiss , Utku
Ozbulak[https://github.com/utkuozbulak/facial-expression-recognition]
  “Robust Real-Time Face Detection”, International Journal of Computer Vision
57(2), 137–154, 2004
 “Facial expressions of emotions: an old controversy and new finding
discussion”, by P. Ekman, E. T. Rolls, D. I. Perrett, H. D. Ellis, Pill Trans. Royal
Soc. London Ser. B, Biol. Sci., 1992, vol. 335, no. 1273, pp. 63-69.
 Going Deeper in Facial Expression Recognition using Deep Neural Networks, by
Ali Mollahosseini1, David Chan2, and Mohammad H. Mahoor1 Department of
Electrical and Computer Engineering, Department of Computer Science, University

7.2 WEB REFERENCES:

https://www.w3schools.com/python/python_ml_mean_median_mode.asp
https://www.tutorialspoint.com/python/python_xml_processing.htm
https://www.javatpoint.com/python-files-io