Proceedings of the 12th INDIACom; INDIACom-2018; IEEE Conference ID: 42835

2018 5 International Conference on “Computing for Sustainable Global Development”, 14 th - 16th March, 2018
th

Bharati Vidyapeeth's Institute of Computer Applications and Management (BVICAM), New Delhi (INDIA)

Emotion Detection using Image Processing in Python

Raghav Puri Archit Gupta Manas Sikri
Electronics & Communication Electronics & Communication Electronics & Communication
Engineering Engineering Engineering
Bharati Vidyapeeth’s College of Bharati Vidyapeeth’s College of Bharati Vidyapeeth’s College of
Engineering Engineering Engineering
New Delhi, India New Delhi, India New Delhi, India
raghavpuri31@gmail.com archit.gupta33@gmail.com manassikri04@gmail.com

Mohit Tiwari, Nitish Pathak, ShivendraGoel,

Assistant Professor, Assistant Professor, Associate Professor,
Bharati Vidyapeeth’s College of BVICAM BVICAM
Engineering New Delhi, India New Delhi, India
New Delhi, India nitishforyou@gmail.com shivendragoel@gmail.com
mohit.tiwari@bharatividyapeeth.edu

Abstract—In this work,user’s emotion using its facial The method involves converting an image into a 2D Matrix.
expressions will be detected. These expressions can be derived
from the live feed via system’s camera or any pre-existing image It consists of three basic steps [2]:
available in the memory.Emotions possessed by humans can be 1.) Scanning the image: a raw image is acquired which
recognized and has a vast scope of study in the computer vision has to be processed. It can be expressed in form of pixels as
industry upon which several researches have already been done. stated above.The aim of this step is to extract information
The work has been implemented using Python (2.7), Open Source
which is suitable for computing.
Computer Vision Library (OpenCV) and NumPy. The scanned
image (testing dataset) is being compared to training dataset and 2.) Processing and Enhancing it: -the image is
thusemotion is predicted. The objective of this paper is to develop converted into digital form by using a digitizer which samples
a system which can analyze the image and predict the expression and quantizes the input signals. The rate of sampling should be
of the person. The study proves that this procedure is workable high for good resolution and high quantization level for human
and produces valid results. perception of different shades using different using gray-scale
Keywords—Face Recognition, Image Processing, Computer 3.) The obtained result describes the property of the
Vision, Emotion Detection, OpenCV image and further classifies the image.
Conversion of Color Image to Gray Scale
I. INTRODUCTION TO IMAGE PROCESSING There are basically two methods to convert a color image to a
gray scale image [8]:
In order to get an enhanced image and to extract some A.) Average Method
useful information out of it, the method of Image Processing
can be used. It is a very efficient way through which an image In Average method, the mean is taken of the three colors i.e.
can be converted into its digital form subsequently performing Red, Blue & Green present in a color image. Thus, we get
various operations on it. This is a technique similar to signal Grayscale= (R+G+B)/3;
processing, in which the input given is a 2D image, which is a
collection of numbers ranging from 0 to 255 which denotes the But what happens sometimes is instead of the grayscale image
corresponding pixel value. [8] we get the black image. This is because we in the converted
image we get 33% each of Red, Blue & Green.
TABLE 1:THIS IS A 2D ARRAY DEPICTING THE PIXELS OF A
SAMPLE IMAGE Therefore, to solve this problem we use the second method
called Weighted Method or Luminosity Method.
134 21 107
64 37 78
B) Weighted or Luminosity Method
42 4 13
To solve the problem in Average Method, we use Luminosity
method. In these method, we decrement the presence of Red

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1389

 Proceedings of the 12th INDIACom; INDIACom-2018; IEEE Conference ID: 42835
2018 5 International Conference on “Computing for Sustainable Global Development”, 14 th - 16th March, 2018
th

Color and increment the color of Green Color and the blue GPU coding. In GPU Module, we cannot change the features
color has the percentage in between these two colors. of an image directly, rather we have to copy the original image
followed by editing it.
Thus, by theequation [8],
Fig.2 Flowchart for Emotion Algorithm [13]
Grayscale=((0.3 * R) + (0.59 * G) + (0.11 * B)).
We use this because of the wavelength patterns of these colors.
Blue has the least wavelength while Red has the maximum
wavelength.
II. Review of Literature
1. Different Strengths & Weaknesses were noted in the
emotion recognition system by Mr. Ashwini, Mr. Jacob and
Dr. Jubilant [14] of St. Joseph’s College of Engineering.
2.Mr. ArunaChakraborty, Mr. Amit Konar, Mr. Uday
Kumar Chakraborty, and Mr. Amita Chatterjee in. Emotion
Recognition from Facial Expressions and Its Control Using
Fuzzy Logic [15] explained that this fuzzy approach that
has accuracy of about 90%.
3. Strengths and weaknesses of facial expression classifiers
and acoustic emotion classifiers were analyzed by Carlos
Busso, Zhigang Deng, SerdarYildirim, MurtazaBulut, Chul
Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann
and Shrikanth Narayanan. [1].

I. IV. STEPS INVOLVED IN INSTALLING PYTHON 2.7 AND THE

NECESSARY PACKAGES
Let’s begin with a sample of image in either .jpg or .png
format and apply the method of image processing to detect
emotion out of the subject in the sample image. (The word
‘Subject’ refers to any living being out of which emotions can
Fig.1 Data Recording System [16] be extracted).
III. Introduction to OpenCV A. Importing Libraries
OpenCV is Open Computer Vision Library [4]. It is a free For successful implementation of this project, the following
for all extensive library which consists of more than 2500 packages of Python 2.7 have to be downloaded and installed:
algorithms specifically designed to carry out Computer Vision Python 2.7.x, NumPy, Glob and Random. Python will be
and Machine Learning related projects. These algorithms can installed in the default location, C drive in this case. Open
be put to use to carry out different tasks such as Face Python IDLE, import all the packages and start working.
Recognition, Object Identification, Camera Movement
Tracking, Scenery Recognition etc. It constitutes a large B. NumPy
community with an estimate of 47,000 odd people who are  NumPy is one of the libraries of Python which is used
active contributors of this library. Its usage extends to various for complex technical evaluation. It is used for
companies both, Private and Public. implementation of multidimensional arrays which
consists of various mathematical formulas to process.
A new feature called GPU Acceleration [12] was added among
[9].
the preexisting libraries. This new feature can handle most of
the operations, though it’s not completely advanced yet. The  The array declared in a program has a dimension which
GPU is run by using CUDA and thus takes advantages from is called as axis.
various libraries such as NPP i.e. NVIDIA performance
primitives. Using GPU is beneficial by the fact that anyone can  The number of axis present in an array is known as rank
use the GPU feature without having a strong knowledge on

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1390

 Emotion Detection using Image Processing in Python

 For e.g. A= [1,2,3,4,5] for name in glob.glob(‘direc/filename?’)

print name
 In the given array A 5 elements are present having rank
1, because of one-dimension property. Result=>
direc/filename1
 Let’s take another example for better understanding direc/filename2
 B= [[1,2,3,4], [5,6,7,8]] direc/filename3
direc/filename4
 In this case the rank is 2 because it is a 2-dimensional direc/filename5
array. First dimension has 2 elements and the second
dimension has 4 elements. [10] This wildcard is limited to one specificdirectory and doesn’t
extend itself. i.e., itdoesn’t find a file in a subdirectory.[5]
C. Glob
D. Random
On the basis of the guidelines specified by Unix Shell, the
Glob module perceives the pattern and with reference to it, Random Module picks or chooses a random number or an
generates a file. It generates full path name. [3] element from a given list of elements. This module supports
those functions which provide access to such operations.
1) Wildcards
Classification of Random Module: -
These wildcards are used to perform various operations on 1) randint(m,n)
files or a part of directory. There are various wildcards [5]
which are functional out of which only two are useful: - It returns a value of x such that-
TABLE 2: Various files created in a Directory m<=x<=n
2) randrange(dog, cat, mouse, lion)
List of all files/working material saved inside a
directory named “direc” It returns any random variable or
direc/filename1List of all files/working material saved element from the given range [7].
inside a directory named “direc”
direc/filename1
direc/filename2
direc/filename3
direc/filename4
direc/filename5
direc/files

a) Asterisk(*): It represents any number of characters

with any combination
For eg.
import glob
for name in glob.glob(‘direc/file*’)
print name

Result=>
direc/filename1
direc/filename2
direc/filename3
direc/filename4
direc/filename5
direc/files
Fig.3 Flowchart for Emotion Detection
b) Question Mark(?): It represents or
finds a single missing character

For e.g.
import glob

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1391

 Proceedings of the 12th INDIACom; INDIACom-2018; IEEE Conference ID: 42835
2018 5 International Conference on “Computing for Sustainable Global Development”, 14 th - 16th March, 2018
th

II. V. DIFFERENT EMOTIONS THAT CAN BE DETECTED OUT OF AN

IMAGE:

A. A. Neutral
B. B. Happy
C. C. Anger
D. D. Disgust
E. Surprise
F. Fear
G. Sad
H. Contempt
Fig.4 Output of a sample image [9]

III. VI. STEPS INVOLVED TO PERFORM EMOTION DETECTION VII. Cohn-Kanade AU-CODED EXPRESSION DATABASE
USING OPENCV-PYTHON:
This is one of the Databases that can be used for emotion
1)After successfully installing all the necessary softwares, we detection. It’s a Database having numerous types of emotions
must start by creating a Dataset. Here, we can create our own [6]. This database is currently available in two different
dataset by analyzing group of images so that our result is categories while the third one is under development stage.
accurate and there is enough data to extract sufficient
information. Or we can use an existing database. The first version is labeled by “CK” while the second version
is labeled as “CK+”. The CK version starts by detecting the
2)The dataset is then organized into two different
neutral emotion and then advances to the other higher
directories. First directory will contain all the images and the
emotions.
second directory will contain all the information about the
different types of emotions. With the release of second type of this database, the frequency
3)After running the sample images through the of processes increased by approximately 25% and the
python code, all the output images will be stored into another frequency of subjects by approximately 30% [6].
directory, sorted in the order of emotions and its subsequent
encoding. The third release is on the verge of development as which will
have both the features of CK and CK+ Database, for e.g.
4)Different types of classes can be used in OpenCV for adding integrated 30 degrees of rotation from the front.
emotion recognition, but we will be mainly using Fisher Face
one. [1] IV. VIII. APPLICATIONS AND FUTURE SCOPE
5)Extracting Faces:OpenCV provides four predefined Computer Vision is a very vast field which is still under
classifiers, so to detect as many faces as possible, we use these developmental phase. Research work in this field is going at a
classifiers in a sequence [1] rapid phase.
6)The dataset is split into Training set and Classification set.
The training set is used to teach the type of emotions by Emotion detection is an inseparable part of computer vision.
extracting information from a number of images and the Loads of tasks and processes can be performed if one can
classification set is used to estimate the classifier performance. become aware about the intricacies and endless possibilities
7)For best results, the images should be of exact same offered under the field of emotion detection.
properties i.e. size.
Some common and widely used applications of emotion
8)The subject on each image is analyzed, converted to detection are:
grayscale, cropped and saved to a directory [1]
9)Finally, we compile training set using 80% of the test data A. App and product development
and classify the remaining 20% on the classification set. Emotion recognition can play a huge role in optimizing
Repeat the process to improve efficiency [1]. various software engineering processes which comprises of
testing of ease with which a product can be used. It’s a long-
established fact that level of comfort with different software
products depends hugely upon human emotions. A products
overall look and feel can also alter human feelings which in
turn makes the person buy the product or not. Thus,

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1392

 Emotion Detection using Image Processing in Python

researching about different emotional states of a human body We humans also sometimes make a mistake while recognizing
and how it is influenced by usage of different products is a someone’s emotion so is our program. The optimum accuracy
matter of prime importance for related industry professionals. was nearly 83%.

B. Improved learning practices In Present Work, we recognized different types of human

Evidence suggests that part of emotional states vouches for emotions using Python 2.7, OpenCV & (CK and CK+)
better learning practices while the other part try to suppress Database [6] and got some interesting insight about it.
them. The difference between the two groups possessing
REFERENCES
different emotional states is not so common to find. For
[1] V Gent, P. (2016). Emotion Recognition with Python, OpenCV and a
example, positive emotional state is thought to be bad for Face Dataset. A tech blog about fun things with Python and embedded
learning purposes while slightly negative emotional state electronics. Retrieved from:
fosters analytical thinking and is also appropriate for carrying http://www.paulvangent.com/2016/04/01/emotion-recognition-with-
out critical tasks. python-opencv-and-a-face-dataset/
[2] https://www.engineersgarage.com/articles/image-processing-tutorial-
C. Improvised web development applications
With the mammoth scale at which the internet is expanding, [3] https://docs.python.org/2/library/glob.html
service providers are interested in collecting tons and tons of [4] https://opencv.org/
data which can be extracted from the users. Correspondingly,
[5] http://www.tldp.org/LDP/GNU-Linux-Tools-
all the content and advertisements are played based on the
users’ profile. Subsequently, adding intricate details about the Summary/html/x11655.htm
different human emotions can provide much more precise [6] - Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database
behavioral models of different types of users. for facial expression analysis. Proceedings of the Fourth IEEE
International Conference on Automatic Face and Gesture Recognition
D. Immersive gaming (FG'00), Grenoble, France, 46-53.
Video games constitute a large chunk of entertainment - Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., &
industry. Thus, in order to make these games much more Matthews, I. (2010). The Extended Cohn-Kanade Dataset (CK+): A
complete expression dataset for action unit and emotion-specified
intensive and intrusive, video game creators base their research
expression. Proceedings of the Third International Workshop on CVPR
on different types of human emotions commonly found. In
for Human Communicative Behavior Analysis (CVPR4HB 2010), San
order to allure more and more players, video games are made Francisco, USA, 94-101
in such a way that they incorporate human emotions naturally
[7] http://docs.python.org/3.4/library/random.html
into their game play.
[8] https://www.tutorialspoint.com/dip/
IX. Challenges
[9] https://pshychmnemonics.wordpress.com/2015/07/03/primary-emtions
The main purpose is to detect various emotions in a given [10] https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
sample image. The most challenging part in this task was to
[11] Kanade, Cohn, &Tian (2000) and Lucey et al. (2010) in any paper of
determine the exact emotion when two emotions look quite mine or my collaborators that makes any use of the database. The
similar, for e.g. “Disgust” being classified as “Sadness”, references are:
“Surprise” like “Happy” and so on. Now for eight different - Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database
categories, the result was approximately 70% accurate which is for facial expression analysis. Proceedings of the Fourth IEEE
quite well actually as our classifier learned quite a bit. So, we International Conference on Automatic Face and Gesture Recognition
must see how can we increase its efficiency and accuracy. (FG'00), Grenoble, France, 46-53.
- Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., &
If we look at our emotions list we can find out that we have Matthews, I. (2010). The Extended Cohn-Kanade Dataset (CK+): A
only limited number of examples for “sad”, “fear” and complete expression dataset for action unit and emotion-specified
“contempt”. By increasing the number of images for these expression. Proceedings of the Third International Workshop on CVPR
emotions we can certainly increase optimization, or if we no for Human Communicative Behavior Analysis (CVPR4HB 2010), San
longer consider these emotions in the list then optimization can Francisco, USA, 94-101.
be increased more than 80%. [12] Kari Pulli, Anatoly Baksheev, Kirill Kornyakov, and Victor Eruhimov.
2012. Realtime Computer Vision with OpenCV. Queue 10, 4, Pages 40
The dataset that we used for our task was Cohn-Kanade (CK (April 2012), 17 pages. DOI: http://dx.doi.org/10.1145/2181796.2206309
and CK+) Database. [6] [13] Kim &Joo, Young Hoon&Bae Park, Jin. (2015). Emotion Detection
Algorithm Using Frontal Face Image
V. X. CONCLUSION
[14] Ashwini Ann Verghese, Jacob P Cherian & Jubilant J Kizhakkethottam
Artificial Intelligence can be used to solve intriguing tasks 2015.Overview on Emotion Recognition System.INSPEC Accession
such as emotion detection, although this task was quite Number: 15508917.
convolute even more when using a great number of images.

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1393

 Proceedings of the 12th INDIACom; INDIACom-2018; IEEE Conference ID: 42835
2018 5 International Conference on “Computing for Sustainable Global Development”, 14 th - 16th March, 2018
th

[15] ArunaChakraborty, Amit Konar,Uday Kumar Chakraborty, and Amita

Chatterjee (2009). Emotion Recognition FromFacial Expressions and Its
Control Using Fuzzy Logic. INSPEC Accession Number: 10705009
[16] Carlos Busso, Zhigang Deng, SerdarYildirim, MurtazaBulut, Chul Min
Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann andShrikanth
Narayanan (2004). Analysis of Emotion Recognition using Facial
Expressions, Speech and Multimodal Information

Copy Right © INDIACom-2018; ISSN 0973-7529; ISBN 978-93-80544-28-1 1394