See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/359046281

Image Processing Based Smart Home Automation using Raspberry Pi and

Python

Preprint · March 2022

CITATIONS READS

0 2,015

3 authors, including:

Ahmed J Jameel

84 PUBLICATIONS   384 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Performance Comparison of Convolutinal Codes & Turbo Codes View project

Single Sideband (SSB) View project

All content following this page was uploaded by Ahmed J Jameel on 20 March 2022.

The user has requested enhancement of the downloaded file.

 Image Processing Based Smart Home Automation
using Raspberry Pi and Python
Rima Idris Aysha Alkooheji Ahmed J. Jameel
Telecommunication Engineering Computer Engineering Department Telecommunication Engineering
Department, Ahlia University Ahlia University Department, Ahlia University
Manama, Bahrain Manama, Bahrain Manama, Bahrain
Rima.idris.2@gmail.com ayshakoo@hotmail.com adulaimi@ahlia.edu.bh

Abstract—The latest advances in computer vision are • Allow for wireless control over the door lock using an
making our life more and more automated; its algorithms Android mobile application developed specifically for this
can run homes, drive cars, make diagnoses, and assist system.
people in many other ways. Home automation is becoming one
of the most increasingly growing topics nowadays since it can
• Obtain the body temperature of the person standing by the
provide us with methods that can make our lives easier and door as a form of COVID-19 precautionary measure. This
better in quality. This system aims to increase the safety rate in feature will be operating in two operational modes.
homes by exposing potential intruders that might try to get in
private properties. It does that by using image processing II. LITERATURE SURVEY
algorithms that are going to analyze the face of whoever is by the A. Existing Techniques
door whenever motion is detected. If the face is authorized, the Many techniques have been innovated to create home
door will automatically open. Whereas any unauthorized faces automated systems. B. Shinde et. al. in [1] has developed a
will lead the system to initiate an automatic video call with the
home automation system using an Arduino microcontroller,
user to see who is by the door. The system also allows wireless
home automation by providing the choice to lock/unlock the door along with an android mobile application that sends
through a mobile application. Moreover, the application will commands to the Arduino which will control the appliances
display the body temperature retrieved from IR contactless accordingly. Both are connected to each other via Wi-Fi. In
temperature sensor that is added to provide COVID-19 [2], Krishna Rathi et. al. made use of hand gestures to control
precautionary measures. To increase the usage range of this the appliances, by using an accelerometer, Raspberry pi, flex
system, it consists of two operational modes controlling the sensor and an Arduino kit along with a hand gesture
functionality of the system in relation with the body temperature recognition algorithm, a person’s gestures was used to control
measured. the appliances at home. Whereas Saddam in [3], made use of
the DTMF technology, this is where each number on the
Keywords—Home automation, face recognition.
keypad has two frequencies, so by using a DTMF decoder, we
I. INTRODUCTION can assign each number a command that must be performed
when pressed. For merely security purposes, B Madhuravani.
Our houses, offices and other private properties are always at
et. al in [4], developed a system using Raspberry Pi, where
risk of being attacked by intruders that would be accessing
When motion is detected, the face will be captured and
them for many different reasons, such as theft, damaging
processed by a face recognition algorithm, which will alert by
property etc. Hence, restricting access to our properties is very
a buzzer and sending an email/SMS, if the face is not
crucial and would surely protect against intruders. One of the
recognized. Speech recognition technique was implemented
most efficient ways is the use facial recognition in smart home
by Nainsi Soni. et. al. in [5]. This system allows total control
applications. Facial recognition can be implemented by using
on home appliances by speech commands only. The process
various algorithms that would be able to identify and highlight
of achieving this technique needs signal pre-processing,
specific facial characteristics. Especially, during these
feature extraction from the signal, language and pronunciation
ongoing circumstances involving Covid-19; it is a much better
modelling and finally decoding the message. Utilizing such
option than anything that would include touching such as
system in home automation can be extremely helpful for
using pins or fingerprints. Implementing such a method in a
people who might struggle with blindness or immobility.
system that could be connected to the internet was made
Sudhasmita Behera. et.al in [6] designed a system that
possible by using the Internet of Things technology. It is a
controls home appliances using SMS messages in which GSM
technology of wirelessly interconnected devices over a
wireless technology was used. First, the user sends SMS
network that can communicate with each other in real-time
message to the system with the desired commands that needs
using the internet without any human intervention. In other
to be done. Second, the system stores this message and
words, it can be considered as machine-to-machine
generates control signal. Finally, the signal is sent to the
communication. This technology can collect data and
hardware that was programmed to perform these commands.
information, analyze it and then perform an action based on
LabVIEW is a programming software that provides graphical
the information gathered. This goal of this paper is to design
approach to develop applications that requires control and
a low-cost intrusion detection system using a Raspberry Pi 4
measure. Due to its features and the ability to visualize every
that is capable of the following:
part of the program, S Jermilla. et. al in [7] used it in a home
• Executing detection/recognition algorithms upon motion
automation project. In [8] Idris used an MLX90614 Non-
detection and controlling the door look according to its
contact Infrared temperature sensor to retrieve the body
results.
temperatures.
• Act as a smart intercom system and initiate an automatic
video call with the owner of the property.
B. Face detection and recognition input port that allows users to record HD video and high-
Facial detection is the ability of the system to detect the resolution photos. Using Python and specific libraries
presence of humans or faces in a photo using algorithms and written for the Pi, users can create tools that take photos and
formulas. There are several different approaches and video, and analyze them in real-time or save them for later
techniques to perform the face detection. First, the feature- processing. Thus, it will allow us to implement a facial
based face detection which was used by Tomasz Orczyk and detection and recognition algorithms and control the door lock
Piotr Porwik in [9], this depends on the face features such as based on that. We chose to work with Raspberry Pi4 due to
nose and eyes to detect the human’s face. Second approach is several reasons. First, it contains 1.5 GHz 64-bit quad core
ARM Cortex-A72 processor which makes it faster than the
the geometric base face detection, it is based on the geometric
previous models and this will provide higher performance for
structure of the face image. V. Starovoitov and D. Samal used
video calls. Second, the gigabit Ethernet that ensures higher
this technique in [10]. Finally, the Haar Cascade classifier speed network connections. Finally, it has a pair of micro-
algorithm which was implemented by Senthamizh HDMI ports that support 4k resolution for video streaming.
Selvi.R.et.al for criminal identification purposes [11].
Whereas Facial recognition is the ability to recognize a new 2. PIR motion sensor
input face image depending on the closest match from the Passive Infrared sensors can detect the changes in the level of
stored database. There are multiple face recognition infrared radiations. Generally, all objects emit infrared
algorithms that are constantly being improved aiming to find radiations, but the strength of this radiation varies depending
the best approach to identify faces. Eigenface algorithm uses on the temperature, so when a human pass by the sensor they
linear algebra concepts to recognize faces. In [12] Marijeta will be detected because of the difference in radiation between
Slavković.et.al implemented this algorithm for a face the human body and other objects in the background. We used
recognition system. In Fisherface algorithm, the classes of the this sensor to build our system with two operational modes:
samples are taken under consideration. The aim of this In the first operational mode whenever motion is detected the
algorithm is to maximize the scatter between classes and temperature will be measured and sent to the application to
minimize the scatter within each. Local Binary Pattern warn the user. If the face is recognized by the algorithm as an
Histogram (LBPH) that was introduced in 1994, is considered authorized user, the door will automatically open even if the
to be one of the simplest face recognition algorithms. Farah temperature is high. This application is useful for homes and
Deeba.et.al developed an LBPH-based Enhanced Real-Time personal properties because even if the user’s body
Face Recognition system in [13]. temperature is high, they still need to enter their own house.
The second operational mode is suitable for offices and
workplaces. Whenever motion is detected, the temperature
III. METHOLODGY sensor will measure the temperature of the visitor and send it
This system consists of both hardware and software to the user. If the temperature exceeds the limit which is 37.5°,
components. The hardware components consist of all the the door will not open automatically even if the user was
electrical components required to run the system. Whereas the recognized by the algorithm. This will ensure that no one with
software part consists of all the algorithms and programs
required to run the main functions of the system. high fever enters the building to reduce the risk of having a
COVID-19 infection in the organization.

3. MLX90614 Non- Contact infrared temperature sensor

It is a contactless IR temperature sensor that can be used to
measure the specific temperature of objects rather than
ambient temperatures.

4. Webcam
A webcam camera will be used to enable the face detection,
recognition, and the video call.

5. Solenoid door lock and Relay Switch

The solenoid lock has a latch for electrical locking and
unlocking. The relay switch is an electromagnetic switch that
converts small input current into larger currents. It is used to
switch on or off a circuit, it can be used also as an amplifier.
In this system we are using it to turn on/Off the electric door
Fig 1: System architecture
strike.
A. Hardware design
B. Software design
1. Raspberry Pi 4
Raspberry Pi is a computer that is built on a single circuit The operating system running the Raspberry pi will be the
board. It contains all the component that is needed to create a official Raspbian Operating system, as it the most widely
functional computer such as microprocessors, input/output adopted. Python programming language is used for running
ports and memory. The Raspberry Pi has a dedicated camera the functions on the Raspberry Pi, and JAVA will be used in
Android Studio. The software components of the system are as
the following:
1. Image Processing Algorithm
a) Facial detection and recognition:
It is based on the Viola-Jones detection algorithm [14] in
which the cascade function is trained using a sample that
contains a lot of positive (containing faces) and negative (does
not contain any faces) images. It is a machine learning based Fig 2: Communication design
approach. It can be used on Raspberry Pi through installing
OpenCV. This algorithm is known to have four steps and they
are as the following:[15] 4. Video call initiation using Google Duo and the
i. Haar feature selection Pymouse and Pykeyboard libraries:
ii. Integral images To provide a high-quality Video over IP through which the
iii. Adaboost training user can see and talk to whoever is standing by the door,
iv. Cascade classifier Google Duo will be used. Google Duo is a video chat mobile
A script would initiate the camera and start the video app developed by Google, available on the Android and iOS
capturing. It will run the Haar cascade algorithm and detect operating systems [19]. It was picked due to its flexibility to be
the face in the video. Upon detection it will ask the user to installed on all devices and the high quality it provides. It can
enter an ID that they would like to associate with the face and be made possible by installing the Pymouse and Pykeyboard
then proceed to take 30 pictures of the face and save them into libraries. These libraries will allow the system to calculate
the dataset directory where they will be used for further cursor positioning on the desktop, and then move the cursor,
processing. click and type automatically without any human intervention.
In other words, it will allow the system to open google duo,
b) Local Binary Pattern Histogram type the contact’s name, and dial them all by itself. [20]
It is the most popular recognition algorithm. It can be used on
Raspberry Pi through installing OpenCV. The purpose of it is IV. RESULTS
that it creates an intermediate image that highlights the facial A. Circuit design
characteristics more efficiently[16]. By using a sliding window To build this project, multiple hardware components had to be
of 3x3 pixels, it creates matrices that with certain calculations
connected to the Raspberry Pi. First, the PIR sensor that is
will produce certain values that will represent a single pixel
used to detect any motion was connected directly to the GPIO
from the picture. Then the image will be divided into multiple
grids and each region will have a histogram that will represent pins (5V, ground and GPIO pin 6). The sensitivity of the
the occurrences of each pixel intensity. Then all histograms sensor was adjusted accordingly to suit the purpose of the
will be combined to form a single histogram. The algorithm system. Secondly, a USB web cam was used to perform the
will be applied for each face in the dataset, and with that the face recognition as well as the video call with the user. For the
algorithm will be trained. If any new input needs to be video call specifically, we had to add speakers and a
recognized, the algorithm will be will be applied to it and its microphone to provide the ability of talking and listening to
histogram will be compared with the ones stored in the dataset; the other person. The speakers got connected to the 3.5mm
the one with the nearest histogram will be returned. That means jack while the microphone was connected to the USB port. To
the one with closest distance, which could be calculated by the add the solenoid door lock, we had to add a power supply and
Euclidean distance, chi-square, absolute value, etc. A a relay. The door lock needs 12V to operate and the Raspberry
threshold value would be determined, in which any distance Pi provides 5V only. Thus, we connected an external power
higher than it is, would immediately be considered as an source (8 AA batteries) to provide the needed power. Relay
unrecognized face. switch is an important component that acts as an amplifier to
2. Android Studio convert the small electrical current from the raspberry pi to
It is a software development kit and environment used for larger current for the solenoid door lock. In addition, the
building applications, it is built on JetBrains' IntelliJ mlx90614 (contactless object temperature sensor) was
IDEA software and designed specifically for Android connected to the GPIO pins directly (3.3V, SDA, SCL and
development. A mobile Application will be developed using ground) [21].
android studio to create an interface through which the user can When the results of the mlx90614 sensor and an actual IR
control appliances. [17] thermo gun were compared, it was found that the reading from
the sensor tends to be lower than the actual temperature of an
3. Firebase
object. This was due to the manufacturing of the sensor itself
A main database must be created in between, through which
and we could not control it. To fix this, the results had to be
allows the communication between the Mobile application and
raspberry pi, as shown in Fig 2. In this paper, Firebase will be calibrated by adding 6.5° to the measured value. One of the
used. It is a real-time cloud-hosted database that would be main factors that affect the measurements is the distance
excellent for Home automation since it uses data between the hand and the sensor, whenever the object is closer
synchronization and updates data within milliseconds [18]. to the sensor higher temperatures are found. The distance must
This database will store all the data concerning the state of the range between 1-5cm. In addition to that, the room’s
door lock and the body temperature in real time. Both the temperature was found to have an impact on the results as
application and Raspberry Pi will be programmed to connect well. The lower the temperature of the surrounding, the higher
to one specific database, where they can exchange data. the measured temperature of the body.
 Fig 3: Circuit design

B. Facial detection and recognition

To evaluate the efficiency of the algorithms, a few samples
were collected assuming that Rima and Aysha are both
authorized faces.

No. of Face Detected? Recognised?

attempt
1 Rima YES YES
2 Aysha YES YES Fig 4: SMART DOOR LOCK mobile application
3 Unauthorised face YES NO
4 Aysha YES YES
5 Unauthorised face YES NO
6 Rima YES NO
7 Unauthorised face YES YES
8 Aysha YES YES
9 Rima YES YES
10 Unauthorised face YES NO
Table 1: Detection and recognition efficiency

Out of the 10 attempts, it was 100% successful in detecting Fig 5: Notifications received from the application
faces, whereas had an accuracy of 80% in recognizing the
faces. It was also noticeable that the lighting conditions had a
great effect on the system. A slight change in the lighting can
change the results. Hence, it is better to provide a light source
with a stable brightness to provide more accuracy.

C. Mobile application development

A mobile application known as SMART DOOR LOCK was
developed by using Android Studio as shown in Fig 4. This
application allows the user to have wireless control over the
door lock using Wi-Fi. It also displays the body temperature
retrieved from IR contactless temperature sensor. If it the
temperature is above the acceptable range it will alert the user.
Whenever a new temperature value is measured and updated
to the database, it will send a notification user regardless of Fig 6: Real-time database
whether the temperature is acceptable or not, as shown in Fig
5. Additionally, it will inform the user at what date and time D. Real-time database
the measured temperature was retrieved to provide more Fig 6 displays the database created. “Locking Door” contains
accuracy. the state of the door lock, where “ON” indicates it is locked
and “OFF” indicates it is unlocked. Whereas “mlx90614”
which is named after the IR temperature sensor contains the
temperature of the body of the object. The values will be
updated according to any changes made.
 V. CONCLUSION [6] Sudhasmita Behera et.al. 2017, “Home Automation control system
using SMS”, International Research Journal of Engineering and
Overall, movement will be detected by the PIR motion sensor Technology (IRJET), Volume 4(3). Available at:
that is connected to the raspberry Pi, then the recognition and https://www.irjet.net/archives/V4/i3/IRJET-V4I3409.pdf [Accessed:
21 Nov 2020]
detection algorithms will run. If user is recognized and the
[7] S Jermilla et. al. 2017, “Home Automation using LabVIEW”,
first operational mode is running, it will automatically unlock. International Journal of Science Technology & Engineering, volume
However, if the 2nd mode is running, the system will check if 3(9). Available at: http://www.ijste.org/articles/IJSTEV3I9073.pdf
their body temperature is within the acceptable range and act [Accessed: 27 Nov 2020]
accordingly. If the visitor was not recognized, a video over IP [8] Idris. 2020, Send data to firebase using Raspberry PI. Available at:
will be initiated with the user, then they will have the option https://tutorial.cytron.io/2020/12/09/send-data-to-firebase-using-
raspberry-pi/ [Accessed: 10 march 2021]
to confront the visitor if it turns out to be an intruder, whereas
[9] Orczyk T., Porwik P. 2011, Feature Based Face Detection Algorithm
if it is not an intruder then they can open the door using the for Digital Still Images. Available at:
Android mobile application and check for their body https://link.springer.com/chapter/10.1007/978-3-642- 23154-4_7
temperature. At the end of this research paper, an efficient [Accessed: 1 Feb 2021]
low-cost intrusion detection and door lock automation system [10] V. Starovoitov et.al. 1999, A GEOMETRIC APPROACH TO FACE
RECOGNITION. Available at:
was developed successfully. Its face detection rate works by https://www.researchgate.net/profile/Valery_Starovoitov/publication/2
100% and recognition rate works by 80%. 21026665_
By working on this project, we have also concluded that A_geometric_approach_to_face_recognition/links/00b49517fd785bdd
the technology of the Internet of Things is the most efficient e5000000/A -geometric-approach-to-face-recognition.pdf [Accessed:
1 Feb 2021]
and stable one to control and monitor smart home devices and
[11] Senthamizh Selvi.R et.al. 2019, Face Recognition Using Haar -
applications. That is due to the availability of Wi-Fi Cascade Classifier for Criminal Identification. Available at:
everywhere and the ability to manage the smart home https://www.researchgate.net/publication/345896024_22_Face_Recog
application from anywhere in the world unlike some other nition_using _Haar_-_Cascade_Classifier_for_Criminal_Identification
available technologies such as Bluetooth and Zigbee, which [Accessed: 1 Feb 2021]
can be limited by distance. [12] Marijeta Slavković et.al. 2012, Face Recognition Using Eigenface
Approach. Available at: http://www.doiserbia.nb.rs/img/doi/1451-
4869/2012/1451- 48691201121S.pdf [Accessed: 30 Jan 2020]
VI. FUTURE WORKS
[13] Farah Deeba et.al. 2019, LBPH-based Enhanced Real-Time Face
This system could be improved in various ways. First, the Recognition. Available at:
recognition accuracy could be increased by using deep https://pdfs.semanticscholar.org/3255/0898eec9c4424932e70e5e32c98
b0220a747. pdf [Accessed: 31 Jan 2020]
learning instead of machine learning. The aim is to reach
[14] P Viola and M Jones.2001. Rapid Object Detection using a Boosted
recognition accuracy between 95%-100%. Additionally, the Cascade of Simple Features. Available at:
reliability of the system could be increased be using a https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-
Raspberry Pi 4G/LTE HAT which could enable Video over 01.pdf [Accessed: 29 Jan 2021]
LTE, also known as VoLTE. Moreover, to improve the [15] Hussain Mujtaba.2020. Face Detection using Viola Jones Algorithm.
accuracy of the body temperature and the flexibility of the Available at: https://www.mygreatlearning.com/blog/viola-jones-
algorithm/ [Accessed: 30 Jan 2021]
system, a thermal camera could be used instead of the
[16] Kelvin S. 2017 Face Recognition: Understanding LBPH Algorithm.
mlx90614 sensor. Available at: https://towardsdatascience.com/face-recognition-how-lbph-
works-90ec258c3d6b [Accessed: 30 Jan 2021]
REFERENCES [17] Android studio. Available at:
[1] B. Shinde et. al. 2017, Smart Home Automation System using Android https://developer.android.com/studio/features [Accessed:1st Dec 2020]
Application. Available at: [18] Firebase Realtime Database. Available at:
https://www.irjet.net/archives/V4/i4/IRJETV4I4604.pdf [Accessed: 15 https://firebase.google.com/docs/database#:~:text=The%20Firebase%
Nov 2020] 20Realtime%20Database%20is%20a%20cloud%2Dhosted%20databa
[2] Krishna. R et. al. 2017, Gesture Human-Machine Interface (GHMI) in se.&text=When%20you%20build%20cross%2Dplatform,updates%20
Home Automation. Available at: with%20the%20newest%20data. [Accessed: 1st Dec 2020]
https://www.irjet.net/archives/V4/i6/IRJETV4I6636.pdf [Accessed: 15 [19] Maggie. T.2020. What is google duo? Available at:
Nov 2020] https://www.pocket-lint.com/apps/news/google/138528-google-duo-
[3] Saddam. 2017, DTMF based Home Automation System. Available at: video-call-app-how-does-it-work-and-does-it-offer-voice-calls
https://circuitdigest.com/electronic-circuits/dtmf-based-home- [Accessed:1st Dec 2020]
automation-system [Accessed: 15 Nov 2020] [20] Martin Mander .2019. 1986 PiNG Video Doorbell.Available at:
[4] B Madhurava et. al. 2017, Motion sensors and face recognition https://www.hackster.io/martin-mander/1986-ping-video-doorbell-
surveillance system using raspberry pi. Available at: 30b666 [Accessed: 1st Feb 2021]
http://www.ijarcs.info/index.php/Ijarcs/article/view/3484 [Accessed: [21] IoT Based Contactless Body Temperature Monitoring using Raspberry
15 Nov 2020] Pi with Camera and Email Alert .Available at:
[5] Nainsi Soni.et.al. 2017, “A Review of Home Automation System with https://circuitdigest.com/microcontrollerprojects/iot-based-contactless-
Speech Recognition and Machine Learning”, International Journal of body-temperature-monitoring-using-raspberry-pi-withcamera-and-
Advance Research in Computer Science and Management Studies, email-alert [Accessed: 14th march 2021]
Volume 5(4). Available at:
http://www.ijarcsms.com/docs/paper/volume5/issue4/V5I4-0022.pdf
[Accessed: 28 Nov 2020]

View publication stats