Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Hand Gesture Control System using Open CV

Nikita Chaudhari, Kaustubh Sonawane, Arathi Boyanapalli
Department of Computer Engineering,
SIES Graduate School of Technology, University of Mumbai, Nerul Navi Mumbai, India

Abstract:- All around the world, researchers are recognition as a means for computers to start deciphering
working to improve the interactivity of digital devices body language. It creates a richer connection between
and to make them operate with the least amount of computers and people than the simple user interfaces, like
physical interaction possible. The interactive computer the keyboard and mouse. Gesture control systems interpret
system proposed in this study can function without a and recognize human body gestures to enable user
keyboard or mouse. By the implementation of this interaction with a computer system.
project, everyone would have an advantage including a
paralyzed person who finds it difficult to use a mouse or Gesture recognition is a topic of active research in
keyboard. The easiest and most natural way to Human Computer Interaction (HCI) technology. In the
communicate is through hand gestures and hence an realm of Human Computer Interaction, gesture recognition
innovative method of computer control using a real-time is a well-liked and in-demand analytical technique. It has
web camera is demonstrated in this Hand Gesture applications in a variety of jobs, including controlling
Control project. The approach is to use a camera and robots, managing virtual environments, translating sign
computer vision technologies to handle tasks and language for medical applications, automating homes, etc.
perform the jobs without the use of a physical mouse or The consumer electronics industry, the automobile industry,
keyboard. A webcam with normal resolution is used to the healthcare industry, and the educational sector are some
track the user's hand in two dimensions. The system is of the industries that have embraced gesture control
created using Python and OpenCV. The output of the technology. Recent HCI research has taken on a special
camera will be seen on the monitor and the tasks significance. Due to its competence, the hand is the most
associated with the hand gestures will be performed. useful communication tool in all the body parts. The term
Thereby, an application of an easy-to-use gesture control "gesture" is used in many situations involving human
system is shown in this project. motion. Only certain human motions, particularly those of
the hands, arms, and face which are referred to as gestures
Keywords:- Interactive Computer System, Hand Gesture, are instructive. The ability for machines to recognize and
Webcam, Keyboard, Mouse. categorize hand gestures is provided by hand gesture
identification, which is regarded as a vital component of
I. INTRODUCTION Human Computer Interaction (HCI). With the development
of current technology and the widespread use of hand
Nowadays, gesture control applications are becoming movements or hand gestures in daily communication, the
an area of wide interest with evolving technology. hand gesture identification is treated as a crucial part of
Keyboards and mouse have been the primary input methods HCI. Recently, the study of hand gestures has seen an
for computers for many years. However, as ubiquitous, and increase in interest and popularity. Moreover, it permits a
ambient technology (like PlayStations) and tools that let straightforward and natural method of interaction.
users hold virtual items become more and more common,
hand or body gestures are becoming increasingly important. The goal of this project is to create an interface that
Gesture controllers are now an integral part of the human dynamically captures human hand gestures and controls
computer interaction system. Computers can grasp human various functions of the computer device
body language with the help of gesture recognition. Instead using hand movements. This will improve the entire
of using only the simple text-user interfaces or graphical- experience of using a computer without the mouse or
user interfaces, the gesture recognition system aids in keyboard and using hand gestures to control the
creating a stronger connection between humans and fundamental operations. As humans, we always look for
technology (GUIs). The gesture recognition is an ways to make our lives easier by the use of technology.
advancement of Touch User Interface (TUI), a type of Moreover, with the help of gesture control, even those with
interaction between a user and a computer-based device by disabilities will be able to operate computers with ease.
making a physical touch on the screen. As the name implies,
gesture recognition aims to identify the physical motions, or II. LITERATURE SURVEY
"gestures," made by people. For example, one could instruct
the gadget to start a specific app and carry out other In terms of popularity, gesture control technologies
operations by waving hand in a precise pattern in front of it. have eclipsed more traditional mechanical communication
In this gesture recognition and control project, the computer techniques. The domain market is divided into many
camera interprets the movements of human hand. This categories depending on a variety of criteria, such as
information is subsequently used as input by the computer location, assistive robots, sign language recognition
camera to handle various computer activities as well as technology, immersive gaming, smart Televisions, virtual
applications. Gesture recognition is the foundation of controllers, virtual mice, etc. Myron W. Krueger initially
gesture control technology. You may think of gesture proposed the method of hand gesture-based computer

IJISRT23NOV1262 www.ijisrt.com 1307

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
communication in 1970 [1]. This study [2] investigates the matching is created in the final PCA stage in order to
use of optical mouse sensors to create a motion based HCI identify the gesture. Techniques for recognizing hand
device called a laser spot tracking system. The primary gestures using vision are examined in surveys spanning the
objective of the work is to characterize laser speckle previous 16 years. Further, this paper [12] analyses 26
detection using optical mouse sensors. As a precise, rapid, publicly accessible hand gesture datasets in the vision-based
and compact sensing method based on the laser speckle hand recognition. In this paper [13], a vision-based hand
displacement measurement capacity of optical mouse gesture recognition application for HCI was built. A real-
sensors, a low-cost optical mouse sensor is utilized to record time system to stop mouse movements in windows without
the motion of the laser spot. The fundamental goal of utilizing ANN training and by using a color-based detection
gesture recognition is to recognize a specific human gesture technique is developed. According to [14], a system that
and give information about it to the computer. The ultimate primarily records human hand gestures onto a disc, converts
objective is to create computer systems that can identify them into binary images by extracting frames from each
human motions and remotely operate objects. Nowadays, video one at a time and then forms 3D Euclidian space for
most human-computer interaction (HCI) involves the use of binary images to classify hand motions has been proposed.
keyboards or mouse, but the capacity to reliably recognize Back propagation technique and supervised feedforward
human gestures has increased the importance of computer neural network are utilized in the training phase because
vision-based techniques [3]. Paper [4] suggests hand they are suitable for simple gestures with limited
gestures recognition for TV management. An actual HCI background complexity. The approach for identifying the
system based on gestures that uses a single monocular uncover hand with no color cap is suggested in this paper
camera and reaches out to the HRI case has been built in this [15]. The Hue image is created by altering the RGB image.
study [5]. A Convolution Neural Network classifier is used For localization and segmentation, the arm region removal
by the created system to comprehend its features and approach and the hand skin segmentation technique are
identify the gestures. A key tool for the real-time, dynamic used. In this paper [16], a method is developed to identify
gesture identification process is the Hidden Markov Model hand motions in real-time photographs that are gathered.
[6]. The HMM approach is practical and designed to The hands that are present for categorization have a number
function in static contexts. In order to create the best of intersecting locations. This technique is also being used
possible state sequence for pattern recognition, the approach to address a few issues with gaming consoles like the Xbox,
uses the HMM's LRB topology along with the Baum Welch PS4, and smartphones. To address the issue, a clever
Algorithm for training, the Forward and Viterbi Algorithms solution is designed. The hand motions are recognized in the
for testing, and the Viterbi Algorithm for evaluating the real-time photos using Python 3.9 and MediaPipe.
input finding sequences. The system in this paper [7] looks
to be simple to use in comparison to more recent systems or III. METHODOLOGY
command-based systems, however it is less effective at
spotting and recognizing patterns. Upgrading the system and One of the methods that can recognize a hand gesture
make an effort to build a more robust algorithm for both in a real-time video in everyday life is the hand gesture
detection and recognition despite a confusing background recognition technology. The goal of this project is to design
and typical lighting conditions. Also, the system must be a hand gesture recognition and control system. One of the
upgraded to include several extra gesture categories because most crucial and essential concepts in computer vision is
it was only designed to support six. For those with hand gesture control. The HCI systems that combine hand
disabilities, a gesture control and hand recognition system processing tasks like hand detection and hand gesture
was developed in this paper [8]. In this study, wearable hand identification are now practical, all thanks to recent
gesture devices were employed with surface advancements in information and media technology.
electromyography (sEMG) sensors, and an artificial neural The project encompasses various libraries such as
network was primarily used as a classifier for hand gestures OpenCV, Numpy, Pycaw, MediaPipe, PyAutoGUI.
used in sign language. The dataset in this study [9] had an Computer vision and image processing activities can be
overfitting issue. The Python programming language, performed with OpenCV. The features it offers include
OpenCV libraries and Linux framework were used to tracking, hand identification, and object detection. NumPy, a
construct the real-time, vision-based system. A virtual library for the Python programming language, supports
gesture control mouse method is created to assist the mouse large, multi-dimensional arrays and matrices as well as a
pointer and enable real-time camera functionality for the large variety of high-level mathematical operations that may
mouse's functions [10]. In an effort to perform mouse be carried out on these arrays. Pycaw (Python Core Audio
indicator activities and icon selection, this technique is also Windows Library) is a volume-controlling Python Audio
constructed on the foundation of picture comparison and Control Library. Google's open-source MediaPipe machine
motion recognition mechanisms. This paper [11] focused on learning library offers encapsulation for python and other
recognition systems that use vision. The skin tone color languages and includes some solutions for gesture detection.
model, thresholding approach, template matching, and PCA A high-fidelity hand and finger tracking solution, MediaPipe
were the variables that served as the foundation for the Hands is also used in the project to recognize the motion and
database of hand motions. The first phase is hand shape of hands which plays a key role in enhancing user
segmentation, which is accomplished by using the skin color experience across a range of technological platforms. It
model. The second step is the separation of the hand picture employs Machine Learning to extrapolate 21 crucial 3D
from the surrounding environment. A format-based hand details from a single frame. It is used to obtain the

IJISRT23NOV1262 www.ijisrt.com 1308

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
coordinate points of the fingertips of hand. The Python and the associated gestures are recognized by the camera.
library PyAutoGUI enables users to simulate keyboard The gestures are thereafter interpreted by the MediaPipe
button pushes as well as mouse pointer clicks and library, and the linked function is
movements. It has several features such as (1) moving the performed on the computer device. Various gestures are
mouse and clicking in other applications' windows (2) included in the project which perform tasks such as
sending keystrokes to other applications (for example, to fill controlling the volume of device, taking screenshots, closing
out forms) (3) Capture screenshots, and when given an desktop applications, putting the device to sleep, system
image (such as a button or checkbox), locate it on the screen shut down, screen-lock, screen-unlock. Other actions related
(4) Find a window for an application, and move, resize, to controlling the Word Document like closing a file, saving
maximize, or close it (currently only available on Windows) a file, printing a file, etc. are also implemented in the
and (5) Display alert and message boxes. project. When a word document is open and the save gesture
is made, the document will be saved. Similarly, when the
In this machine learning hand gesture control system print gesture is made, the printer dialogue box will open.
project, Python and OpenCV are implemented to create a The gesture designated for close/exit will similarly close the
real-time hand gesture controller. The working principle of active application. If no applications are running, the system
the project includes the camera which is used to detect the will shut down automatically. It functions similarly to
hand gestures with the help of associated points. The Windows PC's Alt+F4 key press.
distance between the desired points on the hand is calculated

Fig. 1: Methodology Flowchart

The methodology of the project is explained stepwise Initially, the video input is obtained from the
in Fig 1. It is initiated when the hand movement is computer’s primary camera and then the image is converted
recognized through the camera. OpenCV captures the hand into RGB form, and the processing of the image is
gestures by capturing the frames. The video accessed completed. The primary camera's video input is utilized to
through the camera is flipped. The hand gesture language is generate the video input for the MediaPipe Hand module,
detected using MediaPipe library and then the fingertip which is used to detect hands. The initialization and
coordinates are detected. Further the distance between each configuration of the discovered hands are finished by
point on the hand is calculated. The final gesture made by ‘MpHands.Hands()’. Using ‘mp.solutions.drawing_utils’,
the hand is detected after calculating the distance between the connections and landmarks are depicted on the detected
the points. If any application such as Word Document is hand. It uses a neural network to predict the locations of 21
open, then the associated gestures are performed on the key points on the hand, including the tip of each finger, the
running application. However, if no application is open, then base of each finger, the center of the palm, and the wrist.
the gestures are performed on the operating system itself. The points 2 - 4 represent the thumb, points 5 - 8 represent
the index finger, points 9 - 12 represent the middle finger,
points 13 – 16 represent the ring finger and points 17 – 20
represent the little finger.

IJISRT23NOV1262 www.ijisrt.com 1309

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig. 2: Hand Landmarks [17]

The list of hand components, or the number of points different operations on the operating system or the current
on the hand, as detected by the MediaPipe Hand module, is running application.
produced as an empty list. It also examines whether there
are several hands on the input. Then the key points of the Details of Hardware & Software for the
fingertip are specified in a lmList i.e. lmList[8][1], implementation of the project are as follows:
lmList[8][2]. In this way the x and y coordinates of a key
point are created (such as: x, y = lmList[8][1], lmList[8][2] A. Hardware requirements: -
). After obtaining the coordinates for the key points a  Pentium i3 Processor System
miniature opaque circle is drawn at the coordinates using  4 GB Ram
cv2.circle() function. These circles are then connected by  500 GB Hard Disk Space
drawing solid lines using c2.line() function.  LED Monitor
 Input Devices like Keyboard, Camera, Mouse
Further, the distance between the coordinates is
calculated using the hypot() function imported from the B. Software requirements: -
math library. The distance is calculated to differentiate  Python 3.8 Programming Language
between different gestures and to identify the gestures  Windows 10 Operating system
distinctively. Then it is detected whether an application is in
the running state or not. If an application is running, for IV. RESULTS AND DISCUSSIONS
example, Word Document is open, then a set of gestures can
be used to perform different operations on the word file such After the text edit has been completed, the paper is
as print, save, close, etc. If no application is running then the ready for the template. Duplicate the template file by using
same set of gestures can be used to perform operations such the Save As command and use the naming convention
as volume up and down, system shutdown, screenshot, etc. prescribed by your conference for the name of your paper.
In this newly created file, highlight all of the contents and
As different gestures made by the hand have different import your prepared text file. You are now ready to style
distances between the coordinates, using these different your paper; use the scroll down window on the left of the
distances we can differentiate between the different MS Word Formatting toolbar.
gestures. For example, if a fist is made then the distance
between the finger tips become very close to zero so we The project has resulted in identifying the hand
define a range like 0-15 for the distance between the finger gestures appropriately and performing the associated
tips, but if a ‘yo gesture’ is made then the distance between activities such as volume control, screenshot, closing a file,
the fingertip of thumb, middle and ring finger becomes close save and print operations on a file, device sleep and shut
to zero and the distance between little and index finger down successfully. The implementation of the hand gesture
becomes very large, thereby giving it a higher range. Once control system along with the screenshots can be seen
the gesture is identified, the pycaw library for volume below. Actions associated with specific gestures were
controls and pyAutoGUI library is used to perform other captured by the device camera and the operations for various
tasks were performed.

Fig. 3: Volume Control Hand Gesture

IJISRT23NOV1262 www.ijisrt.com 1310

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig. 4: Screenshot Hand Gesture

Fig. 5: Hand Recognition

Gesture control technology has advanced greatly since version of the same technology might undoubtedly be
the 1980s, when the first research was conducted. Based on advantageous and will result in a better interaction of human
the project's findings, it can be concluded that hand gesture and computer devices by elevating the Touch User Interface
identification and control using Python and OpenCV can be (TUI).
developed utilizing the theories of hand segmentation and
the hand detection system, which apply the Haar-cascade ACKNOWLEDGMENT
classifier. To sum it up, this system has achieved project
goals such as (1) To establish a comprehensive system for We would like to extend our sincere gratitude to all the
hand gesture detection, interpretation, and recognition using people who played a crucial role in the completion of our
Python and OpenCV and (2) To create the numbers and sign project. We would like to express our deepest gratitude to
languages of hand gestures displayed in the system to fulfil our coordinator, Prof. Arathi Boyanapalli, for her
the project's title. continuous support throughout the duration of the project.
We would also like to express our sincere gratitude to our
The implementation of additional gestures can be colleagues, whose efforts were instrumental in the
included in the system's future recommendations so that completion of the project. Through the exchange of
users with varying skin tones and hand sizes will be able to interesting ideas and thoughts, we were able to present a
effortlessly complete more tasks. The right hand with a project with correct information. We also want to express
specified area in ROI is the sole hand used in the current our gratitude to our parents for their personal support and
system to make gestures. As a result, the technique may be encouragement to us to pursue our own paths. In addition,
improved to allow for the employment of both user hands to we would like to express our sincere gratitude to our
execute various signs in conjunction with computer Director, Dr. Atul Kemkar, and our Head of Department,
processes. A recognition system for both the hands working Dr. Aparna Bannore. We also thank our other faculty
simultaneously can be implemented to level the project on a members for their invaluable contribution in providing us
higher scale. This can further be used in gaming with the required resources and references for the project.
technologies, thus enhancing the user experience.
Algorithms for background subtraction can also be DECLARATION
employed for better detection and tracking of the hand We confirm that the contents of this written submission
thereby resulting in better performance. In the future, when reflect our own ideas and opinions. Where we have
users will be able to interpret a gesture's meaning into a sign incorporated ideas or words from other sources, we have
or number and vice versa, the Graphical User Interface provided appropriate citations and sources to confirm the
(GUI) can be implemented. Nevertheless, technology is original source. We also confirm our commitment to
constantly evolving, so implementing a more advanced academic integrity and honesty. We have not

IJISRT23NOV1262 www.ijisrt.com 1311

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
misrepresented, fabricated, or falsified any idea, data, fact, [13]. Chetan Dhule, Trupti Nagrare, "Computer Vision
or source in our submission. As a result, we maintain the Based HumanComputer Interaction Using Color
authenticity and honesty of our work. We pledge to maintain Detection Techniques." Fourth International
the highest level of academic ethics in all our academic Conference on Communication Systems and Network
endeavors. Technologies, pp.934-938, 2014.
[14]. Murthy and Jadon, "Hand Gesture Recognition using
REFERENCES Neural Networks," 2nd IEEE International Advance
Computing Conference (IACC), pp. 134–138, (2010).
[1]. M. Kruger, Artificial reality II, Addison-Wesley [15]. Xu, Yishen, Jihua Gu, Zhi Tao, and Di Wu. "Bare hand
Reading (Ma), 1991. gesture recognition with a single color camera." In
[2]. Guoli Wang, (2010). Optical Mouse Sensor-Based Image and Signal Processing, 2009. CISP'09. 2nd
Laser Spot Tracking for HCI Input, proceedings of the International Congress on, pp. 1-4. IEEE, 2009.
2015 Chinese Intelligent Systems Conference: Volume [16]. Lavanya Vaishnavi D. A., Anil Kumar C., Harish S.,
2, pp.329-340. Divya M. L. “MediaPipe to Recognise the Hand
[3]. J. C. Manresa, R. Varona, R. Mas and F. Perales, Gestures”, WSEAS TRANSACTIONS on SIGNAL
"Hand tracking and gesture recognition for human PROCESSING DOI:
computer interaction", Electronic letters on computer 10.37394/232014.2022.18.19Lavanya, July 2022.
vision and image analysis, vol. 5, pp. 96-104, 2005. [17]. Hands | mediapipe (google.github.io) |
[4]. W. Freeman and C. Weissman, "Television control by https://google.github.io/mediapipe/solutions/hands.htm
hand gestures", Proc. of Intl. Workshop on Automatic l
Face and Gesture Recognition 1995, pp. 179-183,
1995.
[5]. Pei Xu, "A Real-time Hand Gesture Recognition and
Human-Computer Interaction System",
arXiv:1704.07296v1 [cs.CV] 24 Apr 2017.
[6]. J.R Pansare, Malvika Bansal, Shivin Saxena, Devendra
Desale, "Gestuelle: A System to Recognize Dynamic
Hand Gestures using Hidden Markov Model to Control
Windows Applications", International Journal of
Computer Applications (0975 – 8887) Volume 62–
No.17, January 2013.
[7]. Ram Pratap Sharma and Gyanendra K. Verma,
"Human Computer Interaction using Hand Gesture",
Eleventh International Multi-Conference on
Information Processing-2015 (IMCIP-2015).
[8]. Varun, Kollipara Sai, I. Puneeth, and T. Prem Jacob.
"Hand gesture recognition and implementation for
disables using CNN’S." 2019 International Conference
on Communication and Signal Processing (ICCSP).
IEEE, 2019.
[9]. Yasen, Mais, and Shaidah Jusoh. "A systematic review
on hand gesture recognition techniques, challenges and
applications." PeerJ Computer Science 5 (2019):
e218Anna De Liddo, Ágnes Sándor, et.al, (2012).
Contested Collective Intelligence: Rationale,
Technologies, and a Human-Machine Annotation.
Computer Supported Cooperative Work (CSCW)
Volume 21, Issue 4–5, pp 417–448.
[10]. Shibly, Kabid Hassan, et al. "Design and development
of hand gesture based virtual mouse." 2019 1st
International Conference on Advances in Science,
Engineering and Robotics Technology (ICASERT).
IEEE, 2019.
[11]. Ahuja, Mandeep Kaur, and Amardeep Singh. "Static
vision based Hand Gesture recognition using principal
component analysis." In MOOCs, Innovation and
Technology in Education (MITE), 2015 IEEE 3rd
International Conference on, pp. 402-406. IEEE, 2015.
[12]. Pisharady, Pramod Kumar, and Martin Saerbeck.
"Recent methods and databases in vision-based hand
gesture recognition: A review." Computer Vision and
Image Understanding 141 (2015): 152-165.

IJISRT23NOV1262 www.ijisrt.com 1312