International Journal of Scientific Research in Engineering and Management (IJSREM)

Volume: 07 Issue: 05 | May - 2023 SJIF 2023: 8.176 ISSN: 2582-3930

Hand Gesture Controller (Virtual Mouse) and Voice Assistant using OpenCV,
ML, Python

Dr. Pratibha V. Waje1

Ms.Shipranjali K. Gangurde, Ms. Snehal S. Sonawane, Ms. Pallavi S. Avhad, Mr.Shubham S. Raut
Department of Information Technology
Sir Visvesvaraya Institute of Technology
A/p: Chincholi Tal.:Sinnar, Dist.: Nashik, Maharashtra,India- 422102

ABSTRACT- The advancements in artificial usually done through output devices like a mouse.
intelligence technology have led to the increasing However, carrying a hardware mouse everywhere can be
popularity of hand gesture recognition for controlling inconvenient and may get damaged. This led to the
virtual devices. This paper proposes hand gesture development of wireless mouse and speech recognition
controller (virtual mouse) and voice assistant that utilizes techniques for mouse functions. Eye tracking techniques
OpenCV for recognizing hand gestures and translating have also been used, but they have certain drawbacks.
them into mouse movements. The system serves as an Developers have attempted to develop human gesture
alternative interface for individuals who encounter recognition models using expensive gloves and sensors.
difficulties when using a traditional mouse or keyboard. A However, artificial intelligence has emerged as a major
camera captures images of the user's hand, which is technology that makes human life faster and more
processed by an AI algorithm to identify the gestures comfortable. This paper proposes a hand gesture-
made. The system is scalable and adaptable to different controlled virtual mouse using artificial intelligence
environments and devices. Dynamic/static hand gestures technology. This technology allows users to control the
and voice commands can control all input operations movement of their computer mouse through hand gestures,
without any additional hardware requirements. We use without a physical mouse. The system uses a camera
ML and Computer Vision algorithms to recognize hand vision-based approach to track the movements of the
gestures and voice commands, implementing the model user’s hand and perform mouse functions on the computer
using CNN and the (Gesture-Controlled-Virtual-Mouse) screen. The computer vision algorithms analyze the video
[2]controlled virtual mouse system has potential feed from the camera and identify the user’s hand and
applications in hazardous environments for enabling hand- track its movement. Machine learning models have been
free device operation and offering an alternative interface trained to recognize specific hand gestures, such as
to hardware mouse users. Our proposed system has the pointing or swiping, and translate them into corresponding
potential to enhance user experience and improve mouse movements. This technology has several
accessibility through human-computer interaction [1]. advantages, including its potential to improve accessibility
for people with disabilities and provide a more natural and
Keywords: Computer vision, hand gesture recognition, intuitive user experience. It can also be useful in situations
Mediapipe, and virtual mouse. where a physical mouse or touchpad is not available or
practical. The use of hand gestures as a control mechanism
eliminates the need for a physical mouse and provides a
1. INTRODUCTION more intuitive and natural way of interaction with
In our daily lives, technology has become an integral part. computers. The technology has numerous applications in
Computer technology is growing rapidly and being used areas such as gaming, virtual reality, and accessibility [3].
for various tasks that are impossible for humans to
perform. Interactions between humans and computers are

© 2023, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM21885 | Page 1

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 07 Issue: 05 | May - 2023 SJIF 2023: 8.176 ISSN: 2582-3930

2. LITERATURE REVIEW volume control, and brightness control. The model is

implemented using CNN and mediapipe framework, and
Previous work on AI virtual mouse systems involved using it has potential applications like enabling hand-free
gloves or colored pieces of paper attached to the hands for operation of devices in hazardous environments and
gesture recognition. However, these methods have providing an alternative interface for hardware mouse[4].
limitations in terms of accuracy, range of motion, and user
comfort. A hardware-based system that relies on a glove 3. PROBLEM STATEMENT
severely limits the hand's movement and may cause skin
irritation, while colored tips for gesture recognition may To track fingertips as a movable object, and to utilize it for
not always produce optimal results. Recent contributions mouse functions, the camera should be positioned in a way
have focused on camera-based detection of hand gestures so that it can see the user’s hands in the right positions.
using Google's mediapipe framework[5]. This can be used in space-saving situations, for those
patients who don’t have control over their limbs and for
One study developed a hardware-based system that other similar cases. It’s a virtual mouse instead of a
produced accurate results but had limitations in terms of physical mouse which will work only based on webcam
range of motion and user comfort. Another study proposed captured frames & tracking colored fingertips.
a machine-user interface that uses computer vision and
multimedia techniques for gesture detection, but it requires
4. AIMS AND OBJECTIVE
skin pixel identification and hand segmentation from The main objective of the proposed AI virtual mouse
stored frames before working with gesture comparison system is to develop an alternative to the regular and
techniques. Another study used a mobile phone camera traditional mouse system to perform and control the mouse
and projector for visual feedback, while other mobile functions, and this can be achieved with the help of a web
applications can link to their framework for gesture camera that captures the hand gestures and hand tip and
recognition. Another approach used only a webcam for then processes these frames to perform the particular
gesture recognition and detection and performed mouse mouse function such as left click, right click, and scrolling
functions using hand gestures[6]. function.
One study focused on robots with gesture controls and 5. PROPOSED SYSTEM
used optical flow and motion features to differentiate hand
gestures, while another study used the convex hull For hand and finger detection, the project employs the
technique to determine fingertip points for mouse control. open-source library MediaPipe. Developed by Google and
Another study utilized the YOLOv5 algorithm and OpenCV, MediaPipe is a cross-platform framework that
Artificial Intelligence (AI) to recognize hand gestures and uses machine learning concepts to detect hand gestures
improve HCI. A system that creates colored masks for and track their movements in real-time. It provides pre-
gesture recognition has also been proposed, but its made tools and components such as object detection, pose
implementation is difficult. estimation, facial recognition, and more, allowing
developers to create intricate pipelines that combine
In this study, we propose Hand Gesture Controller (Virtual multiple algorithms and execute in real-time on various
Mouse) and Voice Assistant that uses AI algorithms to hardware platforms [1].
recognize hand gestures and translate them into mouse
movements. The system is designed to provide an 1. Real-time Video and Audio Processing: MediaPipe
alternative interface for people who have difficulty using offers functionalities like video decoding, filtering,
a traditional mouse or keyboard. It uses a camera to segmentation, and synchronization for processing and
capture images of the user's hand, which are processed by analyzing video and audio streams in real-time.
an AI algorithm to recognize the gestures being made. The
system is trained using a dataset of hand gestures to 2. Facial Landmark Detection: MediaPipe can detect and
recognize different gestures, and it can perform various track facial landmarks in real-time, including eyes, nose,
mouse operations like left click, right click, drag and drop, mouth, and eyebrows. This functionality is useful for

© 2023, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM21885 | Page 2

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 07 Issue: 05 | May - 2023 SJIF 2023: 8.176 ISSN: 2582-3930

applications like facial recognition, emotion detection, and

augmented reality.

3. Hand Tracking: MediaPipe can track hand movements

in real-time, making it suitable for hand gesture
recognition and interaction with virtual objects.

4. Object Detection: MediaPipe can detect and track

objects using machine learning models in real-time. This
functionality is useful for applications like robotics,
surveillance, and augmented reality.

5. Pose Estimation: MediaPipe can estimate the poses of

human bodies in real-time, making it useful for
applications like fitness tracking, sports analysis, and
augmented reality.

Therefore, MediaPipe is an essential library for computer

vision and ML tasks, offering functionalities such as video
and audio processing, facial recognition, hand tracking,
object detection, and pose estimation.

6. IMPLEMENTATION

Mouse Functions Based on Hand Gestures and Hand Tip

Detection using Computer Vision:

1 2

© 2023, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM21885 | Page 3

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 07 Issue: 05 | May - 2023 SJIF 2023: 8.176 ISSN: 2582-3930

3 4

1. Mouse cursor movement around the computer

4 5 6
window.
2. To Perform Left Button Click operation
3. To Perform Right Button Click operation
1. To launch and end the gesture recognition
4. To perform a double click operation
2. To search for something over internet
3. To find a location what we are looking for
4. To get an idea about Date and time
5. To copy and paste contents
5 6 6. To exit voice assistant
7. RESULTS AND EVALUATION

A virtual mouse controlled by hand gestures has the

potential to serve as an alternative for people with
disabilities who struggle with traditional mouse and
7 8 keyboard usage. By simplifying their computer
interactions, this technology can enhance their ability to
engage with devices. Additionally, those who prefer to
5. To perform drag and drop operation work or play games without a physical mouse or touchpad
6. To perform multiple item selection can benefit from a hand gesture-controlled virtual mouse.
7. To perform volume controlling This model eliminates the need for a physical interface,
8. For No Action / neutral gesture to be Performed on enabling users to control their devices from a distance.
the Screen
Depending on the technology employed, a hand gesture-
Voice Assistant Features controlled virtual mouse may provide greater precision
and speed than traditional mice or video editing. The
success of this innovation is dependent on the user
experience it delivers. If the technology proves to be user-
friendly, dependable, and intuitive, it is likely to be well-
received. However, if the technology proves to be difficult
to use, unreliable, or unintuitive, users are likely to
abandon it quickly.

1 2 3 8. CONCLUSION
hand gesture-controlled virtual mouse is an innovative and
exciting technology that has the potential to revolutionize
the way we interact with computers. Here with the aid of
a real-time camera, we have created a system to manage
the mouse pointer and carry out its function. It offers users
a more natural, intuitive, and accessible way to control the
cursor on the screen, without the need for a traditional
input device, a mouse.

© 2023, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM21885 | Page 4

 International Journal of Scientific Research in Engineering and Management (IJSREM)
Volume: 07 Issue: 05 | May - 2023 SJIF 2023: 8.176 ISSN: 2582-3930

Furthermore, with additional voice assistant support, AI

virtual mouse using hand gestures can further enhance the [2] D. L. Quam, “Gesture recognition with a DataGlove,” IEEE
user experience. Voice assistant which is integrated with Conference on Aerospace and Electronics, vol. 2, pp. 755–760,
the virtual mouse system will provide users with even 1990.
View at: Publisher Site | Google Scholar
more control over their devices. Users can given voice
commands to do a range of tasks, such as opening [3] D.-H. Liou, D. Lee, and C.-C. Hsieh, “A real time hand
applications, navigating through menus, and performing gesture recognition system using motion history image,” in
web searches, in addition to controlling the cursor on the Proceedings of the 2010 2nd International Conference on Signal
screen using hand gestures. As technology continues to Processing Systems, IEEE, Dalian, China, July 2010.
evolve, we can expect to see even more innovative View at: Publisher Site | Google Scholar
solutions that enhance the user experience and improve
accessibility for all. [4] S. U. Dudhane, “Cursor control system using hand gesture
recognition,” IJARCCE, vol. 2, no. 5, 2013.
View at: Google Scholar
Chart Title [5] K. P. Vinay, “Cursor control using hand gestures,”
100 International Journal of Critical Accounting, vol. 0975–8887,
2016.
98
View at: Google Scholar
96
[6] L. Thomas, “Virtual mouse using hand gesture,”
94
International Research Journal of Engineering and Technology
92 (IRJET, vol. 5, no. 4, 2018.
Accuracy View at: Google Scholar

Neutral Left click [7] P. Nandhini, J. Jaya, and J. George, “Computer vision system
Right click Double Click for food quality evaluation—a review,” in Proceedings of the
2013 International Conference on Current Trends in
Drag & Drop Multiple item Selection Engineering and Technology (ICCTET), pp. 85–87,
Scrolling Volume Coimbatore, India, July 2013.
View at: Publisher Site | Google Scholar

Overall, the hand gesture-controlled virtual mouse has the [8] J. Jaya and K. Thanushkodi, “Implementation of certain
potential to improve the accessibility of and convenience system for medical image diagnosis,” European Journal of
of computer interaction for users with physical disabilities Scientific Research, vol. 53, no. 4, pp. 561–567, 2011.
View at: Google Scholar
or for users who prefer an alternative to traditional input
devices.
[9] P. Nandhini and J. Jaya, “Image segmentation for food
Analysis: Dr. Pratibha V. Waje provides Guidelines for quality evaluation using computer vision system,” International
archiving 97% accuracy in Hand Gesture Controller Journal of Engineering Research and Applications, vol. 4, no. 2,
(Virtual Mouse) and Voice Assistant using OpenCV, ML, pp. 1–3, 2014.
Python [10]. View at: Google Scholar

[10] Ms. Pratibha V. Waje, Dr. R. Jain, A Recommendation

9. REFERENCES System for Execution Plans, Journal of Shanghai Jiaotong
University, Volume 16, Issue 11, November - 2020,107-113
[1] J. Katona, “A review of human–computer interaction and
virtual reality research fields in cognitive
InfoCommunications,” Applied Sciences, vol. 11, no. 6, p. 2646,
2021.
View at: Publisher Site | Google Scholar

© 2023, IJSREM | www.ijsrem.com DOI: 10.55041/IJSREM21885 | Page 5