Volume 7, Issue 4, April – 2022 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Sign Language Recognition using

Machine Learning: A Survey
Anagha.G1, Sinchana.S.Bharadwaj1, Sumana.M.R1, Varshini.Krishna.Mohan1, Nagaraj.A2* ,
1
Under Graduate Student,2 Associate Professor,
Dept. Of Computer Science & Engineering Jyothy Institue of Technology,
Visvesvaraya Technological University Thataguni Post,
Bengaluru-560082, India

Abstract:- It is said that more than one billion people in have sensors on them, and thus the movements are captured.
the world are disabled. One of the only ways they can Even though this system provides high accuracy it is not cost-
communicate among themselves or with people who don’t effective. Vision-based systems are the ones in which the
have this disability is sign language. Sign language is a signs are detected using visions, for this purpose either static
creative way to communicate within the deaf community images are used or real-time images are captured. Sign
by using gestures done by hand and other means that do language is all about perspective hence the vision-based is an
not require talking. It involves combining hand effective method.
movements, their shapes, etc. Body movements and facial
expressions are also taken into consideration. All these The proposed algorithm in this solution is CNN,
factors help convey the person’s thoughts in a fluent convolutional neural network that is a part of deep learning. It
manner. Most of the general public who aren’t disabled is mostly used for analyzing visual representations. Tensor
have no knowledge about sign language. Even out of the flow is also used which is a tool used to develop and train the
few who are aware, the majority don’t know how to use it model.
for communication, this stops them from interacting with
the deaf and mute people. Through this software, we want II. LITERATURE SURVERY(ABSTRACT)
to raise awareness towards sign language and help bridge [1]In this publication, an effort has been made to
the gap by creating a sign language interpreter that highlight the work of American Sign Language researchers
recognizes hand gestures. and compare their work. Asl - American sign language is
Keywords:- Sign Language Recognition, Mobile Application, used , as it is the most widely used sign language. This system
Convolutional Neural Network, Machine Learning. uses Microsoft Kinect for image acquisition or extracting the
dataset. Feature extraction is done using PCANet (Principal
I. INTRODUCTION component analysis network) and the classification of of ASL
letter is done using COnvolution neural network(CNN). The
Sign language is a way of communication for the deaf majority of SL and gesture identification problems have been
and sign language recognition is the method through which solved using statistical modeling techniques such as PCA and
the hand gestures in sign language are recognized by using support vector machines. The system proves to be 98%
images or videos. effective due to the fact that the application requires more
features or distinction to be accurate. Less number of signs
Each country has its own sign language. There are more were used to train the model, leading to poor user experience.
than 300 sign languages in the world, each differing from
another based on what part of the world they belong to. Sign [2]This system is a real-time sign language translator
language may even be different in the same country based on that translates sign language to text. It uses GoogLeNet
the accent. region,etc. One of the most prominent sign architecture to train the data .It uses Convolution neural
language systems known is ASL which is the American network to classify each frame in the video to a letter, and it
standard language. ASL is a natural language with its own reconstructs to display the most likely word from the
structure and unlike spoken English. It is a visual language classification.This is a a robust model that correctly classifies
that uses the movement and placement of hands to convey the letters a-e in the majority of cases when used by first-time
meaning of words. Facial expressions and body movements users, and another that correctly classifies letters a-k in the
also play an integral part in the language. majority of situations Given the dataset restrictions and the
promising findings obtained, we are hopeful that with further
In India ISL, Indian Sign Language is used. This is research and data, we will be able to build a fully
different from ASL as it uses its own vocabulary and generalizable translator for all ASL letters. Lack of variety in
grammar. Even though ISL is used in India, a part of the deaf the datasets, the validation accuracies were not directly
community still use ASL so for this proposed system we have replicable when tested on the web application. Better letter
decided to use ASL. segmentation is required, as well as a more smooth method for
retrieving photographs from users at a faster rate.
There are two different types of approaches to recognize
sign language, they are contact-based systems and vision- [3]Sign Language Recognition is one of today's fastest-
based systems. Contact-based systems as the name suggests growing disciplines of study. In these disciplines, many
are based on contact. Gloves are used in these systems, which innovative techniques have lately been developed. For the

IJISRT22APR1566 www.ijisrt.com 1211

Volume 7, Issue 4, April – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
recognition of diverse Indian Sign Languages, the suggested Perceptron is used to recognize signs and normalized central
method employs Eigen value weighted Euclidean distance as moments are used to represent the frames properly.
a classification technique. Skin Filtering, Hand Cropping,
Feature Extraction, and Classification are the four aspects of [7]This paper uses an open-source framework of Media
the system. In this paper, 24 signs were investigated, each Pipe and Support Vector Machine(SVM) to recognize the sign
with 10 samples, for a total of 240 photos, with a recognition languages. They use four different types of datasets for this
rate of 97 percent. Key highlight of this system is that signs model, they are Indian, American, Italian, and Turkey .Media
can be done using both the hands. One drawback is that the Pipe is a framework that helps people in developing audios
alphabets ‘H’and ‘J’are not added as they are considered and video data cross-platform in machine learning pipelines.
dynamic gestures. This is done in three stages, the first stage is detecting and
extraction of hand features, stage two is cleaning the data and
[4]A system that uses hand detection as a learning tool the third stage is analyzing sign languages using machine
for beginners in sign language has been developed. This learning algorithm. The average accuracy of this system is
system is based on explicit skin-color space thresholding, 99%.
which is a skin-color modeling technique. The skin-color
range has been predetermined to separate pixels (hand) from [8]This paper is for recognizing the Indian Sign
non-pixels (background). For image categorization, the photos Language. The ISL is not as common as ASL because of the
were loaded into a model called the Convolutional Neural lack of a datasets, ISL also uses two hands to make
Network (CNN). Keras was utilized for image training.The gestures,which makes it hard to detect. Hence, considering all
system achieved an average testing accuracy of 93.67 percent this paper proposes to bridge the communication gap.
when given suitable illumination and a uniform background, Machine learning algorithms like SVM and random forest are
with 90.04 percent credited to ASL alphabet recognition, used for training. The proposed method in this paper is to
93.44 percent for number recognition, and 97.52 percent for capture the image,process it, extract the features and then
static word identification, outperforming prior related studies. classify them.
The method is used to perform quick computations.
This system recognizes the sign languages by using
[5]SLR (sign language recognition) is a difficult yet Convolutional neural networks (CNNs), Microsoft Kinect and
crucial study subject for numerous computer vision GPU acceleration. Italian gestures are recognized in this
systems.Attempt to improve communication between the deaf system.The dataset used is ChaLearn Looking at People 2014,
and the hearing.Hearing-impaired individuals. An accurate which consists of 20 gestures by different users in different
method is presented in this paper .and a comprehensive deep environments. So the video data is processed using 2D
learning-based sign language technique. Video sequences are convolutions, one CNN is used for extracting hand gestures
also recognised. The method is based on skeletal aspects of and the other one for upper body features. Data augmentation
the hand and body retrieved from RGB videos and, As a is used during training to reduce overfitting.The accuracy is
result, it obtains highly discriminative skeletal data for gesture up to 91.70%
detection without the use of any additional equipment.
Experimentation with a big publicly available dataset of sign This system uses smart gloves to recognize the gesture
language demonstrates the superiority of our methods over that the person wants to portray. It uses artificial language to
others.The proposed SLR system is the first attempt to achieve this. The normal glove solutions only detect motions
combine a vision-based technique (i.e., video sequence of hands or can only recognize single gestures. So this system
processing) with reliable skeletal data extraction without the proposes a system which can recognize sentences as well and
use of data gloves or other sensors that constrain a signer's with more accuracy. The method of segmentation and non
mobility.This work introduces a novel SLR system that segmentation is used for this feat which is assisted in deep
proposes the extraction and processing of hand and body learning model. A total of 20 sentences and 50 words are
skeletal data from video sequences, attempting to overcome recognized. The gestures that are recognized by the sensors in
the limitations of earlier approaches. the glove are projected into a virtual place and they are then
translated to text and audio. The average rate of correctness is
[6]This system proposes a system for the recognition of 86.67%.
Mexican Sign Language. They are using a computer vision
system. This system uses a digital camera and four LED
reflectors to capture the sign language. They are using
artificial neural networks as their pattern recognition model.
The MSL has 27 signs in which 21 and static and in this
system those 21 signs are being recognized.Multi Layer

IJISRT22APR1566 www.ijisrt.com 1212

Volume 7, Issue 4, April – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
III. LITERATURE SURVEY

The below list (TABLE I) outlines survey of papers related to the topic in brief with possible gaps/limitations within the
proposed system.

Papers Title Authors Year Of Proposed system Gaps

Publication

Static sign language Lean Karlo S. A sign language learning This system can
recognition using Tolentino, Ronnie system which is based on recognize only static
deep learning O. Serfa Juan, the modelling technique of sign languages.
[1] August C. Thio-ac, 2019 skin color. A specific range
Maria Abigail B. is agreed upon that will
Pamahoy, Joni extract pixels from non
Rose R. Forteza, pixels.
and Xavier Jet O.
Garcia

Sign language Dimitrios A methodology for sign Hand joints are

[2] recognition based on Konstantinidis, language recognition that considered for
hand and body Kosmas uses hand and skeletal recognition and other
skeletal data Dimitropoulos, and features that are extracted hand parts may cover
Petros Daras from RGB videos. the said joints.
Real-time vernacular A sign language This system only
sign language Arpita Halder and recognition system which recognizes alphabets
recognition using Akshit Tayade uses MediaPipe’s open and numbers.
[3] mediapipe and 2021 framework to detect hand
machine learning gestures.

Real-time American Brandon Garcia Real-time sign language The validation

Sign Language and Sigberto translator that uses accuracies are not
Recognition with Alarcon Viesca GoogleNet architecture to directly replicable
Convolutional Neural train the data to classify when tested on the
[4] Networks each frame in the video to a web application.
letter, and reconstructs Better letter
characters to display the segmentation is
most likely word from the required.
classification.
Review on sign Anamika The proposed system uses Application requires
language detection Srivastava1 and Microsoft Kinect for image more features or
using machine Vikrant Malik acquisition and PCA net distinction to be
[5] learning 2020 architecture for feature accurate. Less number
extraction. SVM technique of signs were used to
is used to classify the data train the model,
and get desired output. leading to poor user
experience.
Sign language Radha A sign language The system can be
recognition using S.Shirbhate,Vedant recognition system that useful for static ISL
machine learning D. Shinde, Sanam uses a variety of algorithms numeral signs only.
[6] algorithm A.Metkari, Pooja 2020 such a SVM, Random
U. Borkar, and forest and Hierarchical
Mayuri A classification.
Sign language Lionel Pigou(B), Sign language recognition The system only
[7] recognition using Sander Dieleman, system using 2D recognises 20 Italian
convolutional neural Pieter-Jan Convolutions and pooling gestures.
networks Kindermans, and method
Benjamin
Schrauwen
Indian sign language Joyeeta Singha and An Indian sign language The system cannot
recognition using Karen Das recognition system using recognize the letters
eigen value weighted Eigen value weighted ‘H’ and ‘J’ as they are

IJISRT22APR1566 www.ijisrt.com 1213

Volume 7, Issue 4, April – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
[8] euclidean distance 2013 Euclidean distance as a dynamic gestures.
based classification classification technique.
technique
Automatic mexican Francisco Solís, The system presents a This system is trained
sign language David Martínez, computer vision system for only to recognize
recognition using and Oscar automatic recognition of static images. LED
normalized moments Espinoza Mexican Sign reflectors are used
[9] and artificial neural 2016 Language(MSL). It uses a which increases the
networks digital camera and 4 LED cost.
reflectors to reduce
shadows and for improving
segmentation of hand with
background.
AI enabled sign Feng Wen, Zixuan The system presents an It uses sensors which
language recognition Zhang, Tianyiyi artificial intelligence is not cost effective. It
and VR space He, and Chengkuo enabled sign language uses text from able
bidirectional Lee recognition system. It uses bodies for
[10] communication using 2021 sensing gloves which are communication
triboelectric smart configured with 15 process which is not
glove triboelectric sensors, using very practical.
Convolutional Neural
Networks(CNN) to process
the input data.
Table 1

IV. EXISTING SOLUTION processing since sign language comprises both static and
dynamic gestures.
ASL recognition isn't a new problem in computer vision.
Researchers have utilized classifiers from a range of Another existing solution for communication for hearing
categories over the last two decades, which we may generally impaired people is a chat application. Chat programmes have
divide into linear classifiers, neural networks, and Bayesian evolved into a great tool for individuals to connect with one
networks. The research was also based on a variety of input another in a variety of languages. There are many different
sensors, gesture segmentation, feature extraction, and chat software that are utilized by different individuals in
classification approaches. different languages, but there isn't one that allows you to
interact with sign languages. The recent release of low-cost
The proposed system analyzes the American Sign depth sensors, such as the widely used Microsoft Kinect
Language gestures and then converts them into human sensor, has facilitated the development of new gesture
readable text.The majority of the works designed to address detection algorithms. Depth photographs generate a three-
this problem have used one of two approaches: contact-based dimensional model of the scene, which can be used to ease
systems, such as system gloves, or vision-based systems, tasks like people segmentation and tracking, body component
which rely solely on cameras. Contact based method is the detection, and motion prediction.
one where the signer has to wear a hardware glove and do the
signs , and the hand movements get captured . This system is V. PROPOSED SOLUTION
uncomfortable for practical and daily use, despite having an
accuracy of 90%.Static and dynamic recognition are two types Developing an android application aided by machine
of vision-based methods. Static is concerned with the learning techniques to recognize hand gestures can be done
identification of static gestures (two-dimensional images), with ease. Hand gesture recognition shows how fast the
whereas dynamic is concerned with the capturing of motions algorithms detect the gestures in a single shot. The faster and
in real time. This entails the employment of a camera to stable it can be, the user experience will be smoother and
record motion.Vision is a key factor in sign language, and better. The proposed SLR system constitutes the first attempt
every sign language is intended to be understood by one to merge a vision-based approach (i.e., processing of images)
person located in front of another, from this perspective, a with the accurate extraction of skeletal data without
gesture can be completely observable. employing data gloves or other sensors that limit the
Sensor-based devices, such as SignSpeak, were movements of a signer. The application opens to a camera,
employed in many research. This device used a variety of which detects the hand movements of the signer. Hand gesture
sensors, including flex and contact sensors for finger and palm features in the uploaded images are extracted and used to
movements, as well as accelerometers and gyros for hand recognize the type of gesture.
movement; the gloves were then trained to recognise different
gestures using Principal Component Analysis, and each
gesture was then classified into alphabets in real time.In
visual-based SLR, several strategies have been
established.Many people experimented with image and video

IJISRT22APR1566 www.ijisrt.com 1214

Volume 7, Issue 4, April – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
 How different does our sign language interpreter work algorithm,” International Journal of Engineering and
using Convolutional Neural Network as compared to Technology, Volume: 07 Issue: 03, Mar 2020.
previous efforts? [7.] Lionel Pigou(B), Sander Dieleman, Pieter-Jan
 We are developing a real-time sign language interpreter Kindermans, and Benjamin Schrauwen, “Sign language
that can seamlessly recognize and phrase American Sign recognition using convolutional neural networks,” ELIS,
Language gestures. Ghent University, Ghent, Belgium.
 This technology is a first of its kind to be implemented [8.] Joyeeta Singha and Karen Das, “Indian sign language
in an android application which will help a user recognition using eigen value weighted euclidean
understand sign language anywhere anytime with just distance based classification technique,” International
the use of a phone. Journal of Advanced Computer Science and
 In terms of convenience, a desktop or web application Applications, Vol. 4, No. 2, 2013.
would always require a desktop or laptop and it would be [9.] Francisco Solís, David Martínez, and Oscar Espinoza,
impractical to make the disabled person sit in front of a “Automatic mexican sign language recognition using
desktop to communicate with an able body. normalized moments and artificial neural networks,”
 This project intends to educate users about sign language Scientific Research Publishing, 2016.
while helping them to translate sign language to English [10.] Feng Wen, Zixuan Zhang, Tianyiyi He, and Chengkuo
by using the sign language scanner feature. Lee, “AI enabled sign language recognition and VR
space bidirectional communication using triboelectric
VI. CONCLUSION smart glove,” Nature Communications, 2021.

Sign language recognition is a difficult task in today’s

world due to the gap between able-bodied and hearing
impaired people. The proposed system is to make this task
easier with the help of an android application. Sign language
is recognized by taking the input from camera and then
recognizing which hand gesture is made by the user. We have
deduced few algorithms that can be used for sign language
recognition. This system can be used to educate the able-
bodied and bridge the gap between able-bodied and hearing
impaired. It takes real-time hand gestures and gives the
output. The resources being used are well suited with today’s
technologies making it robust and accurate.

REFERENCES

[1.] Lean Karlo S. Tolentino, Ronnie O. Serfa Juan, August

C. Thio-ac, Maria Abigail B. Pamahoy, Joni Rose R.
Forteza, and Xavier Jet O. Garcia, “Static sign language
recognition using deep learning,” International Journal of
Machine Learning and Computing, Vol. 9, No. 6,
December 2019.
[2.] Dimitrios Konstantinidis, Kosmas Dimitropoulos, and
Petros Daras, “Sign language recognition based on hand
and body skeletal data,” ITI-CERTH, 6th km Harilaou-
Thermi, 57001, Thessaloniki, Greece.
[3.] Arpita Halder and Akshit Tayade, “Real-time vernacular
sign language recognition using mediapipe and machine
learning,” International Journal of Research Publication
and Reviews Vol (2) Issue (5) (2021).
[4.] Brandon Garcia and Sigberto Alarcon Viesca, “Real-
time American Sign Language Recognition with
Convolutional Neural Networks”.
[5.] Anamika Srivastava1 and Vikrant Malik, “Review on
sign language detection using machine learning,” Journal
of Critical Reviews ISSN- 2394-5125 Vol 7, Issue 10,
2020.
[6.] Radha S. Shirbhate, Vedant D. Shinde, Sanam
A.Metkari, Pooja U. Borkar, and Mayuri A. Khandge,
“Sign language recognition using machine learning

IJISRT22APR1566 www.ijisrt.com 1215