International Journal of Pure and Applied Mathematics

Volume 116 No. 21 2017, 371-377

ISSN: 1311-8080 (printed version); ISSN: 1314-3395 (on-line version)
url: http://www.ijpam.eu
Special Issue
ijpam.eu

SOFTWARE ASSISTANCE TO DEAF AND DUMB USING HANDSHAPE ALGORITHM

1
V.Purushotham Vijay Naidu,2M.R.Sai Hitesh, 3T.Dhikhi
1,2,3
Department of Computer Science and Engineering
Saveetha School of Engineering Saveetha University, Chennai.

Abstract: This application helps the deaf and dumb communication between deaf and normal people in their
person to communicate with the rest of the world using daily activities using sign language and video relay
sign language communication plays an important role for service. Video technology continues to improve and one
human beings. Communication is treated as a life skill. day may be the preferred means of mobile
Keeping these important words in mind we present this communication among the deaf.. There are mainly three
project to mainly focus on aiding the speech impaired parts:
and paralyzed patients. Our work helps in improving the • Speech-Recognition Engine
communication with the deaf and dumb. Speech-to-sign • Database and
technology and VRS enables audible language translation • Recognized Text
on smart phones with signing and application has Under Speech- Recognition Engine we include Sign to
characters feature in mobile without dialing number uses Speech with the help of outfit – 7 and Video Relay
a technology that translates spoken and written words Service (VRS - enables audible language translation on
into sign language with video. Interaction between smart phones with signing) technologies and Speech to
normal people with blind person is very difficult because Sign using Mimix technology. Secondly, SQL lite
of communication problems. There are many database is used to store the inputs given by the
applications available in the market to help the blind application user which are then viewed from the
people to interact with the world. Voice-based email and database. Finally, Text (or video) recognized through
chatting systems are available to communicate with each Mimix makes it easier to have a clear, two-way
other by blinds. This helps to interact with persons by communication with a deaf without having to know sign
blind people. This application includes a voice based, language. It works based on recorder. This feature, along
text based and video based interaction approach. with the power of JSON (Java Script Object Notation)
which establishes it is a great choice for incorporation in
Keywords: video real services, json, video intereter. the proposed architecture.

1. Advent 2. System architecture

Android application have shown a dramatic improvement The goal of this project is to determine gesture
in their practicality to a degree wherever it's currently recognition that might enable the deaf to converse with
attainable to possess mobile phone execute Java the hearing people remotely and is done by a JSON
programs. As a result, cellular users throughout the globe interpreter. We are not aware of any research which aim
square measure currently ready to scan and write email, is to provide un-intermediated mobile communication
browse websites and play java games victimization their between deaf and hearing people, each conversing using
cellular phones. This trend has promoted on propose the their own natural languages. Hence the project has
employment of humanoid application for higher provided the idea of implementing communication
communication. Before SMS/MMS, deaf folks seldom between deaf and hearing people in day-to-day life as
used mobile phones. Currently texting permits deaf folks shown in figure 1.1. Initially, mobile search functionality
remotely to speak with each deaf and hearing parties. must recognize either ASL (American Sign Language)
Our project isan alternative for gesture detection using Text or voice and convert it to both text message as well
image processing technique between deaf people which as video for relevant input. ASL2TXT enable sign
overcomes the above technique and paves the way for the

371
 International Journal of Pure and Applied Mathematics Special Issue

language finger spelling communication (signs displayed broadly, to include any person who communicates
in the keyboard) take text and display video. primarily using American Sign Language (ASL). Some
hearing people use both audible and sign languages, we
use the term “hearing” to suggest a person who speaks in
audible language and does not sign. Technical literature
uses the term “translation” in favor of “interpretation,”
So we follow the standard for that reason.
2.2.2 Hand Shape Recognition Algorithm
We can now build a cost matrix (E) for two shapes,
where the elements (Ei,j) is the cost between the log-
polar histograms of the I th and j th point of the first and
second shapes respectively. This cost matrix, we can then
calculate the optimal association between every point in
the first shape to that of the second shape using the
Hungarian method. This results in a permutation π(i),
where the sum d = k Ck,π(k) is minimum. The value d
will also be used as a matching cost between two shapes.
2.2.3 Video-Relay Service
Figure1. System architecture
Using VRS communicate in sign language, they are able
The process abounds the following: to more fully express themselves through facial
• A deaf person signs expressions and body language, which cannot be
• Software translates signs into text (and video) expressed in text. A VRS call flows back and forth just
• The hearing person reads it (and view it) like a telephone conversation between two hearing
• The hearing person and deaf people speaks into persons. For example, the parties can interrupt each
microphone other. VRS calls may be made between ASL users and
• Software translates voice into text (and ASL video) hearing persons speaking either English or Spanish.
• The deaf person reads it (and sees ASL video)
2.2.4 Texting and Speech Translation
2.1 Features of the project
SMS/MMS enables signers to communicate with both
• Without dialing number we can communicate to deaf and hearing parties. Video chat technology
other like face to face communication. continues to improve and one day may be the preferred
• It does not require large amount of storage as it means of mobile communication among the deaf. Google
uses the Hand speak support through online. Translate allows users to type text in their native tongues
• The sign words are signed in the same order as and receive textual and audible translations in several
letters appear in English alphabets. vernaculars.
• This project prepares individuals to work as 2.3 Human Interpreters
interpreter/translators facilitating and mediating
communication between Deaf/Hard of Hearing and For lengthy, sophisticated conversations it is difficult to
hearing people. imagine a workable computer system that would improve
• Accurate and appropriate transfer of a message over human interpreters. The ability of human
from a source language into a target language from the interpreters to perform language translation may always
point of view of style and culture exceed a computer’s ability. Hence in some situations,
• Learn the culture and history of Deaf people to mobile TXT2ASL translation may be more convenient
better understanding communication between Deaf and than a relay or even a handwritten note. Like texting, we
Hearing individuals. envision TXT2ASL as an enhancement to smart phones
and other mobile devices, not as a replacement for human
2.2 Domain Introduction interpreters.
2.2.1 Deaf-Hearing Communication
Since all deaf are not using sign language in their day to
day life, for ease of exposition, we define the term “deaf”

372
 International Journal of Pure and Applied Mathematics Special Issue

3. Literature Survey RAM capacity. With high RAM size more number of
sign letters can be accommodated into single android
The purpose of the Literature Survey is to give the brief
activity. An android application has been designed to
overview and also to establish complete information
demonstrate the implementation of Viola-Jones
about the reference papers. The goal of Literature Survey
algorithm with LBP features for hand gesture
is to completely specify the technical details related to
recognition. The Open CV libraries have been used to
the main project in a concise and unambiguous manner.
develop the application and android app is available in
In [1] Different approaches are employed by
Google Play store for free. During runtime, camera
completely different researchers for recognition of varied
continuously captures the image of sign formation which
hand gestures that were enforced in numerous fields. the
will be processed and matched with the database. Here,
total approaches can be divided into 3 broad classes
for database the .xml file generated after the cascade
Hand segmentation approaches Feature extraction
training will be used. Further, detected gesture will be
approaches and Gesture recognition approaches.
marked with the rectangle and appropriate sign will be
All the offered systems don't seem to be
named. This implemented algorithm is not a robust and
moveable and not reasonable to poor folks. This paper
real time. Hence we are using the already recorded video
introduces new automaton application which can
stored in a cloud storage which is considered to be the
discover the Indian signing via mobile camera and
easiest way of interpreting the users input in relevant
converts into corresponding text or voice output. This
manner. This above algorithm does not prove its
application uses bound image process techniques to
efficiency in any sort of background but our project
check the input with the already hold on signs and needs
overcomes this issue to the larger extent.
solely automaton phone and doesn't need any special
In [3] a number of developing countries continue
markers or magic gloves on the hand of the user. This
to provide educational services to students with
includes totally different sizes of gestures within the
disabilities in "segregated" schools. Also all students,
pictures, totally different background of pictures, totally
regardless of their personal circumstances, have a right of
different orientations and angle of gestures, etc. the
access to and participation in the education system,
various illumination for various pictures too expose a
according to their potential and ability. However, with
drag. currently our system provides sixty fifth of correct
the rapidly growing population and increasing number of
predicting and that we ar acting on rising its potency.
people with blindness along with other disabilities, need
therefore we tend to took the thought of implementing
for use of technology in the field of education has
the gesture video with the assistance of hand speak
become imminent. In this project, through the use of
technology that helps the deaf folks to look at their
speech technology, attempts to provide solutions for
relevant signing video supported the text given as input.
some of these issues by creating an interactive system.
we tend to embody the thought of providing the link to
Thus, the application will help in creating an
the applying that helps in extracting the video. It proves
environment that provides equal opportunities for all the
its most potency.
students in taking up competitive exams. Application
In [2] Sign language is used as a communication
automates the examination process through reading out
medium among deaf and dumb people to convey the
questions to the user and receiving their input orally.
message with each other. In order to bridge the gap in
Application also provides accessories for other
communication among deaf , dumb community and
requirements, like knowing the time remaining, during
normal community, lot of research work has been carried
exams. Use of this application shall benefit students with
out to automate the process of sign language
Learning disabilities, including dyslexia and dysgraphia,
interpretation with the help of image processing and
Poor or limited motor skills, Vision impairments,
pattern recognition techniques. This paper proposes
Physical disabilities, Limited English Language The
optimized approaches of implementing the famous Viola
application will help the students with reading writing
Jones algorithm with LBP (Local Binary Pattern)
disabilities (students with dyslexia or dysgraphia) as well
features for hand gesture recognition which will
as sensory disabilities (blind or handicapped). This
recognize Indian sign language gestures in a real time
system has implemented proposed system with three
environment. An optimized algorithm has been
individual section and features like timer and result for
implemented in the form of an android application and
each one. For the implemented sections and subjects the
tested with real time data. Optimized classifier can be
system is running perfectly and flawlessly. There by
obtained with less number of positive samples provided
proposed application is suitable for use in real-time with
by considering the suitable number of stages and 1:2
high performance. We took the idea of using voice over
ratio of positive to negative images. It has been proved
text technology from the above proposed system because
through this that sign recognition depends on mobile

373
 International Journal of Pure and Applied Mathematics Special Issue

on considering the deaf people, they either have speech is the first step in the requirements analysis, process; it
ability or be a dumb which again depends on their birth. lists the requirements of the particular software system
It will be a revolutionary change that will benefit hearing including functional, performance, and security
impaired people, boost their confidence and put them requirements. The requirements also provide usage
with regular people. scenarios from a user, an operational and an
In [4]for the past many decades, designers have administrative perspective. The purpose of the software
processed speech for a good style of applications starting requirements specification is to provide a detailed
from mobile communications to automatic reading overview of the software of the project, its parameters
machines. Speech has not been used abundant within the and goals. This describes a project target audience and its
field of physics and computers owing to the complexness user interface, hardware and software requirements. It
and style of speech signals and sounds. However, with defines how the client, team and audience see the project
trendy processes, algorithms, and strategies we are able and its functionality.
to method speech signals simply and acknowledge the
4.1 Hardware Requirement
text. The system acquires speech at run time through a
mike and processes the sampled speech to acknowledge Processor : Pentium P4
the verbalized text. The recognized text are often keep in Motherboard : Genuine Intel
an exceedingly file. we tend to ar developing this on RAM : Min 1 GB
humanoid platform exploitation eclipse work bench. Our Hard Disk : 80 GB
speech-to-text system directly acquires and converts
4.2 Software Requirement
speech to text. It will supplement different larger
systems, giving users a unique selection for knowledge Operating system : Windows XP
entry. A speech-to-text system also can improve system Technology Used : Android
accessibility by providing knowledge entry choices for IDE : Eclipse
blind, deaf, or physically incapacitated users. Voice SMS Emulators : AVD
is AN application developed during this work that Plug-in : ADT plug-in
permits a user to record and convert spoken messages Tools used : Android SDK.
into SMS text message. User will send messages to the 4.3 Language Specification
entered number. Speech recognition for Voice uses a
method supported hidden Markov |Markoff| Andre Mark This project can be implemented only in JAVA because
off |mathematician} models (HMM - Hidden Markov Android supports only JAVA for user applications.
Model). it's presently the foremost undefeated and most
versatile approach to speech recognition. exploitation the 4.4 JAVA
speech recognizer, that works over the web, permits In Java could be a programing language and computing
abundant quicker processing. Another advantage is that platform initial founded by Sun Microsystems in 1995.
the abundant larger databases that ar used. The accuracy There area unit numerous applications and websites that
of the system has considerably inflated and become a lot may not work unless you've got Java put in, and
of accessible to everybody. even supposing this method additional area unit created a day. Java is quick, secure,
has abundant advantage in our project Speech and reliable. From laptops to data centre’s, game
recognition is finished via the web, connecting to consoles to scientific supercomputers, cell phones to the
Google's server. the appliance is customized to input web, Java is everywhere! Java is Platform freelance. Java
messages in English. Double faucet on the contact space is Associate in Nursing object-oriented programing
displayscontacts that has the interface with contact language developed at the start by James goose and
numbers from mobile and allows user to settle on number colleagues at Sun Microsystems. It implements a robust
on that message are going to be sent once pressing button security model, that prevents compiled Java programs
Send. client receives response in an exceedingly very from illicitly accessing resources on the system wherever
little cloud (toast) if message has been sent. they execute or on the network. in style World-Wide
internet browsers, yet as some World-Wide internet
4. System Requirement servers and different systems implement Java
In The Software Requirement Specification is to interpreters. These area unit accustomed show interactive
completely specify the technical requirements for the user interfaces, and to script behaviour on these systems.
software product in a concise and unambiguous
manner.The requirements specification is a technical
specification of requirements for the software products. It

374
 International Journal of Pure and Applied Mathematics Special Issue

4.5 Eclipse most of anglophone Canada. sign language signs have

variety of sound elements, together with movement of
In Eclipse could be a platform that has been designed
the face and body also because the hands. sign language
from the bottom up for building integrated internet and
isn't a kind of pantomime, however iconicity will play a
application development tooling. By design, the platform
bigger role in sign language than in spoken languages.
doesn't offer a good deal of user practicality by itself. the
English loan words area unit usually borrowed through
worth of the platform is what it encourages: fast
sign language, though sign language synchronic
development of integrated options supported aplug-in
linguistics is unrelated thereto of English. sign language
model.Eclipse provides a standard computer programme
has verbal agreement and a spectual marking, and
(UI) model for operating with tools. it's designed to run
includes a productive system of forming agglutinative
on multiple operative systems whereas providing sturdy
classifiers. several linguists believe sign language to be a
integration with every underlying OS. Plug-ins will
subject-verb-object (SVO) language, however there area
program to the Eclipse moveable arthropod genus and
unit many various proposals to account for sign language
run unchanged on any of the supported operative
ordination.This keyboard is represented using Signed
systems. At the core of Eclipse is design for dynamic
English (SE). The pattern of representation exhibits each
discovery, loading, and running of plug-ins. The platform
letter of English Alphabet in its Hand Signs. These
handles the supplying of finding and running the correct
alphabets are used by the user to communicate with the
code. The platform UI provides a customary user
normal people as a messaging service. When the deaf
navigation model. In programing, Eclipse is Associate in
user sends the message to the hearing party, it is received
Nursing integrated development surroundings (IDE). It
as text message on the other side.
contains a base space Associate in Nursingd an extensile
plug-in system for customizing the surroundings. Written
largely in Java, Eclipse will be accustomed develop
applications. By means that of assorted plug-ins, It may
be used underneath the subsequent languages: enzyme,
ABAP,C,C++,COBOL, Fortran, Haskell, JavaScript,
Lasso, Lua, Natural, Perl, PHP, Prolog, Python, R, Ruby,
Scala, Clojure, Groovy, Scheme, and Erlang. It may be
accustomed develop packages for the package
Mathematica. Development environments embody the
Eclipse Java development tools (JDT) for Java and Scala,
Eclipse CDT for C/C++ and Eclipse PDT for PHP, Figure 2. Sign Recognition
among others.
4.7 Sign Language
• A deaf person signs through the sign language
keyboard displayed in an application. In[10] Last module of our project contains the main Sign
• Software translates signs into text and ASL video Language Video. This video is displayed on theScreen of
through interpretation process. the deaf party . Sign Language video is obtained from the
• The hearing person read it or view the sign language JSON and the Hand Speak websites. These websites
video extracted through hand speak includes most of the words from the ASL Dictionary. For
Example: The below picture represents the meaningful
• The hearing person and deaf people speak into
video for the deaf people in the sign language. Good
microphone which is recognized through Google server.
Morning is displayed in the video as follows:
• Software translates voice into text and ASL video
interpreted through JSON (Java Script Object Notation)
• The deaf person reads it and sees ASL video as the
sent SMS is stored in the inbox which can be seen at any
time.
4.6 Sign to text (or) sign recognition
In[9] second module includes the linguistic
communication input, that is displayed because the
keyboard on the mobile screen. yank linguistic
communication (ASL) is that the predominant linguistic Figure 3. sign language
communication of deaf communities within the us and

375
 International Journal of Pure and Applied Mathematics Special Issue

The sign language recognition can beimplemented number.expand of curiosity and consequent usages of
through our project by giving a link to the particular web wearable devices in near future, the price of the gadgets
server. Intent can be used implicitly to get the video as can be expected to fall leading to of wide application
the output in the users mobile phone.The request is given within the society
to the server and is hit to the server with get/post method,
References
where the output is received as the String and converted
to the video. This completes the module and gives a way [1] RaghavendharReddy.B, Mahender.E, “Speech to
to communicate between deaf and the hearing party. Text Conversion using Android Platform”, International
Journal of Engineering Research and Applications
5. Result
(IJERA) Vol. 3, Issue 1, pp.253-258, January -February
By using sign language app we can communicate deaf 2013
people easily with the help of this application. First we [2] j.Prathyusha, g.Sandhya, v.Krishna Reddy,” An
need to register this application with specified details the Improvised Partition-Based Workflow Scheduling
message will convert into video message with the help of Algorithm”, International Innovative Research Journal
video real services. The internet is necessary for of Engineering and Technology,vol 2,no 4,pp 120-
buffering the video and coversion of it. By using 123,2017.
handshape algorithm this application has been [3] Sangeetha, K. and Barathi Krishna, L. “Gesture
developed.It will be very useful to deaf people day to day Detection for Deaf and Dumb People ” International
life. Journal of Development Research Vol. 4, Issue, 3, pp.
749-752, March, 2014
[4] ShanmukhaSwamy, Chethan M P,
MahanteshGatwadi, “Indian Sign Language Interpreter
with Android Implementation”, International Journal of
Computer Applications (0975 – 8887) Volume 97–
No.13, pp. 36-41, July 2014
[5] SinoraGhosalkar
,SaurabhPandey,ShaileshPadhra,TanviApte, “Android
Application on Examination Using Speech Technology
for Blind People ”, International Journal of Research in
Computer and Communication Technology, Vol 3,
Issue3,March-2014
[6] Tapas Kumar Patra, BiplabPatra, Puspanjali
Mohapatra, “Text to Speech Conversion with
Phonematic Concatenation”, International Journal of
Electronics Communication and Computer Technology
(IJECCT) Volume 2 Issue 5 , pp.223-226, (September
2012)
[7] http//educationportal.com/articles/Sign_Language_
Interpreter_Job_Description_Duties_and_Requirements.
html
[8] http//www.healthyhearing.com/content/articles/Tec
hnology/Wireless/50160-The-best-phone-apps-for-
videorelay
6. Conclusion [9] https//play.google.com/store/apps/details?id=me.mi
By using this application deaf person can easily interact mix.roid&hl=en
with normal person anywhere, and he can also use this [10] http//www.googblogs.com/uncategorized/outfit-7s-
application for mobile sign translation using VSR and by talking-friends-built-on-google-app-engine-recently-hit-
using UTF-7 he can communicate in daily activates onebillion-downloads/
without dialing number. We can use this application for [11] http://beta.json_generator.com/api/json/get/AFa87
mobile sign translation using VRS, and with UTF-7 6c
communication can be made without dialing

376
377
378