research-article

UbiEar: Bringing Location-independent Sound Awareness to the Hard-of-hearing People with Smartphones

Authors:

Shangguan Longfei,

Xin WangAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 2

Article No.: 17, Pages 1 - 21

https://doi.org/10.1145/3090082

Published: 30 June 2017 Publication History

Abstract

Non-speech sound-awareness is important to improve the quality of life for the deaf and hard-of-hearing (DHH) people. DHH people, especially the young, are not always satisfied with their hearing aids. According to the interviews with 60 young hard-of-hearing students, a ubiquitous sound-awareness tool for emergency and social events that works in diverse environments is desired. In this paper, we design UbiEar, a smartphone-based acoustic event sensing and notification system. Core techniques in UbiEar are a light-weight deep convolution neural network to enable location-independent acoustic event recognition on commodity smartphons, and a set of mechanisms for prompt and energy-efficient acoustic sensing. We conducted both controlled experiments and user studies with 86 DHH students and showed that UbiEar can assist the young DHH students in awareness of important acoustic events in their daily life.

References

[1]

Audio Analytic. 2017. https://www.audioanalytic.com/. (2017).

[2]

James Bergstra, Norman Casagrande, Dumitru Erhan, Douglas Eck, and Balázs Kégl. 2006. Aggregate features and AdaBoost for music classification. Machine Learning 65, 2-3 (2006), 473--484.

Digital Library

[3]

Sourav Bhattacharya and Nicholas D Lane. 2016. Sparsification and Separation of Deep Learning Layers for Constrained Resource Inference on Wearables. In Proc. SenSys. ACM, 176--189.

Digital Library

[4]

Danielle Bragg, Nicholas Huynh, and Richard E Ladner. 2016. A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In Proc. ASSETS. ACM, 3--13.

Digital Library

[5]

J-F Cardoso. 1998. Multidimensional independent component analysis. In Proc. ICASSP, Vol. 4. IEEE, 1941--1944.

[6]

Andrew Collette. 2015. HDF5 for Python. http://www.h5py.org/. (2015).

[7]

Simon Dixon. 2006. Onset detection revisited. In Proc. DAFx.

[8]

Google. 2016. android.util.LruCache. https://developer.android.com/reference/android/util/LruCache.html. (2016).

[9]

Benjamin M Gorman. 2014. VisAural:: a wearable sound-localisation device for people with impaired hearing. In Proc. ASSETS. ACM, 337--338.

Digital Library

[10]

Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Proc. ICLR.

[11]

Toni Heittola, Annamaria Mesaros, Tuomas Virtanen, and Antti Eronen. 2011. Sound event detection in multisource environments using source separation. In Proc. CHiME. 36--40.

[12]

F Ho-Ching, Jennifer Mankoff, and James A Landay. 2003. Can you see what i hear?: the design and evaluation of a peripheral sound display for the deaf. In Proc. CHI. ACM, 161--168.

Digital Library

[13]

Forrest N Iandola, Matthew W Moskewicz, Khalid Ashraf, Song Han, William J Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size. arXiv preprint arXiv:1602.07360 (2016).

[14]

Keisuke Imoto and Nobutaka Ono. 2015. Acoustic scene analysis from acoustic event sequence with intermittent missing event. In Proc. ICASSP. IEEE, 156--160.

[15]

Leeo Inc. 2017. Meet the Leeo Smart Alert Nightlight. https://www.leeo.com/meet-the-leeo-smart-alert-nightlight/. (2017).

[16]

Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, and Jon E Froehlich. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. In Proc. CHI. ACM, 241--250.

Digital Library

[17]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proc. MM. ACM, 675--678.

Digital Library

[18]

Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proc. CVPR. IEEE, 1725--1732.

Digital Library

[19]

Keras. 2016. Keras: Deep Learning library for Theano and TensorFlow. https://keras.io/. (2016).

[20]

Hamed Ketabdar and Tim Polzehl. 2009. Tactile and Visual Alerts for Deaf People by Mobile Phones. In Proc. ASSETS. ACM, 253--254.

Digital Library

[21]

Adel Khalil, James Sun, Yu Zhang, and Gordon Poole. 2014. RTM noise attenuation and image enhancement using time-shift gathers. In Proc. EAGE Conference and Exhibition.

[22]

George Kimura. 2017. Ambient Noise Database. http://www.ntt-at.com/product/noise-DB/. (2017).

[23]

Sergei Kochkin. 2000. MarkeTrak V: Why my hearing aids are in the drawer: The consumers’ perspective. The Hearing Journal 53, 2 (2000), 34--36.

[24]

Gregoire Lafay. 2017. IEEE DCASE 2016 Challenge-Task 2-Train/Development Datasets. https://archive.org/details/dcase2016_task2_train_dev. (2017). Published February 10, 2016.

[25]

Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, Lei Jiao, Lorena Qendro, and Fahim Kawsar. 2016. DeepX: A software accelerator for low-power deep learning inference on mobile devices. In Proc. IPSN. ACM, 1--12.

Digital Library

[26]

Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proc. UbiComp. ACM, 283--294.

Digital Library

[27]

Joseph Lee Rodgers and W Alan Nicewander. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician (1988), 59--66.

[28]

Andy Liaw and Matthew Wiener. 2002. Classification and regression by randomForest. R news 2, 3 (2002), 18--22.

[29]

Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).

[30]

Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. 2015. Sparse convolutional neural networks. In Proc. CVPR. IEEE, 806--814.

[31]

Sicong Liu and Junzhao Du. 2016. Poster: MobiEar-Building an Environment-independent Acoustic Sensing Platform for the Deaf using Deep Learning. In Proc. MobiSys. ACM, 50--50.

Digital Library

[32]

Nitin N Lokhande, Navnath S Nehe, and Pratap S Vikhe. 2012. Voice activity detection algorithm for speech recognition applications. In Proc. ICCIA. IJCA.

[33]

Hong Lu, Wei Pan, Nicholas D Lane, Tanzeem Choudhury, and Andrew T Campbell. 2009. SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proc. MobiSys. ACM, 165--178.

Digital Library

[34]

Hong Lu, Jun Yang, Zhigang Liu, Nicholas D Lane, Tanzeem Choudhury, and Andrew T Campbell. 2010. The Jigsaw continuous sensing engine for mobile phone applications. In Proc. SenSys. ACM, 71--84.

Digital Library

[35]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research Nov (2008), 2579--2605.

[36]

Tara Matthews, Janette Fong, and Jennifer Mankoff. 2005. Visualizing non-speech sounds for the deaf. In Proc. ASSETS. ACM, 52--59.

Digital Library

[37]

Abby McCormack and Heather Fortnum. 2013. Why do people fitted with hearing aids not wear them? International Journal of Audiology 52, 5 (2013), 360--368.

[38]

Annamaria Mesaros, Toni Heittola, Antti Eronen, and Tuomas Virtanen. 2010. Acoustic event detection in real life recordings. In Proc. EUSIPCO. IEEE, 1267--1271.

[39]

Matthias Mielke and Rainer Brueck. 2015. Design and evaluation of a smartphone application for non-speech sound awareness for people with hearing loss. In Proc. EMBC. IEEE, 5008--5011.

[40]

OtoSense. 2017. https://www.otosense.com/. (2017).

[41]

Jouni Paulus and Tuomas Virtanen. 2005. Drum transcription with non-negative spectrogram factorisation. In Proc. EUSIPCO. IEEE, 1--4.

[42]

Karol J Piczak. 2015. Environmental sound classification with convolutional neural networks. In Proc. MLSP. IEEE, 1--6.

[43]

Ilyas Potamitis, Stavros Ntalampiras, Olaf Jahn, and Klaus Riede. 2014. Automatic bird sound detection in long real-field recordings: Applications and tools. Applied Acoustics 80 (2014), 1--9.

[44]

Tauhidur Rahman, Alexander Travis Adams, Mi Zhang, Erin Cherry, Bobby Zhou, Huaishu Peng, and Tanzeem Choudhury. 2014. BodyBeat: a mobile system for sensing non-speech body sounds. In Proc. MobiSys. ACM, 2--13.

Digital Library

[45]

Ann Mette Rekkedal. 2012. Assistive hearing technologies among students with hearing impairment: Factors that promote satisfaction. Journal of Deaf Studies and Deaf Education 17, 4 (2012), 499--517.

[46]

Android Studio. 2017. Battery Historian Charts. https://developer.android.com/studio/profile/battery-historian-charts.html. (2017).

[47]

Denis Tomé, Luca Bondi, Luca Baroffio, Stefano Tubaro, Emanuele Plebani, and Danilo Pau. 2016. Reduced memory region based deep Convolutional Neural Network detection. In Proc. ICCE-Berlin. IEEE, 15--19.

[48]

Emmanuel Vincent, Nancy Bertin, Rémi Gribonval, and Frédéric Bimbot. 2014. From blind to guided audio source separation: How models and side information can improve the separation of sound. Signal Processing Magazine 31, 3 (2014), 107--115.

[49]

In-Chul Yoo and Dongsuk Yook. 2008. Automatic sound recognition for the hearing impaired. Transactions on Consumer Electronics (2008), 2029--2036.

Digital Library

Cited By

Li XLiu SZhou ZGuo BXu YYu Z(2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643560
V SA HJ DI PThiyagu T(2024)IoT based Wireless Alert System for Individuals with Impaired Hearing2024 3rd International Conference on Sentiment Analysis and Deep Learning (ICSADL)10.1109/ICSADL61749.2024.00115(662-666)Online publication date: 13-Mar-2024
https://doi.org/10.1109/ICSADL61749.2024.00115
Pang BLiu SWang HGuo BWang YWang HSheng ZWang ZYu Z(2023)AdaMEC: Towards a Context-adaptive and Dynamically Combinable DNN Deployment Framework for Mobile Edge ComputingACM Transactions on Sensor Networks10.1145/363009820:1(1-28)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3630098
Show More Cited By

Index Terms

UbiEar: Bringing Location-independent Sound Awareness to the Hard-of-hearing People with Smartphones
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Tender: Smartphone Application for Braille Reading
ACIT '19: Proceedings of the 7th ACIS International Conference on Applied Computing and Information Technology

Visually impaired people use Braille in their daily life. Braille text is printed in many ways like a handrail of stairs, guide maps, and so on. Visually impaired people read these Braille guides to navigate themselves. However, misprinted Braille may ...
Non-visual mainstream smartphone camera interactions for blind and low-vision people
ShadowRine: Accessible game for blind users, and accessible action RPG for visually impaired gamers
2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
Though some games for visually impaired persons have been developed, most of games that use only auditory information present challenges for sighted persons. Moreover, unfortunately, it is still difficult for visually impaired persons to play the same ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 2

June 2017

665 pages

EISSN:2474-9567

DOI:10.1145/3120957

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2017

Accepted: 01 May 2017

Revised: 01 April 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed

Funding Sources

CETC shining Star Innovation.
Fundamental Research Funds for the Central Universities
National Natural Science Foundation of China (NSFC)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
1,249
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)3

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li XLiu SZhou ZGuo BXu YYu Z(2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643560
V SA HJ DI PThiyagu T(2024)IoT based Wireless Alert System for Individuals with Impaired Hearing2024 3rd International Conference on Sentiment Analysis and Deep Learning (ICSADL)10.1109/ICSADL61749.2024.00115(662-666)Online publication date: 13-Mar-2024
https://doi.org/10.1109/ICSADL61749.2024.00115
Pang BLiu SWang HGuo BWang YWang HSheng ZWang ZYu Z(2023)AdaMEC: Towards a Context-adaptive and Dynamically Combinable DNN Deployment Framework for Mobile Edge ComputingACM Transactions on Sensor Networks10.1145/363009820:1(1-28)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3630098
Park JLee KLee SZhang MKo J(2023)AttFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109177:3(1-31)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610917
Huang JChhabria HJain D(2023)“Not There Yet”: Feasibility and Challenges of Mobile Sound Recognition to Support Deaf and Hard-of-Hearing PeopleProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608431(1-14)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608431
Do HDang QHuang JJain D(2023)AdaptiveSound: An Interactive Feedback-Loop System to Improve Sound Recognition for Deaf and Hard of Hearing UsersProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608390(1-12)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608390
Wang YYu ZLiu SZhou ZGuo B(2023)Genie in the ModelProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808157:1(1-29)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580815
Huang JWang YZou YWu KNi L(2023)Ubiquitous WiFi and Acoustic Sensing: Principles, Technologies, and ApplicationsJournal of Computer Science and Technology10.1007/s11390-023-3073-538:1(25-63)Online publication date: 31-Jan-2023
https://doi.org/10.1007/s11390-023-3073-5
Alsharif BIlyas M(2023)Internet of Things Technologies in Healthcare for People with Hearing ImpairmentsIoT and Big Data Technologies for Health Care10.1007/978-3-031-33545-7_21(299-308)Online publication date: 24-May-2023
https://doi.org/10.1007/978-3-031-33545-7_21
Jain DNgo HPatel PGoodman SNguyen KGrossman-Kahn RFindlater LFroehlich J(2022)SoundWatchCommunications of the ACM10.1145/353144765:6(100-108)Online publication date: 20-May-2022
https://dl.acm.org/doi/10.1145/3531447
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents