research-article

Listening through a Vibration Motor

Authors:

Romit Roy ChoudhuryAuthors Info & Claims

MobiSys '16: Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services

Pages 57 - 69

https://doi.org/10.1145/2906388.2906415

Published: 20 June 2016 Publication History

Abstract

This paper demonstrates the feasibility of using the vibration motor in mobile devices as a sound sensor, almost like a microphone. We show that the vibrating mass inside the motor -- designed to oscillate to changing magnetic fields -- also responds to air vibrations from nearby sounds. With appropriate processing, the responses become intelligible, to the extent that humans can understand the vibra-motor recorded words with greater than 80% average accuracy. Even off-the-shelf speech recognition softwares are able to decode at 60% accuracy, without any training or machine learning. We present our overall techniques and results through a system called VibraPhone, and discuss implications to both sensing and security.

References

[1]

Cmu sphinx. http://cmusphinx.sourceforge.net. Last accessed 6 December 2015.

[2]

Sound pressure level chart. http://www.sengpielaudio.com/TableOfSoundPressureLevels.htm. Last accessed 6 December 2015.

[3]

Top 10000 words from google's trillion word corpus. https://github.com/first20hours/google-10000-english. Last accessed 6 December 2015.

[4]

Unit of sound pressure level. http://trace.wisc.edu/docs/2004-About-dB/. Last accessed 9 December 2015.

[5]

Vibraphone project webpage. http://synrg.csl.illinois.edu/vibraphone/. Last accessed 9 December 2015.

[6]

Association, I. P. Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge University Press, 1999.

[7]

Brown, A. International phonetic alphabet. The Encyclopedia of Applied Linguistics (2013).

[8]

Chennoukh, S., Gerrits, A., Miet, G., and Sluijter, R. Speech enhancement via frequency bandwidth extension using line spectral frequencies. In Acoustics, Speech, and Signal Processing, 2001. Proceedings.(ICASSP'01). 2001 IEEE International Conference on (2001), vol. 1, IEEE, pp. 665--668.

[9]

Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G. J., Durand, F., and Freeman, W. T. The visual microphone: Passive recovery of sound from video. ACM Trans. Graph 33, 4 (2014), 79.

Digital Library

[10]

Eargle, J. The Microphone Book: From mono to stereo to surround-a guide to microphone design and application. CRC Press, 2012.

[11]

Fant, G. Acoustic theory of speech production: with calculations based on X-ray studies of Russian articulations, vol. 2. Walter de Gruyter, 1971.

[12]

Farina, A. Simultaneous measurement of impulse response and distortion with a swept-sine technique. In Audio Engineering Society Convention 108 (2000), Audio Engineering Society.

[13]

Fausti, P., and Farina, A. Acoustic measurements in opera houses: comparison between different techniques and equipment. Journal of Sound and Vibration 232, 1 (2000), 213--229.

[14]

Feinberg, D. R., Jones, B. C., Little, A. C., Burt, D. M., and Perrett, D. I. Manipulations of fundamental and formant frequencies influence the attractiveness of human male voices. Animal Behaviour 69, 3 (2005), 561--568.

[15]

Gales, M. J. Maximum likelihood linear transformations for hmm-based speech recognition. Computer speech & language 12, 2 (1998), 75--98.

[16]

Galili, I., Kaplan, D., and Lehavi, Y. Teaching faraday's law of electromagnetic induction in an introductory physics course. American journal of physics 74, 4 (2006), 337--343.

[17]

Gemmeke, J. F., Virtanen, T., and Hurmalainen, A. Exemplar-based sparse representations for noise robust automatic speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on 19, 7 (2011), 2067--2080.

Digital Library

[18]

Hayes, B. Introductory phonology, vol. 32. John Wiley & Sons, 2011.

[19]

Hillenbrand, J., and Sessler, G. M. High-sensitivity piezoelectric microphones based on stacked cellular polymer films (l). The Journal of the Acoustical Society of America 116, 6 (2004), 3267--3270.

[20]

Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.-r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE 29, 6 (2012), 82--97.

[21]

Huggins-daines, D., Kumar, M., Chan, A., Black, A. W., Ravishankar, M., and Rudnicky, A. I. Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices. In in Proceedings of ICASSP (2006).

[22]

Jax, P., and Vary, P. On artificial bandwidth extension of telephone speech. Signal Processing 83, 8 (2003), 1707--1719.

Digital Library

[23]

Johnson, C. D. Accelerometer principles. Process Control Instrumentation Technology (2009).

[24]

Keele Jr, D. Low-frequency loudspeaker assessment by nearfield sound-pressure measurement. Journal of the audio engineering society 22, 3 (1974), 154--162.

[25]

Kontio, J., Laaksonen, L., and Alku, P. Neural network-based artificial bandwidth expansion of speech. Audio, Speech, and Language Processing, IEEE Transactions on 15, 3 (2007), 873--881.

Digital Library

[26]

Laaksonen, L., Kontio, J., and Alku, P. Artificial bandwidth expansion method to improve intelligibility and quality of amr-coded narrowband speech. In ICASSP (1) (2005), pp. 809--812.

[27]

Ladefoged, P. The revised international phonetic alphabet. Language (1990), 550--552.

[28]

Lapteva, O. Speaker Perception and Recognition: An Integrative Framework for Computational Speech Processing. kassel university press GmbH, 2011.

[29]

McGrath, W. Technique and device for through-the-wall audio surveillance, Mar. 30 2005. US Patent App. 11/095,122.

[30]

Michalevsky, Y., Boneh, D., and Nakibly, G. Gyrophone: Recognizing speech from gyroscope signals. In Proc. 23rd USENIX Security Symposium (SEC'14), USENIX Association (2014).

Digital Library

[31]

Moreno, P. J., and Stern, R. M. Sources of degradation of speech recognition in the telephone network. In Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International Conference on (1994), vol. 1, IEEE, pp. I--109.

[32]

Muscatell, R. P. Laser microphone, Oct. 23 1984. US Patent 4,479,265.

[33]

Nation, P., and Waring, R. Vocabulary size, text coverage and word lists. Vocabulary: Description, acquisition and pedagogy 14 (1997), 6--19.

[34]

Ogata, K. System dynamics, vol. 3. Prentice Hall New Jersey, 1998.

[35]

Qi, Y., and Hillman, R. E. Temporal and spectral estimations of harmonics-to-noise ratio in human voice signals. The Journal of the Acoustical Society of America 102, 1 (1997), 537--543.

[36]

Roy, N., Gowda, M., and Choudhury, R. R. Ripple: Communicating through physical vibration.

[37]

Scarborough, J. B. The Gyroscope: Theory and Application. Interscience Pub., 1958.

[38]

Smaragdis, P., Raj, B., and Shashanka, M. Missing data imputation for spectral audio signals. In Machine Learning for Signal Processing, 2009. MLSP 2009. IEEE International Workshop on (2009), IEEE, pp. 1--6.

[39]

Smeets, G. Laser interference microphone for ultrasonics and nonlinear acoustics. The Journal of the Acoustical Society of America 61, 3 (1977), 872--875.

[40]

Speciale, J. R. Pulsed laser microphone, Oct. 9 2001. US Patent 6,301,034.

[41]

Tanner, P., Loebach, J., Cook, J., and Hallen, H. A pulsed jumping ring apparatus for demonstration of lenz's law. American Journal of Physics 69, 8 (2001), 911--916.

[42]

Taylor, B. Guide for the Use of the International System of Units (SI): The Metric System. DIANE Publishing, 1995.

[43]

Titze, I. R. Principles of voice production. National Center for Voice and Speech, 2000.

[44]

Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. J. Phoneme recognition using time-delay neural networks. Acoustics, Speech and Signal Processing, IEEE Transactions on 37, 3 (1989), 328--339.

[45]

Wang, C.-C., Trivedi, S., Jin, F., Swaminathan, V., Rodriguez, P., and Prasad, N. S. High sensitivity pulsed laser vibrometer and its application as a laser microphone. Applied Physics Letters 94, 5 (2009), 051112.

[46]

Wei, T., Wang, S., Zhou, A., and Zhang, X. Acoustic eavesdropping through wireless vibrometry. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, pp. 130--141.

Digital Library

[47]

Zhang, L., Pathak, P. H., Wu, M., Zhao, Y., and Mohapatra, P. Accelword: Energy efficient hotword detection through accelerometer. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (2015), ACM, pp. 301--315.

Digital Library

Cited By

Chen TYang YQiu CFan XGuo XShangguan LOkoshi TKo JLiKamWa R(2024)Enabling Hands-Free Voice Assistant Activation on EarphonesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661890(155-168)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661890
Liao QHuang YHuang YWu K(2024)An Eavesdropping System Based on Magnetic Side-Channel Signals Leaked by SpeakersACM Transactions on Sensor Networks10.1145/363706320:2(1-30)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1145/3637063
Hu PLi WMa YSanthalingam PPathak PLi HZhang HZhang GCheng XMohapatra P(2024)Towards Unconstrained Vocabulary Eavesdropping With mmWave Radar Using GANIEEE Transactions on Mobile Computing10.1109/TMC.2022.322669023:1(941-954)Online publication date: Jan-2024
https://doi.org/10.1109/TMC.2022.3226690
Show More Cited By

Index Terms

Listening through a Vibration Motor
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

MFCC-GMM based accent recognition system for Telugu speech signals

Speech processing is very important research area where speaker recognition, speech synthesis, speech codec, speech noise reduction are some of the research areas. Many of the languages have different speaking styles called accents or dialects. ...
Experimental study on the effect of rubber isolator on the vibration of piston pump
ICCMA 2018: Proceedings of the 6th International Conference on Control, Mechatronics and Automation

Rubber isolator commonly used as support of piston pump, and the stiffness and damping nonlinearity of rubber isolator will significantly affect the vibration characteristics of piston pump. In this paper, two kinds of rubber isolators were selected, ...
sMFCC: exploiting sparseness in speech for fast acoustic feature extraction on mobile devices -- a feasibility study
HotMobile '13: Proceedings of the 14th Workshop on Mobile Computing Systems and Applications

Due to limited processing capability, contemporary smartphones cannot extract frequency domain acoustic features in real-time on the device when the sampling rate is high. We propose a solution to this problem which exploits the sparseness in speech to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '16: Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services

June 2016

440 pages

ISBN:9781450342698

DOI:10.1145/2906388

General Chairs:
Rajesh Balan
Singapore Management University
,
Archan Misra
Singapore Management University
,
Program Chairs:
Sharad Agarwal
Microsoft
,
Cecilia Mascolo
University of Cambridge

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
Huawei
Qualcomm
HP

Conference

MobiSys'16

Sponsor:

SIGMOBILE

MobiSys'16: The 14th Annual International Conference on Mobile Systems, Applications, and Services

June 26 - 30, 2016

Singapore, Singapore

Acceptance Rates

MobiSys '16 Paper Acceptance Rate 31 of 197 submissions, 16%;

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
665
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)4

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen TYang YQiu CFan XGuo XShangguan LOkoshi TKo JLiKamWa R(2024)Enabling Hands-Free Voice Assistant Activation on EarphonesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661890(155-168)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661890
Liao QHuang YHuang YWu K(2024)An Eavesdropping System Based on Magnetic Side-Channel Signals Leaked by SpeakersACM Transactions on Sensor Networks10.1145/363706320:2(1-30)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1145/3637063
Hu PLi WMa YSanthalingam PPathak PLi HZhang HZhang GCheng XMohapatra P(2024)Towards Unconstrained Vocabulary Eavesdropping With mmWave Radar Using GANIEEE Transactions on Mobile Computing10.1109/TMC.2022.322669023:1(941-954)Online publication date: Jan-2024
https://doi.org/10.1109/TMC.2022.3226690
Lin FWang CLiu TLiu ZShen YBa ZLu LXu WRen K(2024)High-Quality Speech Recovery Through Soundproof Protections via mmWave SensingIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.332229521:4(3065-3081)Online publication date: Jul-2024
https://doi.org/10.1109/TDSC.2023.3322295
Zhang GXiang ZFu HYang YHu P(2024)EchoLight: Sound Eavesdropping based on Ambient Light ReflectionIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621338(341-350)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621338
Xu XChen YLing ZLu LLuo JFu X(2024)mmEar: Push the Limit of COTS mmWave Eavesdropping on HeadphonesIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621229(351-360)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621229
Zhao RYu JZhao HNgai E(2023)Radio2TextProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108737:3(1-28)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610873
Shahid IRoy NHui PAmiri Sani ANurmi PLiu Y(2023)"Is this my president speaking?" Tamper-proofing Speech in Live RecordingsProceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services10.1145/3581791.3596862(219-232)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3581791.3596862
Garg NShahid IAvllazagaj EHill JHan JRoy N(2023)ThermWareProceedings of the 24th International Workshop on Mobile Computing Systems and Applications10.1145/3572864.3580339(81-88)Online publication date: 22-Feb-2023
https://dl.acm.org/doi/10.1145/3572864.3580339
Cai CPu HYe LJiang HLuo J(2023)Active Acoustic Sensing for “Hearing” Temperature Under Acoustic InterferenceIEEE Transactions on Mobile Computing10.1109/TMC.2021.309679222:2(661-673)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TMC.2021.3096792
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

ePub

View this article in ePub.

Media

Figures

Other

Tables

View Table of Contents