research-article

AccMyrinx: Speech Synthesis with Non-Acoustic Sensor

Authors:

Yanyong ZhangAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 6, Issue 3

Article No.: 127, Pages 1 - 24

https://doi.org/10.1145/3550338

Published: 07 September 2022 Publication History

Abstract

The built-in loudspeakers of mobile devices (e.g., smartphones, smartwatches, and tablets) play significant roles in human-machine interaction, such as playing music, making phone calls, and enabling voice-based interaction. Prior studies have pointed out that it is feasible to eavesdrop on the speaker via motion sensors, but whether it is possible to synthesize speech from non-acoustic signals with sub-Nyquist sampling frequency has not been studied. In this paper, we present an end-to-end model to reconstruct the acoustic waveforms that are playing on the loudspeaker through the vibration captured by the built-in accelerometer. Specifically, we present an end-to-end speech synthesis framework dubbed AccMyrinx to eavesdrop on the speaker using the built-in low-resolution accelerometer of mobile devices. AccMyrinx takes advantage of the coexistence of an accelerometer with the loudspeaker on the same motherboard and compromises the loudspeaker by the solid-borne vibrations captured by the accelerometer. Low-resolution vibration signals are fed to a wavelet-based MelGAN to generate intelligible acoustic waveforms. We conducted extensive experiments on a large-scale dataset created based on audio clips downloaded from Voice of America (VOA). The experimental results show that AccMyrinx is capable of reconstructing intelligible acoustic signals that are playing on the loudspeaker with a smoothed word error rate (SWER) of 42.67%. The quality of synthesized speeches could be severely affected by several factors including gender, speech rate, and volume.

References

[1]

S. A. Anand and N. Saxena. 2018. Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors. In 2018 IEEE Symposium on Security and Privacy (SP). 1000--1017. https://doi.org/10.1109/SP.2018.00004

[2]

S Abhishek Anand, Chen Wang, Jian Liu, Nitesh Saxena, and Yingying Chen. 2021. Spearphone: A Lightweight Speech Privacy Exploit via Accelerometer-Sensed Reverberations from Smartphone Loudspeakers. In Proceedings of the 14th ACM Conference on Security and Privacy in Wireless and Mobile Networks (Abu Dhabi, United Arab Emirates) (WiSec '21). Association for Computing Machinery, New York, NY, USA, 288--299. https://doi.org/10.1145/3448300.3468499

Digital Library

[3]

Sercan Ö. Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep Voice 2: Multi-Speaker Neural Text-to-Speech. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 2966--2974.

[4]

Zhongjie Ba, Tianhang Zheng, Zhan Qin, Hanlin Yu, Liu Liu, Baochun Li, Xue Liu, and Kui Ren. 2020. Accelerometer-Based Smartphone Eavesdropping. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCom '20). Association for Computing Machinery, New York, NY, USA, Article 73, 2 pages. https://doi.org/10.1145/3372224.3417323

Digital Library

[5]

Zhongjie Ba, Tianhang Zheng, Qinyu Zhang, Zhan Qin, Baochun Li, Xue Liu, and Kui Ren. 2020. Learning-based Practical Smartphone Eavesdropping with Built-in Accelerometer. In Proceedings of Network and Distributed Systems Security (NDSS) Symposium 2020 (San Diego, USA) (NDSS '20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3372224.3417323

Digital Library

[6]

Marimuthu C., K. Chandrasekaran, and Sridhar Chimalakonda. 2020. Energy Diagnosis of Android Applications: A Thematic Taxonomy and Survey. ACM Comput. Surv. 53, 6, Article 117 (Dec. 2020), 36 pages. https://doi.org/10.1145/3417986

Digital Library

[7]

Jian Cong, Shan Yang, Lei Xie, and Dan Su. 2021. Glow-WaveGAN: Learning Speech Representations from GAN-Based Variational Auto-Encoder for High Fidelity Flow-Based Speech Synthesis. In Proc. Interspeech 2021. 2182--2186. https://doi.org/10.21437/Interspeech.2021-414

[8]

Matthias Gazzari, Annemarie Mattmann, Max Maass, and Matthias Hollick. 2022. My(o) Armband Leaks Passwords: An EMG and IMU Based Keylogging Side-Channel Attack. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 157 (dec 2022), 24 pages. https://doi.org/10.1145/3494986

Digital Library

[9]

Jun Han, Albert Jin Chung, and Patrick Tague. 2017. PitchIn: Eavesdropping via Intelligible Speech Reconstruction Using Non-acoustic Sensor Fusion. In 2017 16th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 181--192.

[10]

Yan Huang, Lei He, Wenning Wei, William Gale, Jinyu Li, and Yifan Gong. 2020. Using Personalized Speech Synthesis and Neural Language Generator for Rapid Speaker Adaptation. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 7399--7403. https://doi.org/10.1109/ICASSP40776.2020.9053104

[11]

Emma Jokinen, Ulpu Remes, Paavo Alku, Emma Jokinen, Ulpu Remes, and Paavo Alku. 2017. Intelligibility Enhancement of Telephone Speech Using Gaussian Process Regression for Normal-to-Lombard Spectral Tilt Conversion. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 25, 10 (Oct. 2017), 1985--1996. https://doi.org/10.1109/TASLP.2017.2740004

Digital Library

[12]

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17022--17033. https://proceedings.neurips.cc/paper/2020/file/c5d736809766d46260d816d8dbc9eb44-Paper.pdf

[13]

R. Kubichek. 1993. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, Vol. 1. 125--128. https://doi.org/10.1109/PACRIM.1993.407206

[14]

Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. 2019. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/6804c9bca0a615bdb9374d00a9fcba59-Paper.pdf

[15]

M. C. Leong, W. Y. Lam, M. S. D. Zan, N. Arsad, A. A. A. Bakar, and M. H. Hafiz Mokhtar. 2020. Contactless Vibration Sensing and Acoustics Wave Detection Based on Laser Feedback Interferometer. In 2020 IEEE 8th International Conference on Photonics (ICP). 91--92. https://doi.org/10.1109/ICP46580.2020.9206494

[16]

Jian Liu, Chen Wang, Yingying Chen, and Nitesh Saxena. 2017. VibWrite: Towards Finger-Input Authentication on Ubiquitous Surfaces via Physical Vibration. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS '17). Association for Computing Machinery, New York, NY, USA, 73--87. https://doi.org/10.1145/3133956.3133964

Digital Library

[17]

Héctor A. Cordourier Maruri, Paulo Lopez-Meyer, Jonathan Huang, Willem Marco Beltman, Lama Nachman, and Hong Lu. 2018. V-Speech: Noise-Robust Speech Capturing Glasses Using Vibration Sensors. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 180 (dec 2018), 23 pages. https://doi.org/10.1145/3287058

Digital Library

[18]

Yan Michalevsky, Dan Boneh, and Gabi Nakibly. 2014. Gyrophone: Recognizing Speech from Gyroscope Signals. In Proceedings of the 23rd USENIX Conference on Security Symposium (San Diego, CA) (SEC'14). USENIX Association, USA, 1053--1067.

[19]

Frank Mokaya, Roland Lucas, Hae Young Noh, and Pei Zhang. 2015. MyoVibe: Vibration Based Wearable Muscle Activation Detection in High Mobility Exercises. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (UbiComp '15). Association for Computing Machinery, New York, NY, USA, 27--38. https://doi.org/10.1145/2750858.2804258

Digital Library

[20]

Jeanne Parson, Daniela Braga, Michael Tjalve, and Jieun Oh. 2013. Evaluating Voice Quality and Speech Synthesis Using Crowdsourcing. In Text, Speech, and Dialogue, Ivan Habernal and Václav Matoušek (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 233--240.

[21]

Vinod Veera Reddy, Andy W. H. Khong, and Boon Poh Ng. 2014. Unambiguous Speech DOA Estimation under Spatial Aliasing Conditions. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 22, 12 (Dec. 2014), 2133--2145. https://doi.org/10.1109/TASLP.2014.2344856

Digital Library

[22]

Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. 2011. CROWDMOS: An approach for crowdsourcing mean opinion score studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2416--2419. https://doi.org/10.1109/ICASSP.2011.5946971

[23]

A.-R. Sedighi, M.-R. Haghifam, O.P. Malik, and M.-H. Ghassemian. 2005. High impedance fault detection based on wavelet transform and statistical pattern recognition. IEEE Transactions on Power Delivery 20, 4 (2005), 2414--2421. https://doi.org/10.1109/TPWRD.2005.852367

[24]

Jonathan Shen, R. Pang, Ron J. Weiss, M. Schuster, Navdeep Jaitly, Z. Yang, Z. Chen, Yu Zhang, Yuxuan Wang, R. Skerry-Ryan, R. A. Saurous, Yannis Agiomyrgiannakis, and Y. Wu. 2018. Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2018), 4779--4783.

Digital Library

[25]

Weigao Su, Daibo Liu, Taiyuan Zhang, and Hongbo Jiang. 2022. Towards Device Independent Eavesdropping on Telephone Conversations with Built-in Accelerometer. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 177 (dec 2022), 29 pages. https://doi.org/10.1145/3494969

Digital Library

[26]

Yusuke Sugawara, Sayaka Shiota, and Hitoshi Kiya. 2019. Checkerboard artifacts free convolutional neural networks. APSIPA Transactions on Signal and Information Processing 8 (2019), e9. https://doi.org/10.1017/ATSIP.2019.2

[27]

Xiao Sun, Li Qiu, Yibo Wu, Yeming Tang, and Guohong Cao. 2017. SleepMonitor: Monitoring Respiratory Rate and Body Position During Sleep Using Smartwatch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 104 (sep 2017), 22 pages. https://doi.org/10.1145/3130969

Digital Library

[28]

Teng Wei, Shu Wang, Anfu Zhou, and Xinyu Zhang. 2015. Acoustic Eavesdropping through Wireless Vibrometry. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (Paris, France) (MobiCom '15). Association for Computing Machinery, New York, NY, USA, 130--141. https://doi.org/10.1145/2789168.2790119

Digital Library

[29]

Qingxin Xia, Joseph Korpela, Yasuo Namioka, and Takuya Maekawa. 2020. Robust Unsupervised Factory Activity Recognition with Body-Worn Accelerometer Using Temporal Structure of Multiple Sensor Data Motifs. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 97 (sep 2020), 30 pages. https://doi.org/10.1145/3411836

Digital Library

[30]

Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. 2021. Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech. In 2021 IEEE Spoken Language Technology Workshop (SLT). 492--498. https://doi.org/10.1109/SLT48900.2021.9383551

[31]

Jinhyeok Yang, Junmo Lee, Youngik Kim, Hoon-Young Cho, and Injung Kim. 2020. VocGAN: A High-Fidelity Real-Time Vocoder with a Hierarchically-Nested Adversarial Network. In Proc. Interspeech 2020. 200--204. https://doi.org/10.21437/Interspeech.2020-1238

[32]

Li Zhang, Parth H. Pathak, Muchen Wu, Yixin Zhao, and Prasant Mohapatra. 2015. AccelWord: Energy Efficient Hotword Detection through Accelerometer. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 301--315. https://doi.org/10.1145/2742647.2742658

Digital Library

Cited By

Gao MCui HXie YChen YYu ZGuo BYuan X(2023)Practical Earphone Eavesdropping with Built-in Motion Sensors2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00299(2219-2226)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00299

Index Terms

AccMyrinx: Speech Synthesis with Non-Acoustic Sensor
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
      1. Mobile devices
2. Security and privacy
  1. Security in hardware
    1. Embedded systems security

Recommendations

Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Dysarthria is a motor speech disorder that causes inability to control and coordinate one or more articulators. This makes it difficult for a dysarthric speaker to utter certain speech sound units, thereby producing poorly articulated, slurred, and ...
Automatic Speech Recognition Used for Intelligibility Assessment of Text-to-Speech Systems
Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction

Speech intelligibility is the most important parameter in evaluation of speech quality. In the contribution, a new objective intelligibility assessment of general speech processing algorithms is proposed. It is based on automatic recognition methods ...
The physiological microphone (PMIC): A competitive alternative for speaker assessment in stress detection and speaker verification

Interactive speech system scenarios exist which require the user to perform tasks which exert limitations on speech production, thereby causing speaker variability and reduced speech performance. In noisy stressful scenarios, even if noise could be ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 6, Issue 3

September 2022

1612 pages

EISSN:2474-9567

DOI:10.1145/3563014

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 September 2022

Published in IMWUT Volume 6, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Science Foundation of China
National Key Research and Development Program of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
284
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)3

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao MCui HXie YChen YYu ZGuo BYuan X(2023)Practical Earphone Eavesdropping with Built-in Motion Sensors2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00299(2219-2226)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00299

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents