research-article

RF-Mic: Live Voice Eavesdropping via Capturing Subtle Facial Speech Dynamics Leveraging RFID

Authors:

Yi-Chao ChenAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 7, Issue 2

Article No.: 49, Pages 1 - 25

https://doi.org/10.1145/3596259

Published: 12 June 2023 Publication History

Abstract

Eavesdropping on human voice is one of the most common but harmful threats to personal privacy. Glasses are in direct contact with human face, which could sense facial motions when users speak, so human speech contents could be inferred by sensing the movements of glasses. In this paper, we present a live voice eavesdropping method, RF-Mic, which utilizes common glasses attached with a low-cost RFID tag to sense subtle facial speech dynamics for inferring possible voice contents. When a user with a glasses, which is attached an RFID tag on the glass bridge, is speaking, RF-Mic first collects RF signals through forward propagation and backscattering. Then, body motion interference is eliminated from the collected RF signals through a proposed Conditional Denoising AutoEncoder (CDAE) network. Next, RF-Mic extracts three kinds of facial speech dynamic features (i.e., facial movements, bone-borne vibrations, and airborne vibrations) by designing three different deep-learning models. Based on the extracted features, a facial speech dynamics model is constructed for live voice eavesdropping. Extensive experiments in different real environments demonstrate that RF-Mic can achieve robust and accurate human live voice eavesdropping.

References

[1]

S. Abhishek Anand and Nitesh Saxena. 2018. Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors. In Proc. IEEE Symposium on Security and Privacy. San Francisco, USA, 1000--1017.

[2]

Zhongjie Ba, Tianhang Zheng, Xinyu Zhang, Zhan Qin, Baochun Li, Xue Liu, and Kui Ren. 2020. Learning-based Practical Smartphone Eavesdropping with Built-in Accelerometer. In proc. NDSS. San Diego, USA, 23--26.

[3]

C. BYU. 2020. Word frequency: based on 450 million word coca corpus. [Online]. Available: https://www.wordfrequency.info/.

[4]

Zhe Chen, Tianyue Zheng, Chao Cai, and Jun Luo. 2021. MoVi-Fi: motion-robust vital signs waveform recovery via deep interpreted RF sensing. In Proc. ACM Mobicom. New Orleans, USA, 392--405.

Digital Library

[5]

M Dobhn Daniel et al. 2008. The rf in rfid passive uhf rfid in practice. In Elsevier.

[6]

Abe Davis, Michae Rubinstein, Nea Wadhwa, Gautham J. Mysore, Fredo Durand, and William T. Freeman. 2014. The visual microphone: Passive recovery of sound from video. Acm Transactions on Graphics 33 (2014), 79--88.

Digital Library

[7]

Han Ding, Longfei Shangguan, Zheng Yang, Jinsong Han, Zimu Zhou, Panlong Yang, Wei Xi, and Jizhong Zhao. 2015. FEMO: A Platform for Free-weight Exercise Monitoring with RFIDs. In Proc SenSys. Seoul, South Korea, 141--154.

Digital Library

[8]

Pierre Divenyi, Steven Greenberg, and Georg Meyer. 2006. Dynamics of speech production and perception. Vol. 374. Ios Press.

[9]

Chao Feng, Jie Xiong, Liqiong Chang, Fuwei Wang, Ju Wang, and Dingyi Fang. 2021. RF-Identity: Non-Intrusive Person Identification Based on Commodity RFID Devices. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (2021), 1--23.

Digital Library

[10]

Yuanhao Feng, Panlong Yang, Yanyong Zhang, Xiang-Yang Li, Ziyang Chen, and Gang Huang. 2019. Demo: The RFID Can Hear Your Music Play. In Proc. MobiCom. Los Cabos, Mexico, 21--25.

Digital Library

[11]

Google. 2023. Google Assistant, your own personal Google. [Online]. Available: https://assistant.google.com/.

[12]

Pengfei Hu, Wenhao Li, Yifan Ma, Panneer Selvam Santhalingam, Parth Pathak, Hong Li, Huanle Zhang, Guoming Zhang, Xiuzhen Cheng, and Prasant Mohapatra. 2022. Towards Unconstrained Vocabulary Eavesdropping With Mmwave Radar Using GAN. IEEE Transactions on Mobile Computing 01 (2022), 1--14.

[13]

Pengfei Hu, Yifan Ma, Panneer Selvam Santhalingam, Parth H Pathak, and Xiuzhen Cheng. 2022. Milliear: Millimeter-wave acoustic eavesdropping with unconstrained vocabulary. In Proc. INFOCOM. Virtual Conference, 11--20.

Digital Library

[14]

Pengfei Hu, Hui Zhuang, Panneer Selvam Santhalingam, Riccardo Spolaor, Parth Pathak, Guoming Zhang, and Xiuzhen Cheng. 2022. AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary. In Proc. IEEE Symposium on Security and Privacy (SP). San Francisco, CA, USA, 1530--1530.

[15]

iflytek. 2022. iFlytek Input. [Online]. Available: https://srf.xunfei.cn/.

[16]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

[17]

Martin G Larson. 2006. Descriptive statistics and graphical displays. Circulation 114, 1 (2006), 76--81.

[18]

Mike Lenehan. 2021. Impinj, Inc. Application Note -- Low Level User Data Support. [Online]. Available: https://support.impinj.com/hc/en-us/articles/202755318-Application-Note-Low-Level-User-Data-Support.

[19]

Ping Li, Zhenlin An, Lei Yang, and Panlong Yang. 2019. Towards Physical-Layer Vibration Sensing with RFIDs. In Proc. INFOCOM. Paris, France, 892--900.

Digital Library

[20]

Ping Li, Zhenlin An, Lei Yang, Panlong Yang, and QiongZheng Lin. 2019. RFID harmonic for vibration sensing. IEEE Transactions on Mobile Computing 20, 4 (2019), 1614--1626.

[21]

Héctor A. Cordourier Maruri, Paulo Lopez-Meyer, Jonathan Huang, Willem Marco Beltman, Lama Nachman, and Hong Lu. 2018. V-Speech: Noise-Robust Speech Capturing Glasses Using Vibration Sensors. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4 (2018), 180:1--180:23.

[22]

Yan Michalevsky, Dan Boneh, and Gabi Nakibly. 2014. Gyrophone: Recognizing Speech from Gyroscope Signals. In Proc. USENIX. San Diego, CA,USA, 1053--1067.

[23]

F. Mavromatis N. Kargas and A. Bletsas. 2019. USRP reader. [Online]. Available: https://github.com/nkargas/Gen2-UHF-RFID-Reader.

[24]

Ben Nassi, Yaron Pirutin, Adi Shamir, Yuval Elovici, and Boris Zadov. 2020. Lamphone: Real-Time Passive Sound Recovery from Light Bulb Vibrations. Cryptology ePrint Archive, Paper 2020/708.

[25]

Louis C.W. Pols. 2011. SPEECH DYNAMICS. In Plenary Lecture.

[26]

Richard Raspet, Jeremy Webster, and Kevin Dillion. 2006. Framework for wind noise studies. The Journal of the Acoustical Society of America 119, 2 (2006), 834--843.

[27]

rfidhy. 2022. The Smallest RFID Tag as Thin as Sand. [Online]. Available: https://www.rfidhy.com/the-smallest-rfid-tag-as-thin-as-sand/.

[28]

Sriram Sami, Yimin Dai, Sean Rui Xiang Tan, Nirupam Roy, and Jun Han. 2020. Spying with Your Robot Vacuum Cleaner: Eavesdropping via Lidar Sensors. In Proc. SenSys. Yokohama, Japan, 354--367.

Digital Library

[29]

Baoguang Shi, Xiang Bai, and Cong Yao. 2016. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence 39, 11 (2016), 2298--2304.

Digital Library

[30]

Cong Shi, Xiangyu Xu, Tianfang Zhang, Payton Walker, Yi Wu, Jian Liu, Nitesh Saxena, Yingying Chen, and Jiadi Yu. 2021. Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In Proc. MobiCom. New Orleans, United States, 478--490.

Digital Library

[31]

Weigao Su, Daibo Liu, Taiyuan Zhang, and Hongbo Jiang. 2021. Towards Device Independent Eavesdropping on Telephone Conversations with Built-in Accelerometer. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4 (2021), 177:1--177:29.

Digital Library

[32]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).

[33]

Chuyu Wang and Lei Xie. 2018. Rf-ecg: Heart rate variability assessment based on cots rfid tag array. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 2 (2018), 1--26.

Digital Library

[34]

Chuyu Wang, Lei Xie, Yuancan Lin, Wei Wang, and Yingying Chen et al. 2021. Thru-the-wall Eavesdropping on Loudspeakers via RFID by Capturing Sub-mm Level Vibration. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4 (2021), 182:1--182:25.

[35]

DeLiang Wang. 2005. On ideal binary mask as the computational goal of auditory scene analysis. In Speech separation by humans and machines. Springer, 181--197.

[36]

Guanhua Wang, Yongpan Zou, Zimu Zhou, Kaishun Wu, and Lionel M Ni. 2016. We can hear you with Wi-Fi! IEEE Transactions on Mobile Computing 15, 11 (2016), 2907--2920.

[37]

Zi Wang, Yili Ren, Yingying Chen, and Jie Yang. 2022. Toothsonic: Earable authentication via acoustic toothprint. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--24.

Digital Library

[38]

Teng Wei, Shu Wang, Anfu Zhou, and Xinyu Zhang. 2015. Acoustic Eavesdropping through Wireless Vibrometry. In Proc. MobiCom. Paris, France, 130--141.

Digital Library

[39]

Zhichen Wu, Jianda Li, Jiadi Yu, Yanmin Zhu, Guangtao Xue, and Minglu Li. 2016. L3: Sensing driving conditions for vehicle lane-level localization on highways. In Proc. IEEE INFOCOM. San Francisco, CA, USA, 1--9.

Digital Library

[40]

Fu Xiao, Zhongqin Wang, Ning Ye, Ruchuan Wang, and Xiang-Yang Li. 2017. One more tag enables fine-grained RFID localization and tracking. IEEE/ACM Transactions on Networking 26, 1 (2017), 161--174.

Digital Library

[41]

Binbin Xie, Jie Xiong, Xiaojiang Chen, and Dingyi Fang. 2020. Exploring commodity rfid for contactless sub-millimeter vibration sensing. In Proc. ACM Sensys. Yokohama, Japan, 15--27.

Digital Library

[42]

Chenhan Xu, Zhengxiong Li, Hanbin Zhang, Aditya Singh Rathore, Huining Li, Chen Song, Kun Wang, and Wenyao Xu. 2019. WaveEar: Exploring a mmWave-based Noise-resistant Speech Sensing for Voice-User Interface. In Proc. MobiSys. Seoul, Korea, 14--26.

Digital Library

[43]

Xiangyu Xu, Jiadi Yu, Yingying Chen, Yanmin Zhu, Shiyou Qian, and Minglu Li. 2017. Leveraging audio signals for early recognition of inattentive driving with smartphones. IEEE Transactions on Mobile Computing 17, 7 (2017), 1553--1567.

[44]

Lei Yang, Yao Li, Qiongzheng Lin, Huanyu Jia, Xiang-Yang Li, and Yunhao Liu. 2017. Tagbeat: Sensing mechanical vibration period with cots rfid systems. IEEE/ACM transactions on networking 25, 6 (2017), 3823--3835.

[45]

Lei Yang, Yao Li, Qiongzheng Lin, Xiang-Yang Li, and Yunhao Liu. 2016. Making sense of mechanical vibration period with sub-millisecond accuracy using backscatter signals. In Proc. MobiCom. New York City, NY, USA, 16--28.

Digital Library

[46]

Panlong Yang, Yuanhao Feng, Jie Xiong, Ziyang Chen, and Xiang-Yang Li. 2020. RF-Ear: Contactless Multi-device Vibration Sensing and Identification Using COTS RFID. In Proc. INFOCOM. Toronto, ON, Canada, 297--306.

Digital Library

[47]

Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Sumeet Jain, Yiming Pu, Sinan Hersek, Kent Lyons, Kenneth A Cunefare, Omer T Inan, and Gregory D Abowd. 2017. Soundtrak: Continuous 3d tracking of a finger using active acoustics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1--25.

Digital Library

[48]

Li Zhang, Parth H. Pathak, Muchen Wu, Yixin Zhao, and Prasant Mohapatra. 2015. AccelWord: Energy Efficient Hotword Detection through Accelerometer. In Proc. MobiSys. Florence, Italy, 301--315.

Digital Library

[49]

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, and Michael Pecht. 2019. Deep residual shrinkage networks for fault diagnosis. IEEE Transactions on Industrial Informatics 16, 7 (2019), 4681--4690.

[50]

Yanmin Zhu, Ruobing Jiang, Jiadi Yu, Zhi Li, and Minglu Li. 2014. Geographic routing based on predictive locations in vehicular ad hoc networks. EURASIP Journal on Wireless Communications and Networking 2014 (2014), 1--9.

Cited By

Hu PQian YZheng TLi AChen ZGao YCheng XLuo J(2025)t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous DrivingIEEE Transactions on Mobile Computing10.1109/TMC.2024.346243724:1(135-149)Online publication date: Jan-2025
https://doi.org/10.1109/TMC.2024.3462437
Chen YYu JKong LZhu Y(2025)A Comprehensive Survey of Side-Channel Sound-Sensing MethodsIEEE Internet of Things Journal10.1109/JIOT.2024.350133412:2(1554-1578)Online publication date: 15-Jan-2025
https://doi.org/10.1109/JIOT.2024.3501334
Abdelnasser HHeggo MPang OKovac MMcCann J(2024)RaDro: Indoor Drone Tracking Using Millimeter Wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785498:3(1-23)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678549
Show More Cited By

Index Terms

RF-Mic: Live Voice Eavesdropping via Capturing Subtle Facial Speech Dynamics Leveraging RFID
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Security and privacy
  1. Network security
    1. Mobile and wireless security

Recommendations

Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors
MobiCom '21: Proceedings of the 27th Annual International Conference on Mobile Computing and Networking

Augmented reality/virtual reality (AR/VR) has extended beyond 3D immersive gaming to a broader array of applications, such as shopping, tourism, education. And recently there has been a large shift from handheld-controller dominated interactions to ...
A new unpredictability-based radio frequency identification forward privacy model and a provably secure construction

The privacy model of radio frequency identification RFID systems is for formalizing the adversarial capabilities and the security requirements of RFID anonymity and untraceability. Existing unpredictability-based privacy models such as unp-privacy, eunp-...
Combining appearance and motion for face and gender recognition from videos

While many works consider moving faces only as collections of frames and apply still image-based methods, recent developments indicate that excellent results can be obtained using texture-based spatiotemporal representations for describing and analyzing ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 7, Issue 2

June 2023

969 pages

EISSN:2474-9567

DOI:10.1145/3604631

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Published in IMWUT Volume 7, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
531
Total Downloads

Downloads (Last 12 months)209
Downloads (Last 6 weeks)24

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hu PQian YZheng TLi AChen ZGao YCheng XLuo J(2025)t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous DrivingIEEE Transactions on Mobile Computing10.1109/TMC.2024.346243724:1(135-149)Online publication date: Jan-2025
https://doi.org/10.1109/TMC.2024.3462437
Chen YYu JKong LZhu Y(2025)A Comprehensive Survey of Side-Channel Sound-Sensing MethodsIEEE Internet of Things Journal10.1109/JIOT.2024.350133412:2(1554-1578)Online publication date: 15-Jan-2025
https://doi.org/10.1109/JIOT.2024.3501334
Abdelnasser HHeggo MPang OKovac MMcCann J(2024)RaDro: Indoor Drone Tracking Using Millimeter Wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785498:3(1-23)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678549
Pettys-Baker RClarke MHolschuh BKostakos VKay JHoang T(2024)Functional Now, Wearable Later: Examining the Design Practices of Wearable TechnologistsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676615(71-81)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676615
Bañuelos JSigg SHe JSalim FCosta-Requena J(2024)Generating Multivariate Synthetic Time Series Data for Absent Sensors from Correlated SourcesProceedings of the 2nd International Workshop on Networked AI Systems10.1145/3662004.3663553(19-24)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3662004.3663553
Su YHuang SLiu HChen YDu YWang YRen YChen Y(2024)PPG-Hear: A Practical Eavesdropping Attack with Photoplethysmography SensorsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596038:2(1-28)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659603
Ma JZhang FJin BSu CLi SWang ZNi J(2024)Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596028:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659602
Chen YYu JChen YKong LZhu YChen YOkoshi TKo JLiKamWa R(2024)RFSpy: Eavesdropping on Online Conversations with Out-of-Vocabulary Words by Sensing Metal Coil Vibration of Headsets Leveraging RFIDProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661887(169-182)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661887
Chang ZZhang FXiong JChen WZhang DGanesan DLane NShi W(2024)MSense: Boosting Wireless Sensing Capability Under Motion InterferenceProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649350(108-123)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3636534.3649350
Zhu DHan CGuo JSun L(2024)TWLip: Exploring Through-Wall Word-Level Lip Reading Based on Coherent SISO RadarIEEE Internet of Things Journal10.1109/JIOT.2024.342732911:19(32310-32323)Online publication date: 1-Oct-2024
https://doi.org/10.1109/JIOT.2024.3427329
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents