research-article

EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking

Authors:

Xiang-Yang LiAuthors Info & Claims

SenSys '20: Proceedings of the 18th Conference on Embedded Networked Sensor Systems

Pages 95 - 108

https://doi.org/10.1145/3384419.3430730

Published: 16 November 2020 Publication History

Abstract

Acoustic motion tracking is an exciting new research area with promising progress in the last few years. Due to the inherent low propagation speed in the air, acoustic signals have the unique advantage of fine sensing granularity compared to RF signals. Speakers and microphones nowadays are pervasively available in devices surrounding us, such as smartphones and voice-controlled smart speakers. Though promising, one fundamental issue hindering the adoption of acoustic-based motion tracking is that the positions of microphones and speakers inside a device are fixed, which greatly limits the flexibility of acoustic motion tracking. In this work, we propose a new modality of acoustic motion tracking using earphones. Earphone-based tracking mitigates the constraints associated with traditional smartphone-based tracking. With novel designs and comprehensive experiments, we show earphone-based motion tracking can achieve a great flexibility and a high accuracy at the same time. We believe this is an important step towards "earable" sensing.

References

[1]

Fadel Adib, Zachary Kabelac, Dina Katabi, and Rob Miller. 2014. WiTrack: motion tracking via radio reflections off the body. In Proc. of NSDI.

Digital Library

[2]

Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank DeRuyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services. 15--28.

Digital Library

[3]

John R. Buck Alan V. Oppenheim, Ronald W. Schafer. 1989. Discrete-time signal processing. Vol. volume 2. Prentice-Hall, Inc.

[4]

Inc Amazon.com. 2019. Amazon Alexa. https://developer.amazon.com/en-US/alexa.

[5]

Inc Apple. 2019. AirPods. https://www.apple.com/shop/product/MRXJ2AM/A/airpods-with-wireless-charging-case.

[6]

Inc Arduino.com. 2019. Arduino. https://www.arduino.cc/.

[7]

Inc Audio-Technica. 2019. ATH-CKL220iS. https://audio-technica.com.au/products/ath-ckl220is/.

[8]

Inc Bela.com. 2019. Bela. https://www.blta.com.

[9]

Kongyang Chen and Guang Tan. 2018. BikeGPS: Accurate Localization of Shared Bikes in Street Canyons via Low-Level GPS Cooperation. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 150--162.

Digital Library

[10]

Zicheng Chi, Yao Yao, Tiantian Xie, Xin Liu, Zhichuan Huang, Wei Wang, and Ting Zhu. 2018. EAR: Exploiting uncontrollable ambient RF signals in heterogeneous networks for gesture recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 237--249.

Digital Library

[11]

Audio-Technica Corporation. 2019. ATH-CK350IS. http://sea.audio-technica.com/happenings/2018-05/audio-technica-unveils-ath-ck350is-new-in-ear-headphones-for-smartphones.

[12]

HUAWEI Corporation. 2019. Flypods Lite. https://www.hihonor.com/global/products/accessories/honor-flypods-lite/.

[13]

HTC Corporation. 2019. HTC VIVE. https://www.vive.com/us/product/vive-virtual-reality-system/.

[14]

HUAWEI Corporation. 2019. HUAWEI P20. https://consumer.huawei.com/en/support/phones/p20/.

[15]

Lenovo Corporation. 2019. ThinkPad P1 Gen2. https://www.lenovo.com/us/en/laptops/thinkpad/thinkpad-p/P1-Gen-2/p/22WS2WPP102.

[16]

Moshi Corporation. 2019. Mythro Earbuds. https://www.moshi.com/en/product/audio-earbuds-mythro-mic/gray.

[17]

Philips Corporation. 2019. Philips PRO6105BK. https://www.usa.philips.com/cp/PRO6105BK_00/6000-series-in-ear-headphones-with-mic.

[18]

SaLusTex Corporation. 2019. Exhibit at Healthcare Next Generation. https://salustek.com/en/25/.

[19]

Sony Corporation. 2019. Sony MDR-XB75AP. https://www.sony.com.my/electronics/in-ear-headphones/mdr-xb75ap.

[20]

Sony Corporation. 2019. WF-1000XM3. https://helpguide.sony.net/mdr/wf1000xm3/v1/en/contents/TP0002289865.html.

[21]

Samsung Corporation. 2020. Samsung Galaxy Buds+. https://www.samsung.com/us/mobile/audio/galaxy-buds-plus/.

[22]

Inc Google.com. 2019. Google Home. https://store.google.com/gb/product/google_home.

[23]

Apple Inc. 2019. AirPods Pro. https://support.apple.com/kb/SP811?viewlocale=en_US&locale=en_US.

[24]

AVTech Media Americas Inc. 2019. Headphone measures. https://www.innerfidelity.com/headphone-measurements.

[25]

Google Inc. 2019. Google Daydream. https://arvr.google.com/daydream/.

[26]

Quebec Inc. 2019. Headphones Frequency Response. https://www.thephonograph.net/headphones-frequency-response/.

[27]

Quebec Inc. 2019. Raw Frequency Response. https://www.rtings.com/headphones/tests/sound-quality/raw-frequency-response.

[28]

Sony Interactive Entertainment Inc. 2019. PlayStation VR. https://www.playstation.com/.

[29]

Pravein Govindan Kannan, Seshadri Padmanabha Venkatagiri, Mun Choon Chan, Akhihebbal L Ananda, and Li-Shiuan Peh. 2012. Low cost crowd counting using audio tones. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems. 155--168.

Digital Library

[30]

Samuel J. Ling, Jeff Sanny, and William Moebs. 2016. University Physics - Volume 2 (OpenStax). (2016).

[31]

Yang Liu, Wuxiong Zhang, Yang Yang, Weidong Fang, Fei Qin, and Xuewu Dai. 2019. PAMT: Phase-based Acoustic Motion Tracking in Multipath Fading Environments. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2386--2394.

[32]

Hiroyuki Manabe and Masaaki Fukumoto. 2012. Headphone taps: A simple technique to add input function to regular headphones. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion.

Digital Library

[33]

Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 69--81.

Digital Library

[34]

Robert A. Meyers. 1987. Encyclopedia of physical science and technology.

[35]

Chulhong Min, Akhil Mathur, and Fahim Kawsar. 2018. Exploring audio and kinetic sensing on earable devices. In Proceedings of the 4th ACM Workshop on Wearable Systems and Applications. ACM, 5--10.

Digital Library

[36]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. (2016).

[37]

Hiroki Ota, Minghan Chao, Yuji Gao, Eric Wu, Li-Chia Tai, Kevin Chen, Yasutomo Matsuoka, Kosuke Iwai, Hossain M Fahad, Wei Gao, et al. 2017. 3d printed "earable" smart devices for real-time detection of core body temperature. ACS sensors 2, 7 (2017), 990--997.

[38]

Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. 2007. Beepbeep: a high accuracy acoustic ranging system using cots mobile devices. In Proceedings of the 5th international conference on Embedded networked sensor systems. ACM, 1--14.

Digital Library

[39]

Alexander D. Poularikas. 1998. The handbook of formulas and tables for signal processing.

[40]

Swadhin Pradhan, Eugene Chai, Karthikeyan Sundaresan, Lili Qiu, Mohammad A Khojastepour, and Sampath Rangarajan. 2017. Rio: A pervasive rfid-based touch gesture interface. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. 261--274.

Digital Library

[41]

Nissanka B Priyantha, Anit Chakraborty, and Hari Balakrishnan. 2000. The cricket location-support system. In Proceedings of the 6th annual international conference on Mobile computing and networking. ACM, 32--43.

Digital Library

[42]

Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. 474--485.

Digital Library

[43]

Akane Sano, Takashi Tomita, and Haruo Oba. 2010. Applications using earphone with biosignal sensors. In Human Interface Society Meeting, Vol. 12. 1--6.

[44]

Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 429--444.

Digital Library

[45]

Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 591--605.

Digital Library

[46]

Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 77--89.

Digital Library

[47]

Kazuhiro Taniguchi, Hisashi Kondo, Toshiya Tanaka, and Atsushi Nishikawa. 2018. Earable RCC: development of an earphone-type reliable chewing-count measurement device. Journal of healthcare engineering 2018 (2018).

[48]

Deepak Vasisht, Swarun Kumar, and Dina Katabi. 2016. Decimeter-level localization with a single WiFi access point. In 13th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 16). 165--178.

[49]

Aditya Virmani and Muhammad Shahzad. 2017. Position and orientation agnostic gesture recognition using wifi. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. 252--264.

Digital Library

[50]

Anran Wang and Shyamnath Gollakota. 2019. MilliSonic: Pushing the Limits of Acoustic Motion Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 18.

Digital Library

[51]

Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1691--1699.

Digital Library

[52]

Hao Wang, Daqing Zhang, Junyi Ma, Yasha Wang, Yuxiang Wang, Dan Wu, Tao Gu, and Bing Xie. 2016. Human respiration detection with commodity wifi devices: do user location and body orientation matter?. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 25--36.

Digital Library

[53]

Jue Wang, Deepak Vasisht, and Dina Katabi. 2015. RF-IDraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Computer Communication Review 44, 4 (2015), 235--246.

Digital Library

[54]

Wang Wei, Alex X. Liu, and Sun Ke. 2016. Device-free gesture tracking using acoustic signals. In International Conference on Mobile Computing & Networking.

[55]

Ning Xiao, Panlong Yang, Xiang-Yang Li, Yanyong Zhang, Yubo Yan, and Hao Zhou. 2019. MilliBack: Real-Time Plug-n-Play Millimeter Level Tracking Using Wireless Backscattering. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 112.

Digital Library

[56]

Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13). 71--84.

Digital Library

[57]

Jie Yang, Simon Sidhom, Gayathri Chandrasekaran, Tam Vu, Hongbo Liu, Nicolae Cecan, Yingying Chen, Marco Gruteser, and Richard P Martin. 2011. Detecting driver phone use leveraging car speakers. In Proceedings of the 17th annual international conference on Mobile computing and networking. ACM, 97--108.

Digital Library

[58]

Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 237--248.

Digital Library

[59]

Shichao Yue, Hao He, Hao Wang, Hariharan Rahul, and Dina Katabi. 2018. Extracting multi-person respiration from entangled RF signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 1--22.

Digital Library

[60]

Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--29.

Digital Library

[61]

Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--28.

Digital Library

[62]

Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. 2015. Extending mobile interaction through near-field visible light sensing. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 345--357.

Digital Library

[63]

Yunting Zhang, Jiliang Wang, Weiyi Wang, Zhao Wang, and Yunhao Liu. 2018. Vernier: Accurate and Fast Acoustic Motion Tracking Using Mobile Devices. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1709--1717.

[64]

Zengbin Zhang, David Chu, Xiaomeng Chen, and Thomas Moscibroda. 2012. Swordfight: Enabling a new class of phone-to-phone action games on commodity phones. In Proceedings of the 10th international conference on Mobile systems, applications, and services. ACM, 1--14.

Digital Library

Cited By

Hu CMa XHuang XShen YMa D(2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699793
Yu THu GZhang RLim HMahmud SLee CLi KAgarwal DNie SOh JGuimbretière FZhang C(2024)Ring-a-Pose: A Ring for Continuous Hand Pose TrackingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997418:4(1-30)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699741
Yuan KIbrahim MSong YDeng GNerone RVijayan SGadre AKumar S(2024)ToMoBrush: Exploring Dental Health Sensing Using a Sonic ToothbrushProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785058:3(1-27)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678505
Show More Cited By

Index Terms

EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Sound-based input / output

Recommendations

MilliSonic: Pushing the Limits of Acoustic Motion Tracking
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Recent years have seen interest in device tracking and localization using acoustic signals. State-of-the-art acoustic motion tracking systems however do not achieve millimeter accuracy and require large separation between microphones and speakers, and ...
Motion Tracking with an Active Camera

This paper describes a method for real-time motion detection using an active camera mounted on a pan/tilt platform. Image mapping is used to align images of different viewpoints so that static camera motion detection can be applied. In the presence of ...
Human Motion Reconstruction Based on Multi-view Video
IHMSC '13: Proceedings of the 2013 5th International Conference on Intelligent Human-Machine Systems and Cybernetics - Volume 02

In this paper, the key technologies of human motion reconstruction Based on multi-view video are discussed. A more robust motion reconstruction method with markers is presented, which includes camera calibration, markers extraction, markers tracking and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SenSys '20: Proceedings of the 18th Conference on Embedded Networked Sensor Systems

November 2020

852 pages

ISBN:9781450375900

DOI:10.1145/3384419

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key R&D Program of China
China National Funds for Distinguished Young Scientists
China National Natural Science Foundation
Key Research Program of Frontier Sciences

Conference

SenSys '20

Sponsor:

SenSys '20: The 18th ACM Conference on Embedded Networked Sensor Systems

November 16 - 19, 2020

Virtual Event, Japan

Acceptance Rates

Overall Acceptance Rate 198 of 990 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,284
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)10

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hu CMa XHuang XShen YMa D(2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699793
Yu THu GZhang RLim HMahmud SLee CLi KAgarwal DNie SOh JGuimbretière FZhang C(2024)Ring-a-Pose: A Ring for Continuous Hand Pose TrackingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997418:4(1-30)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699741
Yuan KIbrahim MSong YDeng GNerone RVijayan SGadre AKumar S(2024)ToMoBrush: Exploring Dental Health Sensing Using a Sonic ToothbrushProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785058:3(1-27)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678505
Wang SWang XJiang WMiao CCao QWang HSun KXue HSu L(2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3677122
Li ZLuo WZhang YChen JShu YZhang YShu YLiu JTan RHe YChen J(2024)ASLiquid: Non-Intrusive Liquid Counterfeit Identification with Your EarphonesProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699321(41-53)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699321
Zhang JWang J(2024)FusionTrack: Towards Accurate Device-free Acoustic Motion Tracking with Signal FusionACM Transactions on Sensor Networks10.1145/365466620:3(1-30)Online publication date: 30-Mar-2024
https://dl.acm.org/doi/10.1145/3654666
Cao YDhekne AAmmar MOkoshi TKo JLiKamWa R(2024)UTrack3D: 3D Tracking Using Ultra-wideband (UWB) RadiosProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661881(345-358)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661881
Wu YChen SMeng XTong XLiu XXie XQu WOkoshi TKo JLiKamWa R(2024)Enabling 6D Pose Tracking on Your Acoustic DevicesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661875(15-28)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661875
Duan DSun ZNi TLi SJia XXu WLi TOkoshi TKo JLiKamWa R(2024)F2Key: Dynamically Converting Your Face into a Private Key Based on COTS Headphones for Reliable Voice InteractionProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661860(127-140)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661860
Zhou HYuan KGowda MQiu LXiong JGanesan DLane NShi W(2024)Rethinking Orientation Estimation with Smartphone-equipped Ultra-wideband ChipsProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690677(1045-1059)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1145/3636534.3690677
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten