Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3372224.3419213acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article
Public Access

Ear-AR: indoor acoustic augmented reality on earphones

Published: 18 September 2020 Publication History

Abstract

This paper aims to use modern earphones as a platform for acoustic augmented reality (AAR). We intend to play 3D audio-annotations in the user's ears as she moves and looks at AAR objects in the environment. While companies like Bose and Microsoft are beginning to release such capabilities, they are intended for outdoor environments. Our system aims to explore the challenges indoors, without requiring any infrastructure deployment. Our core idea is two-fold. (1) We jointly use the inertial sensors (IMUs) in earphones and smartphones to estimate a user's indoor location and gazing orientation. (2) We play 3D sounds in the earphones and exploit the human's responses to (re)calibrate errors in location and orientation. We believe this fusion of IMU and acoustics is novel, and could be an important step towards indoor AAR. Our system, Ear-AR, is tested on 7 volunteers invited to an AAR exhibition - like a museum - that we set up in our building's lobby and lab. Across 60 different test sessions, the volunteers browsed different subsets of 24 annotated objects as they walked around. Results show that Ear-AR plays the correct audio-annotations with good accuracy. The user-feedback is encouraging and points to further areas of research and applications.

References

[1]
2019. 3Dio: Professional Binaural Microphones. Retrieved Sept 18, 2019 from https://3diosound.com/
[2]
2019. Beats Solo3 Wireless - Beats by Dre. Retrieved Sep 13, 2019 from https://www.beatsbydre.com/uk/headphones/solo3-wireless
[3]
2019. Bragi Earbuds. Retrieved Sept 18, 2019 from https://bragi.com/
[4]
2019. Jabra Wireless Workout Headphones. Retrieved Sept 18, 2019 from https://www.jabra.com/sports-headphones/jabra-sport-coach-wireless#/
[5]
2019. Listen HRTF Database. Retrieved Sep 19, 2019 from http://recherche.ircam.fr/equipes/salles/listen/
[6]
2019. Samsung Gear IconX. Retrieved Sept 18, 2019 from https://www.samsung.com/global/galaxy/gear-iconx/
[7]
2019. Sennheiser's short film shows the power of binaural audio. Retrieved Sept 18, 2019 from https://thenextweb.com/plugged/2018/09/17/sennheisers-short-film-shows-the-power-of-binaural-audio/
[8]
2020. Airpods Pro. Retrieved Jul 21, 2020 from https://www.apple.com/airpods-pro/
[9]
2020. Apple spatial audio: what is it? How do you get it? Retrieved Jul 21, 2020 from https://www.whathifi.com/advice/what-is-apple-spatial-audio
[10]
Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 14). 317--329.
[11]
Muhammad Haris Afzal, Valérie Renaudin, and Gérard Lachapelle. 2010. Assessment of indoor magnetic field anomalies using multiple magnetometers. In ION GNSS, Vol. 10. 21--24.
[12]
Hyun-Sung An, Gregory C Jones, Seoung-Ki Kang, Gregory J Welk, and Jung-Min Lee. 2017. How valid are wearable physical activity trackers for measuring steps? European journal of sport science 17, 3 (2017), 360--368.
[13]
Apple. 2020. Apple Airpods. https://www.apple.com/airpods/ [Online; accessed 20-March-2020].
[14]
C Ascher, C Kessler, M Wankerl, and GF Trommer. 2010. Dual IMU indoor navigation with particle filter based map-matching on a smartphone. In 2010 International Conference on Indoor Positioning and Indoor Navigation. IEEE, 1--5.
[15]
Haitao Bao and Wai-Choong Wong. 2013. Improved PCA based step direction estimation for dead-reckoning localization. In 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. IEEE, 325--331.
[16]
Stéphane Beauregard. 2006. A helmet-mounted pedestrian dead reckoning system. In 3rd International Forum on Applied Wearable Computing 2006. VDE, 1--11.
[17]
Durand R Begault and Leonard J Trejo. 2000. 3-D sound for virtual reality and multimedia. (2000).
[18]
Alastair R Beresford and Frank Stajano. 2003. Location privacy in pervasive computing. IEEE Pervasive computing 1 (2003), 46--55.
[19]
Johann Borenstein, Lauro Ojeda, and Surat Kwanmuang. 2009. Heuristic reduction of gyro drift in IMU-based personnel tracking systems. In Optics and Photonics in Global Homeland Security V and Biometric Technology for Human Identification VI, Vol. 7306. International Society for Optics and Photonics, 73061H.
[20]
BOSE. 2020. Bose Developer Portal | NaviGuide Around Your World: Heads Up, Hands Free. https://developer.bose.com/bose-ar/get-inspired/naviguide-around-your-world-heads-hands-free [Online; accessed 16-March-2020].
[21]
BOSE. 2020. Bose Developer Portal | Tap Into Otocast, Your Friendly AR-enabled Tour Guide. https://developer.bose.com/bose-ar/get-inspired/tap-otocast-your-friendly-ar-enabled-tour-guide [Online; accessed 17-March-2020].
[22]
C Phillip Brown and Richard O Duda. 1998. A structural model for binaural sound synthesis. IEEE transactions on speech and audio processing 6, 5 (1998), 476--488.
[23]
Christophe Combettes and Valerie Renaudin. 2017. Walking direction estimation based on statistical modeling of human gait features with handheld MIMU. IEEE/ASME Transactions on Mechatronics 22, 6 (2017), 2502--2511.
[24]
Alejandro Correa, Estefania Munoz Diaz, Dina Bousdar Ahmed, Antoni Morell, and Jose Lopez Vicario. 2016. Advanced pedestrian positioning system to smartphones and smartwatches. Sensors 16, 11 (2016), 1903.
[25]
Daniel de Godoy, Bashima Islam, Stephen Xia, Md Tamzeed Islam, Rishikanth Chandrasekaran, Yen-Chun Chen, Shahriar Nirjon, Peter R Kinget, and Xiaofan Jiang. 2018. Paws: A wearable acoustic system for pedestrian safety. In 2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 237--248.
[26]
Zhi-An Deng, Guofeng Wang, Ying Hu, and Di Wu. 2015. Heading estimation for indoor pedestrian navigation using a smartphone in the pocket. Sensors 15, 9 (2015), 21518--21536.
[27]
Richard O Duda. 1993. Modeling head related transfer functions. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers. IEEE, 996--1000.
[28]
Google. 2020. Pixel Buds. https://store.google.com/product/pixel_buds [Online; accessed 20-March-2020].
[29]
Dorte Hammershøi and Henrik Møller. 2002. Methods for binaural recording and reproduction. Acta Acustica united with Acustica 88, 3 (2002), 303--311.
[30]
Fabian Höflinger, Rui Zhang, and Leonhard M Reindl. 2012. Indoor-localization system using a micro-inertial measurement unit (imu). In 2012 European Frequency and Time Forum. IEEE, 443--447.
[31]
JC Hung, JR Thacher, and HV White. 1989. Calibration of accelerometer triad of an IMU with drifting Z-accelerometer bias. In Proceedings of the IEEE National Aerospace and Electronics Conference. IEEE, 153--158.
[32]
Tong-Hun Hwang, Julia Reh, Alfred Effenberg, and Holger Blume. 2016. Real-time gait event detection using a single head-worn inertial measurement unit. In 2016 IEEE 6th International Conference on Consumer Electronics-Berlin (ICCE-Berlin). IEEE, 28--32.
[33]
Tong-Hun Hwang, Julia Reh, Alfred O Effenberg, and Holger Blume. 2018. Realtime gait analysis using a single head-worn inertial measurement unit. IEEE Transactions on Consumer Electronics 64, 2 (2018), 240--248.
[34]
Ian Dickson. 2020. Bose and HERE - creating the future of AR. https://360.here.com/bose-and-here-creating-the-future-of-ar [Online; accessed 24-March-2020].
[35]
Ian Dickson. 2020. Bose and HERE - creating the future of AR. https://developer.bose.com/content/here-maps-and-bose-ar-integration [Online; accessed 24-March-2020].
[36]
Lloyd A Jeffress. 1948. A place theory of sound localization. Journal of comparative and physiological psychology 41, 1 (1948), 35.
[37]
Antonio R Jimenez, Fernando Seco, Carlos Prieto, and Jorge Guevara. 2009. A comparison of pedestrian dead-reckoning algorithms using a low-cost MEMS IMU. In 2009 IEEE International Symposium on Intelligent Signal Processing. IEEE, 37--42.
[38]
Antonio Ramón Jiménez, Fernando Seco, José Carlos Prieto, and Jorge Guevara. 2010. Indoor pedestrian navigation using an INS/EKF framework for yaw drift reduction and a foot-mounted IMU. In 2010 7th Workshop on Positioning, Navigation and Communication. IEEE, 135--143.
[39]
Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, and Jason Hong. 2018. Rf-wear: Towards wearable everyday skeleton tracking using passive rfids. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 369--372.
[40]
Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, and Jason Hong. 2018. Wish: Towards a wireless shape-aware world using passive rfids. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 428--441.
[41]
Haojian Jin, Zhijian Yang, Swarun Kumar, and Jason I Hong. 2018. Towards wearable everyday body-frame tracking using passive rfids. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--23.
[42]
Wonho Kang and Youngnam Han. 2014. SmartPDR: Smartphone-based pedestrian dead reckoning for indoor localization. IEEE Sensors journal 15, 5 (2014), 2906--2916.
[43]
Kleomenis Katevas, Hamed Haddadi, and Laurissa Tokarchuk. 2016. Sensingkit: Evaluating the sensor power consumption in ios devices. In 2016 12th International Conference on Intelligent Environments (IE). IEEE, 222--225.
[44]
Maan Khedr and Nasser El-Sheimy. 2017. A smartphone step counter using IMU and magnetometer for navigation and health monitoring applications. Sensors 17, 11 (2017), 2573.
[45]
Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In ACM SIGCOMM computer communication review, Vol. 45. ACM, 269--282.
[46]
Zeqi Lai, Y Charlie Hu, Yong Cui, Linhui Sun, Ningwei Dai, and Hung-Sheng Lee. 2019. Furion: Engineering high-quality immersive virtual reality on today's mobile devices. IEEE Transactions on Mobile Computing (2019).
[47]
Teesid Leelasawassuk, Dima Damen, and Walterio W Mayol-Cuevas. 2015. Estimating visual attention from a head mounted IMU. In Proceedings of the 2015 ACM International Symposium on Wearable Computers. 147--150.
[48]
Tobias Lentz, Dirk Schröder, Michael Vorländer, and Ingo Assenmacher. 2007. Virtual reality system with integrated sound field simulation and reproduction. EURASIP journal on advances in signal processing 2007, 1 (2007), 070540.
[49]
Hui Liu, Houshang Darabi, Pat Banerjee, and Jing Liu. 2007. Survey of wireless indoor positioning techniques and systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 6 (2007), 1067--1080.
[50]
Yu Liu, Yanping Chen, Lili Shi, Zengshan Tian, Mu Zhou, and Lingxia Li. 2015. Accelerometer based joint step detection and adaptive step length estimation algorithm using handheld devices. Journal of Communications 10, 7 (2015), 520--525.
[51]
Yunhao Liu, Zheng Yang, Xiaoping Wang, and Lirong Jian. 2010. Location, localization, and localizability. Journal of Computer Science and Technology 25, 2 (2010), 274--297.
[52]
Kieran Mansley, Alastair R Beresford, and David Scott. 2004. The carrot approach: encouraging use of location systems. In International Conference on Ubiquitous Computing. Springer, 366--383.
[53]
Wenguang Mao, Mei Wang, and Lili Qiu. 2018. Aim: acoustic imaging on a mobile. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 468--481.
[54]
Wenguang Mao, Zaiwei Zhang, Lili Qiu, Jian He, Yuchen Cui, and Sangki Yun. 2017. Indoor follow me drone. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 345--358.
[55]
Microsoft. 2020. Microsoft Soundscape: A Map Delivered in 3D Sound - Microsoft Research. https://www.microsoft.com/en-us/research/video/microsoft-soundscape-map-delivered-3d-sound/ [Online; accessed 17-March-2020].
[56]
Pauli Minnaar, S Krarup Olesen, Flemming Christensen, and Henrik Møller. 2001. Localization with binaural recordings from artificial and human heads. Journal of the Audio Engineering Society 49, 5 (2001), 323--336.
[57]
Henrik Møller. 1992. Fundamentals of binaural technology. Applied acoustics 36, 3--4 (1992), 171--218.
[58]
Rajalakshmi Nandakumar, Vikram Iyer, and Shyamnath Gollakota. 2018. 3D Localization for Sub-Centimeter Sized Devices. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. ACM, 108--119.
[59]
Anh Nguyen, Raghda Alqurashi, Zohreh Raghebi, Farnoush Banaei-Kashani, Ann C Halbower, and Tam Vu. 2016. A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM. ACM, 230--244.
[60]
David R Perrott and AD Musicant. 1977. Minimum auditory movement angle: Binaural localization of moving sound sources. The Journal of the Acoustical Society of America 62, 6 (1977), 1463--1466.
[61]
Swadhin Pradhan, Ghufran Baig, Wenguang Mao, Lili Qiu, Guohai Chen, and Bo Yang. 2018. Smartphone-based Acoustic Indoor Space Mapping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 75.
[62]
Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Roy Choudhury. 2019. STEAR: Robust Step Counting from Earables. In Proceedings of the 1st International Workshop on Earable Computing. 36--41.
[63]
Sujatha Rajagopal. 2008. Personal dead reckoning system with shoe mounted inertial sensors. Master's Degree Project, Stockholm, Sweden (2008).
[64]
Eike Rehder, Horst Kloeden, and Christoph Stiller. 2014. Head detection and orientation estimation for pedestrian safety. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 2292--2297.
[65]
Zhi-Hong Ren, Gui-Lin Wang, and David Ho. 2006. Mobile phone with pedometer. US Patent App. 11/136,265.
[66]
Rico Malvar. 2020. A new Soundscape experience with Bose Frames - Microsoft Accessibility Blog. https://blogs.microsoft.com/accessibility/soundscape-boseframes/ [Online; accessed 17-March-2020].
[67]
Nirupam Roy, He Wang, and Romit Roy Choudhury. 2014. I am a smartphone and I can tell my user's walking direction. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 329--342.
[68]
Sheng Shen, Daguan Chen, Yu-Lin Wei, Zhijian Yang, and Romit Roy Choudhury. 2020. Voice localization using nearby wall reflections. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
[69]
SH Shin, CG Park, JW Kim, HS Hong, and JM Lee. 2007. Adaptive step length estimation algorithm using low-cost MEMS inertial sensors. In 2007 ieee sensors applications symposium. IEEE, 1--5.
[70]
Anran Wang and Shyamnath Gollakota. 2019. MilliSonic: Pushing the Limits of Acoustic Motion Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 18.
[71]
He Wang, Souvik Sen, Ahmed Elgohary, Moustafa Farid, Moustafa Youssef, and Romit Roy Choudhury. 2012. No Need to War-drive: Unsupervised Indoor Localization. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services (MobiSys '12). ACM, New York, NY, USA, 197--210.
[72]
Harvey Weinberg. 2002. Using the ADXL202 in pedometer and personal navigation applications. Analog Devices AN-602 application note 2, 2 (2002), 1--6.
[73]
Jens Windau and Laurent Itti. 2016. Walking compass with head-mounted IMU sensor. In 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5542--5547.
[74]
Zhuoling Xiao, Hongkai Wen, Andrew Markham, and Niki Trigoni. 2014. Robust pedestrian dead reckoning (R-PDR) for arbitrary mobile device placement. In 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN). IEEE, 187--196.
[75]
Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging Multi-Dimensionality for Passive Indoor Wi-Fi Tracking. In The 25th Annual International Conference on Mobile Computing and Networking. ACM, 1--16.
[76]
Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13). 71--84.
[77]
Zheng Yang, Chenshu Wu, and Yunhao Liu. 2012. Locating in fingerprint space: wireless indoor localization with little human intervention. In Proceedings of the 18th annual international conference on Mobile computing and networking. ACM, 269--280.
[78]
Ali Yassin, Youssef Nasser, Mariette Awad, Ahmed Al-Dubai, Ran Liu, Chau Yuen, Ronald Raulefs, and Elias Aboutanios. 2016. Recent advances in indoor localization: A survey on theoretical approaches and applications. IEEE Communications Surveys & Tutorials 19, 2 (2016), 1327--1346.
[79]
Ji Zhang and Sanjiv Singh. 2017. Low-drift and real-time lidar odometry and mapping. Autonomous Robots 41, 2 (2017), 401--416.
[80]
Yanzi Zhu, Zhujun Xiao, Yuxin Chen, Zhijing Li, Max Liu, Ben Y. Zhao, and Haitao Zheng. 2018. Et Tu Alexa? When Commodity WiFi Devices Turn into Adversarial Motion Sensors. arXiv e-prints, Article arXiv:1810.10109 (Oct 2018), arXiv:1810.10109 pages. arXiv:cs.CR/1810.10109

Cited By

View all
  • (2024)Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air NonlinearityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785468:3(1-30)Online publication date: 9-Sep-2024
  • (2024)AudioMove: Applying the Spatial Audio to Multi-Directional Limb Exercise GuidanceProceedings of the ACM on Human-Computer Interaction10.1145/36764898:MHCI(1-26)Online publication date: 24-Sep-2024
  • (2024)BrushBuds: Toothbrushing Tracking Using Earphone IMUsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680521(655-660)Online publication date: 5-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MobiCom '20: Proceedings of the 26th Annual International Conference on Mobile Computing and Networking
April 2020
621 pages
ISBN:9781450370851
DOI:10.1145/3372224
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. acoustics
  2. augmented reality
  3. dead reckoning
  4. head related transfer function (HRTF)
  5. indoor localization
  6. inertial measurement unit (IMU)
  7. motion tracking
  8. sensor fusion
  9. smart earphones
  10. spatial audio
  11. wearable computing

Qualifiers

  • Research-article

Funding Sources

  • NSF
  • Google
  • NIH
  • Samsung

Conference

MobiCom '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)406
  • Downloads (Last 6 weeks)50
Reflects downloads up to 10 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air NonlinearityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785468:3(1-30)Online publication date: 9-Sep-2024
  • (2024)AudioMove: Applying the Spatial Audio to Multi-Directional Limb Exercise GuidanceProceedings of the ACM on Human-Computer Interaction10.1145/36764898:MHCI(1-26)Online publication date: 24-Sep-2024
  • (2024)BrushBuds: Toothbrushing Tracking Using Earphone IMUsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680521(655-660)Online publication date: 5-Oct-2024
  • (2024)ASLiquid: Non-Intrusive Liquid Counterfeit Identification with Your EarphonesProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699321(41-53)Online publication date: 4-Nov-2024
  • (2024)Enabling 6D Pose Tracking on Your Acoustic DevicesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661875(15-28)Online publication date: 3-Jun-2024
  • (2024)Exploring the Feasibility of Remote Cardiac Auscultation Using EarphonesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649366(357-372)Online publication date: 29-May-2024
  • (2024)Indoor Drone Localization and Tracking Based on Acoustic Inertial MeasurementIEEE Transactions on Mobile Computing10.1109/TMC.2023.333586023:6(7537-7551)Online publication date: Jun-2024
  • (2024)Live Speech Recognition via Earphone Motion SensorsIEEE Transactions on Mobile Computing10.1109/TMC.2023.333321423:6(7284-7300)Online publication date: Jun-2024
  • (2024)BLEAR: Practical Wireless Earphone Tracking under BLE protocol2024 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom59722.2024.10494424(164-173)Online publication date: 11-Mar-2024
  • (2024)EHTrack: Earphone-Based Head Tracking via Only Acoustic SignalsIEEE Internet of Things Journal10.1109/JIOT.2023.329841211:3(4063-4075)Online publication date: 1-Feb-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media