research-article

Environment-aware Multi-person Tracking in Indoor Environments with MmWave Radars

Authors:

Daqing ZhangAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 7, Issue 3

Article No.: 89, Pages 1 - 29

https://doi.org/10.1145/3610902

Published: 27 September 2023 Publication History

Abstract

Device-free indoor localization and tracking using commercial millimeter wave radars have attracted much interest lately due to their non-intrusive nature and high spatial resolution. However, it is challenging to achieve high tracking accuracy due to rich multipath reflection and occlusion in indoor environments. Static objects with non-negligible reflectance of mmWave signals interact with moving human subjects and generate time-varying multipath ghosts and shadow ghosts, which can be easily confused as real subjects. To characterize the complex interactions, we first develop a geometric model that estimates the location of multipath ghosts given the locations of humans and static reflectors. Based on this model, the locations of static reflectors that form a reflection map are automatically estimated from received radar signals as a single person traverses the environment along arbitrary trajectories. The reflection map allows for the elimination of multipath and shadow ghost interference as well as the augmentation of weakly reflected human subjects in occluded areas. The proposed environment-aware multi-person tracking system can generate reflection maps with a mean error of 15.5cm and a 90-percentile error of 30.3cm, and achieve multi-person tracking accuracy with a mean error of 8.6cm and a 90-percentile error of 17.5cm, in four representative indoor spaces with diverse subjects using a single mmWave radar.

References

[1]

Fadel Adib, Zachary Kabelac, and Dina Katabi. 2015. Multi-person localization via RF body reflections. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 279--292.

Digital Library

[2]

Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). 317--329.

Digital Library

[3]

Juhana Ahtiainen, Sami Terho, and Sampsa Koponen. 2010. Radar based detection and tracking of a walking human. IFAC Proceedings Volumes 43, 16 (2010), 437--442.

[4]

Yaakov Bar-Shalom, X Rong Li, and Thiagalingam Kirubarajan. 2001. Estimation with applications to tracking and navigation: theory algorithms and software. John Wiley & Sons.

[5]

Raffaele Bruno and Franca Delmastro. 2003. Design and analysis of a bluetooth-based indoor localization system. In Personal Wireless Communications: IFIP-TC6 8th International Conference, PWC 2003, Venice, Italy, September 23-25, 2003. Proceedings 8. Springer, 711--725.

[6]

SangHyun Chang, Rangoli Sharan, Michael Wolf, Naoki Mitsumoto, and Joel W Burdick. 2010. People tracking with UWB radar using a multiple-hypothesis tracking of clusters (MHTC) method. International Journal of Social Robotics 2, 1 (2010), 3--18.

[7]

SangHyun Chang, Michael Wolf, and Joel W Burdick. 2009. An MHT algorithm for UWB radar-based multiple human target tracking. In 2009 IEEE International Conference on Ultra-Wideband. IEEE, 459--463.

[8]

SangHyun Chang, Michael Wolf, and Joel W Burdick. 2010. Human detection and tracking via ultra-wideband (UWB) radar. In 2010 IEEE International Conference on Robotics and Automation. IEEE, 452--457.

[9]

Weiyan Chen, Fusang Zhang, Tao Gu, Kexing Zhou, Zixuan Huo, and Daqing Zhang. 2021. Constructing floor plan through smoke using ultra wideband radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--29.

Digital Library

[10]

Byeong-Gon Choi, Won-Ho Jeong, and Kyung-Seok Kim. 2015. Characteristics analysis of reflection and transmission according to building materials in the millimeterwave band. Recent advances on Electroscience and Computers (2015), 154--158.

[11]

Nicolo Decarli, Francesco Guidi, and Davide Dardari. 2013. A novel joint RFID and radar sensor network for passive localization: Design and performance bounds. IEEE Journal of Selected Topics in Signal Processing 8, 1 (2013), 80--95.

[12]

Tobias Deissler and Jorn Thielecke. 2009. Feature based indoor mapping using a bat-type UWB radar. In 2009 IEEE International Conference on Ultra-Wideband. IEEE, 475--479.

[13]

Javier JM Diaz, Rodrigo de A Maues, Rodrigo B Soares, Eduardo F Nakamura, and Carlos MS Figueiredo. 2010. Bluepass: An indoor bluetooth-based localization system for mobile applications. In The IEEE symposium on Computers and Communications. IEEE, 778--783.

Digital Library

[14]

Ruoyu Feng, Eddy De Greef, Maxim Rykunov, Hichem Sahli, Sofie Pollin, and André Bourdoux. 2021. Multipath ghost recognition for indoor mimo radar. IEEE Transactions on Geoscience and Remote Sensing 60 (2021), 1--10.

[15]

Anna Guerra, Francesco Guidi, Lilia Ahtaryieva, Nicoló Decarli, and Davide Dardari. 2016. Crowd-based personal radars for indoor mapping using UWB measurements. In 2016 IEEE International Conference on Ubiquitous Wireless Broadband (ICUWB). IEEE, 1--4.

[16]

Steven B Heymsfield, Allison Martin-Nguyen, Tung M Fong, Dympna Gallagher, and Angelo Pietrobelli. 2008. Body circumferences: clinical implications emerging from a new geometric model. Nutrition & Metabolism 5, 1 (2008), 1--17.

[17]

Lukasz Januszkiewicz, Paolo Di Barba, and Sawomir Hausman. 2016. Automated identification of human-body model parameters. International Journal of Applied Electromagnetics and Mechanics 51, s1 (2016), S41--S47.

[18]

Mahfuza Khatun, Changyu Guo, and Hani Mehrpouyan. 2021. Penetration and Reflection Characteristics in Millimeter-Wave Indoor Channels. In 2021 IEEE-APS Topical Conference on Antennas and Propagation in Wireless Communications (APWC). IEEE, 1--5.

[19]

Calvin Le, Traian Dogaru, Lam Nguyen, and Marc A Ressler. 2009. Ultrawideband (UWB) radar imaging of building interior: Measurements and predictions. IEEE Transactions on Geoscience and Remote Sensing 47, 5 (2009), 1409--1420.

[20]

Mattia Lecci, Paolo Testolina, Marco Giordani, Michele Polese, Tanguy Ropitault, Camillo Gentile, Neeraj Varshney, Anuraag Bodi, and Michele Zorzi. 2020. Simplified ray tracing for the millimeter wave channel: A performance evaluation. In 2020 Information Theory and Applications Workshop (ITA). IEEE, 1--6.

[21]

Huquan Li, Guolong Cui, Shisheng Guo, Lingjiang Kong, and Xiaobo Yang. 2020. Target tracking and ghost mitigation based on multi-view through-the-wall radar imaging. In 2020 IEEE Radar Conference (RadarConf20). IEEE, 1--5.

[22]

Xiang Li, Shengjie Li, Daqing Zhang, Jie Xiong, Yasha Wang, and Hong Mei. 2016. Dynamic-music: accurate device-free indoor localization. In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. 196--207.

Digital Library

[23]

Xiang Li, Daqing Zhang, Qin Lv, Jie Xiong, Shengjie Li, Yue Zhang, and Hong Mei. 2017. IndoTrack: Device-free indoor human tracking with commodity Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--22.

Digital Library

[24]

Jihye Lim, Hyunyoung Choi, Eui Kyung Roh, Hwasook Yoo, and Eunae Kim. 2015. Assessment of airflow and microclimate for the running wear jacket with slits using CFD simulation. Fashion and Textiles 2, 1 (2015), 1.

[25]

Chenwen Liu, Shengheng Liu, Cheng Zhang, Yongming Huang, and Haiming Wang. 2020. Multipath propagation analysis and ghost target removal for FMCW automotive radars. In IET International Radar Conference (IET IRC 2020), Vol. 2020. IET, 330--334.

[26]

Oren Longman, Shahar Villeval, and Igal Bilik. 2021. Multipath Ghost Targets Mitigation in Automotive Environments. In 2021 IEEE Radar Conference (RadarConf21). IEEE, 1--5.

[27]

Chris Xiaoxuan Lu, Stefano Rosa, Peijun Zhao, Bing Wang, Changhao Chen, John A Stankovic, Niki Trigoni, and Andrew Markham. 2020. See through smoke: robust indoor mapping with low-cost mmwave radar. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 14--27.

Digital Library

[28]

Kan Ngamakeur, Sira Yongchareon, Jian Yu, and Saeed Ur Rehman. 2020. A survey on device-free indoor localization and tracking in the multi-resident environment. ACM Computing Surveys (CSUR) 53, 4 (2020), 1--29.

Digital Library

[29]

Andreas Östman. 2016. Detection and Tracking of Human Targets using Ultra-Wideband Radar.

[30]

Jacopo Pegoraro, Francesca Meneghello, and Michele Rossi. 2020. Multiperson continuous tracking and identification from mm-wave micro-Doppler signatures. IEEE Transactions on Geoscience and Remote Sensing 59, 4 (2020), 2994--3009.

[31]

Jacopo Pegoraro and Michele Rossi. 2021. Real-time people tracking and identification from sparse mm-wave radar point-clouds. IEEE Access 9 (2021), 78504--78520.

[32]

Jacopo Pegoraro and Michele Rossi. 2022. Human Tracking with mmWave Radars: a Deep Learning Approach with Uncertainty Estimation. arXiv preprint arXiv:2205.03274 (2022).

[33]

Jacopo Pegoraro, Domenico Solimini, Federico Matteo, Enver Bashirov, Francesca Meneghello, and Michele Rossi. 2020. Deep Learning for Accurate Indoor Human Tracking with a mm-Wave Radar. In 2020 IEEE Radar Conference (RadarConf20). IEEE, 1--6.

[34]

Kun Qian, Zhaoyuan He, and Xinyu Zhang. 2020. 3D point cloud generation with millimeter-wave radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--23.

Digital Library

[35]

Theodore S Rappaport, Shu Sun, Rimma Mayzus, Hang Zhao, Yaniv Azar, Kevin Wang, George N Wong, Jocelyn K Schulz, Mathew Samimi, and Felix Gutierrez. 2013. Millimeter wave mobile communications for 5G cellular: It will work! IEEE access 1 (2013), 335--349.

[36]

Wenjie Ruan, Quan Z Sheng, Lina Yao, Tao Gu, Michele Ruta, and Longfei Shangguan. 2016. Device-free indoor localization and tracking through human-object interactions. In 2016 IEEE 17th international symposium on a world of wireless, mobile and multimedia networks (WoWMoM). IEEE, 1--9.

[37]

Wenjie Ruan, Quan Z Sheng, Lina Yao, Xue Li, Nickolas JG Falkner, and Lei Yang. 2018. Device-free human localization and tracking with UHF passive RFID tags: A data-driven approach. Journal of Network and Computer Applications 104 (2018), 78--96.

[38]

Katsuyoshi Sato, Hideki Kozima, Hiroshi Masuzawa, Takeshi Manabe, Toshio Ihara, Yoshinori Kasashima, and Katsunori Yamaki. 1995. Measurements of reflection characteristics and refractive indices of interior construction materials in millimeter-wave bands. In 1995 IEEE 45th Vehicular Technology Conference. Countdown to the Wireless Twenty-First Century, Vol. 1. IEEE, 449--453.

[39]

Babak Shahian Jahromi, Theja Tulabandhula, and Sabri Cetin. 2019. Real-time hybrid multi-sensor fusion framework for perception in autonomous vehicles. Sensors 19, 20 (2019), 4357.

[40]

Junyang Shen, Andreas F Molisch, and Jussi Salmi. 2012. Accurate passive location estimation using TOA measurements. IEEE Transactions on Wireless Communications 11, 6 (2012), 2182--2192.

[41]

Yongping Song, Jun Hu, Ning Chu, Tian Jin, Jianwen Zhang, and Zhimin Zhou. 2018. Building layout reconstruction in concealed human target sensing via UWB MIMO through-wall imaging radar. IEEE Geoscience and Remote Sensing Letters 15, 8 (2018), 1199--1203.

[42]

Jan AJ Stolwijk and James D Hardy. 2010. Control of body temperature. Comprehensive Physiology (2010), 45--68.

[43]

Laurent Storrer, Hasan Can Yildirim, Morgane Crauwels, Evert I Pocoma Copa, Sofie Pollin, Jérôme Louveaux, Philippe De Doncker, and François Horlin. 2021. Indoor tracking of multiple individuals with an 802.11 ax Wi-Fi-based multi-antenna passive radar. IEEE Sensors Journal 21, 18 (2021), 20462--20474.

[44]

Filippo Valmori, Andrea Giorgetti, Matteo Mazzotti, Enrico Paolini, and Marco Chiani. 2016. Indoor detection and tracking of human targets with UWB radar sensor networks. In 2016 IEEE International Conference on Ubiquitous Wireless Broadband (ICUWB). IEEE, 1--4.

[45]

Christoph Will, Prachi Vaishnav, Abhiram Chakraborty, and Avik Santra. 2019. Human target detection, tracking, and classification using 24-GHz FMCW radar. IEEE Sensors Journal 19, 17 (2019), 7283--7299.

[46]

Chenshu Wu, Feng Zhang, Beibei Wang, and KJ Ray Liu. 2020. mmTrack: Passive multi-person localization using commodity millimeter wave radio. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications. IEEE, 2400--2409.

Digital Library

[47]

Ting Wu, Theodore S Rappaport, and Christopher M Collins. 2015. The human body and millimeter-wave wireless communication systems: Interactions and implications. In 2015 IEEE International Conference on Communications (ICC). IEEE, 2423--2429.

[48]

Shichao Yue, Hao He, Peng Cao, Kaiwen Zha, Masayuki Koizumi, and Dina Katabi. 2022. CornerRadar: RF-Based Indoor Localization Around Corners. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1--24.

Digital Library

[49]

Peijun Zhao, Chris Xiaoxuan Lu, Jianan Wang, Changhao Chen, Wei Wang, Niki Trigoni, and Andrew Markham. 2019. mid: Tracking and identifying people with millimeter wave radar. In 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE, 33--40.

[50]

Peijun Zhao, Chris Xiaoxuan Lu, Jianan Wang, Changhao Chen, Wei Wang, Niki Trigoni, and Andrew Markham. 2021. Human tracking and identification through a millimeter wave radar. Ad Hoc Networks 116 (2021), 102475.

[51]

Jingyue Zheng, Xingyu Gu, Shisheng Guo, and Guolong Cui. 2022. Multipath False Target Removal for Indoor Localization. In Proceedings of 2021 International Conference on Autonomous Unmanned Systems (ICAUS 2021). Springer, 1367--1378.

Cited By

Liu YZhang JChen YWang WYang SNa XSun YHe Y(2025)Real-Time Continuous Activity Recognition With a Commercial mmWave RadarIEEE Transactions on Mobile Computing10.1109/TMC.2024.348381324:3(1684-1698)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3483813
Kim JJung JLim SKim S(2025)Multipath Suppression Using 2-D Angle Analysis Based on the MIMO FMCW RadarIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.350006174(1-12)Online publication date: 2025
https://doi.org/10.1109/TIM.2024.3500061
Mattos ABrante GMoritz GSouza R(2024)Human and Small Animal Detection Using Multiple Millimeter-Wave Radars and Data Fusion: Enabling Safe ApplicationsSensors10.3390/s2406190124:6(1901)Online publication date: 16-Mar-2024
https://doi.org/10.3390/s24061901
Show More Cited By

Index Terms

Environment-aware Multi-person Tracking in Indoor Environments with MmWave Radars
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing
    2. Ubiquitous and mobile computing systems and tools

Recommendations

A Survey on Device-free Indoor Localization and Tracking in the Multi-resident Environment

Indoor device-free localization and tracking can bring both convenience and privacy to users compared with traditional solutions such as camera-based surveillance and RFID tag-based tracking. Technologies such as Wi-Fi, wireless sensor, and infrared ...
A multi-model sequential Monte Carlo methodology for indoor tracking: Algorithms and experimental results

In this paper we address the problem of indoor tracking using received signal strength (RSS) as a position-dependent data measurement. Since RSS is highly influenced by multipath propagation, it turns out very hard to adequately model the correspondence ...
Indoor Tracking and Navigation Using Received Signal Strength and Compressive Sensing on a Mobile Device

An indoor tracking and navigation system based on measurements of received signal strength (RSS) in wireless local area network (WLAN) is proposed. In the system, the location determination problem is solved by first applying a proximity constraint to ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 7, Issue 3

September 2023

1734 pages

EISSN:2474-9567

DOI:10.1145/3626192

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2023

Published in IMWUT Volume 7, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China A3 Foresight Program

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
1,037
Total Downloads

Downloads (Last 12 months)552
Downloads (Last 6 weeks)60

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YZhang JChen YWang WYang SNa XSun YHe Y(2025)Real-Time Continuous Activity Recognition With a Commercial mmWave RadarIEEE Transactions on Mobile Computing10.1109/TMC.2024.348381324:3(1684-1698)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3483813
Kim JJung JLim SKim S(2025)Multipath Suppression Using 2-D Angle Analysis Based on the MIMO FMCW RadarIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.350006174(1-12)Online publication date: 2025
https://doi.org/10.1109/TIM.2024.3500061
Mattos ABrante GMoritz GSouza R(2024)Human and Small Animal Detection Using Multiple Millimeter-Wave Radars and Data Fusion: Enabling Safe ApplicationsSensors10.3390/s2406190124:6(1901)Online publication date: 16-Mar-2024
https://doi.org/10.3390/s24061901
Chang ZZhang FMa XWang PChen WZhang DJouaber BZhang D(2024)MmECare: Enabling Fine-grained Vital Sign Monitoring for Emergency Care with Handheld MmWave RadarsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997668:4(1-24)Online publication date: 21-Nov-2024
https://doi.org/10.1145/3699766
Li YWang CXie LJin QFan LNing JLu S(2024)Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997398:4(1-26)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699739
Wang DZhang XWang KWang LFan XZhang Y(2024)RDGait: A mmWave Based Gait User Recognition System for Complex Indoor Environments Using Single-chip RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785528:3(1-31)Online publication date: 9-Sep-2024
https://doi.org/10.1145/3678552
Abdelnasser HHeggo MPang OKovac MMcCann J(2024)RaDro: Indoor Drone Tracking Using Millimeter Wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785498:3(1-23)Online publication date: 9-Sep-2024
https://doi.org/10.1145/3678549
Pettys-Baker RClarke MHolschuh BKostakos VKay JHoang T(2024)Functional Now, Wearable Later: Examining the Design Practices of Wearable TechnologistsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676615(71-81)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676615
Zheng ZLiang YLyu RBao JHuang YZhou AMa HWang JMeng XShao CTang YZhang QShu YLiu JTan RHe YChen J(2024)BP3: Improving Cuff-less Blood Pressure Monitoring Performance by Fusing mmWave Pulse Wave Sensing and Physiological FactorsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699370(730-743)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699370
Bañuelos JSigg SHe JSalim FCosta-Requena J(2024)Generating Multivariate Synthetic Time Series Data for Absent Sensors from Correlated SourcesProceedings of the 2nd International Workshop on Networked AI Systems10.1145/3662004.3663553(19-24)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3662004.3663553
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents