View-agnostic Human Exercise Cataloging with Single MmWave Radar
Article No.: 117, Pages 1 - 23
Abstract
Advances in mmWave-based sensing have enabled a privacy-friendly approach to pose and gesture recognition. Yet, providing robustness with the sparsity of reflected signals has been a long-standing challenge towards its practical deployment, constraining subjects to often face the radar. We present RF-HAC- a first-of-its-kind system that brings robust, automated and real-time human activity cataloging to practice by not only classifying exercises performed by subjects in their natural environments and poses, but also tracking the corresponding number of exercise repetitions. RF-HAC's unique approach (i) brings the diversity of multiple radars to scalably train a novel, self-supervised, pose-agnostic transformer-based exercise classifier directly on 3D RF point clouds with minimal manual effort and be deployed on a single radar; and (ii) leverages the underlying doppler behavior of exercises to design a robust self-similarity based segmentation algorithm for counting the repetitions in unstructured RF point clouds. Evaluations on a comprehensive set of challenging exercises in both seen and unseen environments/subjects highlight RF-HAC's robustness with high accuracy (over 90%) and readiness for real-time, practical deployments over prior art.
Supplemental Material
MP4 File - Video for RF-HAC
Supplemental video showing the functionality of RF-HAC
- Download
- 35.59 MB
References
[1]
[n. d.]. Intel® NUC 11 Enthusiast Mini PC - NUC11PHKi7CAA. https://www.intel.com/content/www/us/en/products/sku/195961/intel-nuc-11-enthusiast-mini-pc-nuc11phki7caa/specifications.html.
[2]
[n. d.]. Linux cgroups. https://man7.org/linux/man-pages/man7/cgroups.7.html.
[3]
[n. d.]. Review: Amazon Halo Rise. https://www.wired.com/review/amazon-halo-rise/.
[4]
[n. d.]. RF-HAC datasets and code. https://bit.ly/493DXQw.
[5]
[n.d.]. RF-HAC github. https://github.com/Ohesachite/radar-nn-model.
[6]
[n. d.]. TI 3x4 radar chips. https://www.ti.com/product/AWR6843?utm_source=google&utm_medium=cpc&utm_campaign=epd-null-null-GPN_EN-cpc-pf-google-wwe&utm_content=AWR6843&ds_k=AWR6843&DCM=yes&gclid=CjwKCAjwoqGnBhAcEiwAwK-OkdCizqhgLwaVF_urOlSGw9HlM3UdDdlVbRSW7tEIcjmomfrNwm3xOBoCK4oQAvD_BwE&gclsrc=aw.ds.
[7]
Aakriti Adhikari, Hem Regmi, Sanjib Sur, and Srihari Nelakuditi. 2022. MiShape: Accurate Human Silhouettes and Body Joints from Commodity Millimeter-Wave Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--31.
[8]
Adeel Ahmad, June Chul Roh, Dan Wang, and Aish Dubey. 2018. Vital signs monitoring of multiple people using a FMCW millimeter-wave sensor. In 2018 IEEE Radar Conference (RadarConf18). IEEE, 1450--1455.
[9]
Mubarak A Alanazi, Abdullah K Alhazmi, Osama Alsattam, Kara Gnau, Meghan Brown, Shannon Thiel, Kurt Jackson, and Vamsy P Chodavarapu. 2022. Towards a low-cost solution for gait analysis using millimeter wave sensor and machine learning. Sensors 22, 15 (2022), 5470.
[10]
Sizhe An and Umit Y Ogras. 2021. Mars: mmwave-based assistive rehabilitation system for smart healthcare. ACM Transactions on Embedded Computing Systems (TECS) 20, 5s (2021), 1--22.
[11]
Robert Bodor, Bennett Jackson, and Nikolaos Papanikolopoulos. 2003. Vision-based human tracking and activity recognition. In Proc. of the 11th Mediterranean Conf. on Control and Automation, Vol. 1. Citeseer, 1--6.
[12]
Baicheng Chen, Huining Li, Zhengxiong Li, Xingyu Chen, Chenhan Xu, and Wenyao Xu. 2020. ThermoWave: a new paradigm of wireless passive temperature monitoring via mmWave sensing. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
[13]
L Minh Dang, Kyungbok Min, Hanxiang Wang, Md Jalil Piran, Cheol Hee Lee, and Hyeonjoon Moon. 2020. Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recognition 108 (2020), 107561.
[14]
Ashutosh Dhekne, Mahanth Gowda, Yixuan Zhao, Haitham Hassanieh, and Romit Roy Choudhury. 2018. Liquid: A wireless liquid identifier. In Proceedings of the 16th annual international conference on mobile systems, applications, and services. 442--454.
[15]
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. 12, null (jul 2011), 2121--2159.
[16]
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. 2020. Counting Out Time: Class Agnostic Video Repetition Counting in the Wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[17]
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (Portland, Oregon) (KDD'96). AAAI Press, 226--231.
[18]
Hehe Fan, Yi Yang, and Mohan Kankanhalli. 2021. Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR.
[19]
Tianbo Gu, Zheng Fang, Zhicheng Yang, Pengfei Hu, and Prasant Mohapatra. 2019. Mmsense: Multi-person detection and identification via mmwave sensing. In Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems. 45--50.
[20]
Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh. 2020. Through fog high-resolution imaging using millimeter wave radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11464--11473.
[21]
Unsoo Ha, Salah Assana, and Fadel Adib. 2020. Contactless Seismocardiography via Deep Learning Radars. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCom '20). Association for Computing Machinery, New York, NY, USA, Article 62, 14 pages. https://doi.org/10.1145/3372224.3419982
[22]
Dan Hendrycks and Kevin Gimpel. 2023. Gaussian Error Linear Units (GELUs). arXiv:1606.08415 [cs.LG]
[23]
Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, and Akhil Mathur. 2022. ColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 1, Article 17 (mar 2022), 28 pages. https://doi.org/10.1145/3517246
[24]
Sorachi Kato, Tomoki Murakami, Takuya Fujihashi, Takashi Watanabe, and Shunsuke Saruwatari. 2022. CBR-ACE: Counting human exercise using Wi-Fi beamforming reports. Journal of Information Processing 30 (2022), 66--74.
[25]
Hao Kong, Xiangyu Xu, Jiadi Yu, Qilin Chen, Chenguang Ma, Yingying Chen, Yi-Chao Chen, and Linghe Kong. 2022. M3Track: mmwave-Based multi-User 3D Posture Tracking (MobiSys '22). Association for Computing Machinery, New York, NY, USA, 491--503. https://doi.org/10.1145/3498361.3538926
[26]
Der-Tsai Lee and Bruce J Schachter. 1980. Two algorithms for constructing a Delaunay triangulation. International Journal of Computer & Information Sciences 9, 3 (1980), 219--242.
[27]
Ofir Levy and Lior Wolf. 2015. Live repetition counting. In Proceedings of the IEEE international conference on computer vision. 3020--3028.
[28]
Zhengxiong Li, Fenglong Ma, Aditya Singh Rathore, Zhuolin Yang, Baicheng Chen, Lu Su, and Wenyao Xu. 2020. Wavespy: Remote and through-wall screen attack via mmwave sensing. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 217--232.
[29]
J. Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37, 1 (1991), 145--151. https://doi.org/10.1109/18.61115
[30]
Haipeng Liu, Kening Cui, Kaiyuan Hu, Yuheng Wang, Anfu Zhou, Liang Liu, and Huadong Ma. 2022. mTransSee: Enabling Environment-Independent mmWave Sensing Based Gesture Recognition via Transfer Learning. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 1, Article 23 (mar 2022), 28 pages. https://doi.org/10.1145/3517231
[31]
Hankai Liu, Xiulong Liu, Xin Xie, Xinyu Tong, and Keqiu Li. 2024. PmTrack: Enabling Personalized mmWave-based Human Tracking. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 4, Article 167 (jan 2024), 30 pages. https://doi.org/10.1145/3631433
[32]
Lingfei Mo, Fan Li, Yanjia Zhu, and Anjie Huang. 2016. Human physical activity recognition based on computer vision with deep learning model. In 2016 IEEE international instrumentation and measurement technology conference proceedings. IEEE, 1--6.
[33]
Cecily Morrison, Peter Culmer, Helena Mentis, and Tamar Pincus. 2016. Vision-based body tracking: turning Kinect into a clinical tool. Disability and Rehabilitation: Assistive Technology 11, 6 (2016), 516--520.
[34]
Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics 22, 3 (1951), 400 - 407. https://doi.org/10.1214/aoms/1177729586
[35]
Dariush Salami, Ramin Hasibi, Sameera Palipana, Petar Popovski, Tom Michoel, and Stephan Sigg. 2021. Tesla-Rapture: A Lightweight Gesture Recognition System From mmWave Radar Sparse Point Clouds. IEEE Transactions on Mobile Computing 22 (2021), 4946--4960. https://api.semanticscholar.org/CorpusID:247092918
[36]
David Schneider, Saquib Sarfraz, Alina Roitberg, and Rainer Stiefelhagen. 2022. Pose-based contrastive learning for domain agnostic activity representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3433--3443.
[37]
Akash Deep Singh, Sandeep Singh Sandha, Luis Garcia, and Mani Srivastava. 2019. RadHAR: Human Activity Recognition from Point Clouds Generated through a Millimeter-Wave Radar. In Proceedings of the 3rd ACM Workshop on Millimeter-Wave Networks and Sensing Systems (Los Cabos, Mexico) (mmNets'19). Association for Computing Machinery, New York, NY, USA, 51--56. https://doi.org/10.1145/3349624.3356768
[38]
Edward M Sitar and Sanjib Sur. 2022. MilliFit: Millimeter-Wave Wireless Sensing Based At-Home Exercise Classification. In 2022 18th International Conference on Mobility, Sensing and Networking (MSN). IEEE, 150--154.
[39]
Elahe Soltanaghaei, Akarsh Prabhakara, Artur Balanuta, Matthew Anderson, Jan M Rabaey, Swarun Kumar, and Anthony Rowe. 2021. Millimetro: mmWave retro-reflective tags for accurate, long range localization. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 69--82.
[40]
Girish Tiwari and Shalabh Gupta. 2021. An mmWave radar based real-time contactless fitness tracker using deep CNNs. IEEE Sensors Journal 21, 15 (2021), 17262--17270.
[41]
Chao Wang, Feng Lin, Tiantian Liu, Kaidi Zheng, Zhibo Wang, Zhengxiong Li, Ming-Chun Huang, Wenyao Xu, and Kui Ren. 2022. mmEve: eavesdropping on smartphone's earpiece via COTS mmWave device. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 338--351.
[42]
Yuheng Wang, Haipeng Liu, Kening Cui, Anfu Zhou, Wensheng Li, and Huadong Ma. 2021. m-Activity: Accurate and Real-Time Human Activity Recognition Via Millimeter Wave Radar. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8298--8302. https://doi.org/10.1109/ICASSP39728.2021.9414686
[43]
Yong Wang, Wen Wang, Mu Zhou, Aihu Ren, and Zengshan Tian. 2020. Remote monitoring of human vital signs based on 77-GHz mm-wave FMCW radar. Sensors 20, 10 (2020), 2999.
[44]
Jingmiao Wu, Jie Wang, Qinghua Gao, Miao Pan, and Haixia Zhang. 2021. Path-independent device-free gait recognition using mmwave signals. IEEE Transactions on Vehicular Technology 70, 11 (2021), 11582--11592.
[45]
Yang Xiao, Yuming Du, and Renaud Marlet. 2021. PoseContrast: Class-agnostic object viewpoint estimation in the wild with pose-aware contrastive learning. In 2021 International Conference on 3D Vision (3DV). IEEE, 74--84.
[46]
Yucheng Xie, Ruizhe Jiang, Xiaonan Guo, Yan Wang, Jerry Cheng, and Yingying Chen. 2022. mmFit: Low-Effort Personalized Fitness Monitoring Using Millimeter Wave. In 2022 International Conference on Computer Communications and Networks (ICCCN). 1--10. https://doi.org/10.1109/ICCCN54977.2022.9868878
[47]
Satyapreet Singh Yadav, Radha Agarwal, Kola Bharath, Sandeep Rao, and Chetan Singh Thakur. 2023. tinyRadar for Fitness: A Contactless Framework for Edge Computing. IEEE Transactions on Biomedical Circuits and Systems (2023).
[48]
Jie Yan, Xianlin Zeng, Anfu Zhou, and Huadong Ma. 2022. MM-HAT: Transformer for Millimeter-Wave Sensing Based Human Activity Recognition. In GLOBECOM 2022 - 2022 IEEE Global Communications Conference. 547--553. https://doi.org/10.1109/GLOBECOM48099.2022.10000673
[49]
Zhicheng Yang, Parth H Pathak, Yunze Zeng, Xixi Liran, and Prasant Mohapatra. 2016. Monitoring vital signs using millimeter wave. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. 211--220.
[50]
Chengxi Yu, Zhezhuang Xu, Kun Yan, Ying-Ren Chien, Shih-Hau Fang, and Hsiao-Chun Wu. 2022. Noninvasive Human Activity Recognition Using Millimeter-Wave Radar. IEEE Systems Journal 16, 2 (2022), 3036--3047. https://doi.org/10.1109/JSYST.2022.3140546
[51]
Huaidong Zhang, Xuemiao Xu, Guoqiang Han, and Shengfeng He. 2020. Context-aware and scale-insensitive temporal repetition counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 670--678.
[52]
Long Zhao, Yuxiao Wang, Jiaping Zhao, Liangzhe Yuan, Jennifer J. Sun, Florian Schroff, Hartwig Adam, Xi Peng, Dimitris Metaxas, and Ting Liu. 2021. Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization. arXiv:2012.01405 [cs.CV]
[53]
Tao Zhou, Zhaoyang Xia, Xiangfeng Wang, and Feng Xu. 2021. Human sleep posture recognition based on millimeter-wave radar. In 2021 Signal Processing Symposium (SPSympo). IEEE, 316--321.
Index Terms
- View-agnostic Human Exercise Cataloging with Single MmWave Radar
Recommendations
SelfPAB: large-scale pre-training on accelerometer data for human activity recognition
AbstractAnnotating accelerometer-based physical activity data remains a challenging task, limiting the creation of robust supervised machine learning models due to the scarcity of large, labeled, free-living human activity recognition (HAR) datasets. ...
Utilizing Self-Supervised Learning for Recognizing Human Activity in Older Adults through Labeling Applications in Real-World Smart Homes
GoodIT '24: Proceedings of the 2024 International Conference on Information Technology for Social GoodDeep learning models have significantly contributed to recognizing older adults’ daily activities for telemonitoring and assistance. However, recognizing human activities in real-world smart homes over the long term presents substantial challenges. ...
Self-supervised Learning for Accelerometer-based Human Activity Recognition: A Survey
Self-supervised learning (SSL) has emerged as a promising alternative to purely supervised learning, since it can learn from labeled and unlabeled data using a pre-train-then-fine-tune strategy, achieving state-of-the-art performances across many ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 September 2024
Published in IMWUT Volume 8, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 419Total Downloads
- Downloads (Last 12 months)419
- Downloads (Last 6 weeks)82
Reflects downloads up to 18 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in