research-article

MI-Mesh: 3D Human Mesh Construction by Fusing Image and Millimeter Wave

Authors:

Jizhong ZhaoAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 7, Issue 1

Article No.: 10, Pages 1 - 24

https://doi.org/10.1145/3580861

Published: 28 March 2023 Publication History

Abstract

Estimating 3D human mesh is appealing for various application scenarios. Current mainstream solution predicts the meshes either from the image or the human reflected RF-signals. In this paper, instead of investigating which approach is better, we propose to design a multi-modality fusion framework, namely MI-Mesh, which estimates 3D meshes by fusing image and mmWave. To realize this, we design a deep neural network model. It first automatically correlate mmWave point clouds to certain human joints and extracts useful fused features from two modalities. Then, the features are refined by predicting 2D joints and silhouette. Finally, we regress pose and shape parameters and feed them to SMPL model to generate the 3D human meshes. We build a prototype on commercial mmWave radar and camera. The experimental results demonstrate that with the integration of multi-modality strengths, MI-Mesh can effectively recover human meshes on dynamic motions and across different conditions.

References

[1]

[n.d.]. AWR1443 Single-Chip 76-GHz to 81-GHz Automotive Radar Sensor Evaluation Module. https://www.ti.com/tool/AWR1443BOOST? keyMatch=AWR1443BOOST. Texas Instruments, 2020.

[2]

[n.d.]. Real-Time Data-Capture Adapter for Radar Sensing Evaluation Module. https://www.ti.com/tool/DCA1000EVM. Texas Instruments, 2019.

[3]

Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the Human Figure through a Wall. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--13.

Digital Library

[4]

Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D Tracking via Body Radio Reflections. In Proceedings of USENIX NSDI.

Digital Library

[5]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In Proceedings of IEEE/CVF CVPR.

[6]

Ching-Hang Chen and Deva Ramanan. 2017. 3D Human Pose Estimation = 2D Pose Estimation + Matching. In Proceedings of IEEE/CVF CVPR.

[7]

Zhe Chen, Tianyue Zheng, Chao Cai, and Jun Luo. 2021. MoVi-Fi: Motion-Robust Vital Signs Waveform Recovery via Deep Interpreted RF Sensing. In Proceedings of ACM MobiCom.

Digital Library

[8]

Endri Dibra, Himanshu Jain, Cengiz Oztireli, Remo Ziegler, and Markus Gross. 2017. Human Shape from Silhouettes using Generative HKS Descriptors and Cross-Modal Neural Networks. In Proceedings of IEEE/CVF CVPR.

[9]

Han Ding, Chen Qian, Jinsong Han, Ge Wang, Wei Xi, Kun Zhao, and Jizhong Zhao. 2017. RFiPad: Enabling Cost-Efficient and Device-Free In-Air Handwriting using Passive Tags. In Proceedings of IEEE ICDCS.

[10]

Han Ding, Longfei Shangguan, Zheng Yang, Jinsong Han, Zimu Zhou, Panlong Yang, Wei Xi, and Jizhong Zhao. 2015. Femo: A Platform for Free-Weight Exercise Monitoring with RFIDs. In Proceedings of ACM SenSys.

Digital Library

[11]

John C Gower. 1975. Generalized Procrustes Analysis. Psychometrika 40, 1 (1975), 33--51.

[12]

Xiaonan Guo, Jian Liu, Cong Shi, Hongbo Liu, Yingying Chen, and Mooi Choo Chuah. 2018. Device-Free Personalized Fitness Assistant using WiFi. (2018).

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity Mappings in Deep Residual Networks. In Proceedings of Springer ECCV.

[14]

Wenjun Jiang, Hongfei Xue, Chenglin Miao, Shiyang Wang, Sen Lin, Chong Tian, Srinivasan Murali, Haochen Hu, Zhi Sun, and Lu Su. 2020. Towards 3D Human Pose Construction using WiFi. In Proceedings of ACM MobiCom.

Digital Library

[15]

Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019. Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop. In Proceedings of IEEE/CVF ICCV.

[16]

Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. 2019. Convolutional Mesh Regression for Single-Image Human Shape Reconstruction. In Proceedings of IEEE/CVF CVPR.

[17]

Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. 2021. Probabilistic Modeling for Human Mesh Recovery. In Proceedings of IEEE/CVF ICCV.

[18]

Dong Li, Jialin Liu, Sunghoon Ivan Lee, and Jie Xiong. 2022. LASense: Pushing the Limits of Fine-grained Activity Sensing Using Acoustic Signals. Proceedings of ACM UbiComp (2022).

Digital Library

[19]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM transactions on graphics (TOG) 34, 6 (2015), 1--16.

Digital Library

[20]

Sameera Palipana, Dariush Salami, Luis A Leiva, and Stephan Sigg. 2021. Pantomime: Mid-Air Gesture Recognition with Sparse Millimeter-Wave Radar Point Clouds. Proceedings of ACM UbiComp (2021).

Digital Library

[21]

Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. 2018. Learning to Estimate 3D Human Pose and Shape from A Single Color Image. In Proceedings of IEEE/CVF CVPR.

[22]

Albert Pumarola, Jordi Sanchez-Riera, Gary Choi, Alberto Sanfeliu, and Francesc Moreno-Noguer. 2019. 3DPeople: Modeling the Geometry of Dressed Humans. In Proceedings of IEEE/CVF ICCV.

[23]

Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. 2017. Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation. In Proceedings of IEEE/CVF CVPR.

[24]

Charles Qi, Li Yi, Hao Su, and Leonidas Guibas. 2017. Pointnet++: Deep Hierarchical Feature Learning on Point Sets in A Metric Space. Advances in Neural Information Processing Systems 30 (2017).

[25]

Yili Ren, Zi Wang, Sheng Tan, Yingying Chen, and Jie Yang. 2021. Winect: 3D Human Pose Tracking for Free-Form Activity using Commodity WiFi. In Proceedings of ACM UbiComp/IMWUT.

Digital Library

[26]

Yili Ren, Zi Wang, Yichao Wang, Sheng Tan, Yingying Chen, and Jie Yang. 2022. GoPose: 3D Human Pose Estimation Using WiFi. In Proceedings of ACM UbiComp/IMWUT.

Digital Library

[27]

Longfei Shangguan, Zimu Zhou, and Kyle Jamieson. 2017. Enabling Gesture-based Interactions with Objects. In Proceedings of ACM MobiSys.

Digital Library

[28]

Yonglong Tian, Guang-He Lee, Hao He, Chen-Yu Hsu, and Dina Katabi. 2018. RF-based Fall Monitoring using Convolutional Neural Networks. In Proceedings of ACM UbiComp/IMWUT.

Digital Library

[29]

Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. 2014. Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation. In Proceedings of NeurIPS.

[30]

Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. 2017. Learning from Synthetic Humans. In Proceedings of IEEE/CVFCVPR.

[31]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. Advances in Neural Information Processing Systems 30 (2017).

[32]

Yuxi Wang, Kaishun Wu, and Lionel M Ni. 2016. WiFall: Device-Free Fall Detection by Wireless Networks. IEEE Transactions on Mobile Computing 16, 2 (2016), 581--594.

Digital Library

[33]

Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging Multi-dimensionality for Passive Indoor Wi-Fi Tracking. In Proceedings of ACM MobiCom.

Digital Library

[34]

Tianhan Xu and Wataru Takano. 2021. Graph Stacked Hourglass Networks for 3D Human Pose Estimation. In Proceedings of IEEE/CVF CVPR.

[35]

Hongfei Xue, Yan Ju, Chenglin Miao, Yijiang Wang, Shiyang Wang, Aidong Zhang, and Lu Su. 2021. mmMesh: Towards 3D Real-Time Dynamic Human Mesh Construction using Millimeter-Wave. In Proceedings of ACM MobiSys.

Digital Library

[36]

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. 2016. Lift: Learned Invariant Feature Transform. In Proceedings of Springer ECCV.

[37]

Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang. 2020. 3d Human Mesh Regression with Dense Correspondence. In Proceedings of IEEE/CVF CVPR.

[38]

Jin Zhang, Fuxiang Wu, Bo Wei, Qieshi Zhang, Hui Huang, Syed W Shah, and Jun Cheng. 2020. Data Augmentation and Dense-LSTM for Human Activity Recognition using WiFi Signal. IEEE Internet of Things Journal 8, 6 (2020), 4628--4641.

[39]

Mingmin Zhao, Yingcheng Liu, Aniruddh Raghu, Tianhong Li, Hang Zhao, Antonio Torralba, and Dina Katabi. 2019. Through-Wall Human Mesh Recovery using Radio Signals. In Proceedings of IEEE/CVF ICCV.

[40]

Yue Zheng, Yi Zhang, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang. 2019. Zero-Effort Cross-Domain Gesture Recognition with WiFi. In Proceedings of ACM MobiSys.

Digital Library

[41]

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019. On the Continuity of Rotation Representations in Neural Networks. In Proceedings of IEEE/CVF CVPR.

Cited By

Mei LLiu RYin ZZhao QJiang WWang SWang SLu KHe T(2024)mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual RealityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997728:4(1-29)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699772
Li YWang CXie LJin QFan LNing JLu S(2024)Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997398:4(1-26)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699739
Zhuang YZhao YLiu J(2024)Data-driven Soil Moisture Sensing with mmWave RadarProceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3671127.3698787(287-290)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3671127.3698787
Show More Cited By

Index Terms

MI-Mesh: 3D Human Mesh Construction by Fusing Image and Millimeter Wave
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

A wifi vision-based 3D human mesh reconstruction
MobiCom '22: Proceedings of the 28th Annual International Conference on Mobile Computing And Networking

In this work, we present, Wi-Mesh, a WiFi vision-based 3D human mesh construction system. Our system leverages the advances of WiFi to visualize the shape and deformations of the human body for 3D mesh construction. In particular, it estimates the two-...
Wi-Mesh: A WiFi Vision-Based Approach for 3D Human Mesh Construction
SenSys '22: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems

In this paper, we present, Wi-Mesh, a WiFi vision-based 3D human mesh construction system. Our system leverages the advances of WiFi to visualize the shape and deformations of the human body for 3D mesh construction. In particular, it leverages multiple ...
Mesh simplification algorithm based on n-edge mesh collapse
ICAT'06: Proceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence

This paper presents a method for dividing the triangle mesh into n-edge mesh and puts forward a new mesh simplification algorithm based on n-edge mesh collapse. An n-edge mesh can be in the form of an edge, a triangle or a quadrangle, it depends on the ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 7, Issue 1

March 2023

1243 pages

EISSN:2474-9567

DOI:10.1145/3589760

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 March 2023

Published in IMWUT Volume 7, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
728
Total Downloads

Downloads (Last 12 months)291
Downloads (Last 6 weeks)24

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mei LLiu RYin ZZhao QJiang WWang SWang SLu KHe T(2024)mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual RealityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997728:4(1-29)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699772
Li YWang CXie LJin QFan LNing JLu S(2024)Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997398:4(1-26)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699739
Zhuang YZhao YLiu J(2024)Data-driven Soil Moisture Sensing with mmWave RadarProceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3671127.3698787(287-290)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3671127.3698787
Wu ZXie FFang YLiang ZWan QXiong YCai X(2024)Seeing through the TactileProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596128:2(1-39)Online publication date: 15-May-2024
https://doi.org/10.1145/3659612
Liu CDong ZHuang LYan WWang XFang DChen X(2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643512
Lin CWang ZXiong JLi FWu GGanesan DLane NShi W(2024)Fine-grained Textile Moisture Sensing with Commodity UWBProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690679(1074-1088)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1145/3636534.3690679
Zhang XZhang DXie YWu DLi YZhang D(2024)WaffleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314587:4(1-29)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631458
Wang ZGuo YRen ZSong WSun ZChen CGuo BYu Z(2024)LiqDetectorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314437:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631443
Zhang DZhang XXie YZhang FWang XLi YZhang D(2024)LoCalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314367:4(1-27)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631436
Choi JKang KKim K(2024)RF-Vital: Radio-Based Contactless Respiration Monitoring for a Moving IndividualIEEE Internet of Things Journal10.1109/JIOT.2023.332942711:8(13137-13151)Online publication date: 15-Apr-2024
https://doi.org/10.1109/JIOT.2023.3329427
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents