Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Public Access

GoPose: 3D Human Pose Estimation Using WiFi

Published: 07 July 2022 Publication History
  • Get Citation Alerts
  • Abstract

    This paper presents GoPose, a 3D skeleton-based human pose estimation system that uses WiFi devices at home. Our system leverages the WiFi signals reflected off the human body for 3D pose estimation. In contrast to prior systems that need specialized hardware or dedicated sensors, our system does not require a user to wear or carry any sensors and can reuse the WiFi devices that already exist in a home environment for mass adoption. To realize such a system, we leverage the 2D AoA spectrum of the signals reflected from the human body and the deep learning techniques. In particular, the 2D AoA spectrum is proposed to locate different parts of the human body as well as to enable environment-independent pose estimation. Deep learning is incorporated to model the complex relationship between the 2D AoA spectrums and the 3D skeletons of the human body for pose tracking. Our evaluation results show GoPose achieves around 4.7cm of accuracy under various scenarios including tracking unseen activities and under NLoS scenarios.

    References

    [1]
    Karan Ahuja, Sven Mayer, Mayank Goel, and Chris Harrison. 2021. Pose-on-the-Go: Approximating User Pose with Smartphone Sensor Fusion and Inverse Kinematics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--12.
    [2]
    Teo Babic, Florian Perteneder, Harald Reiterer, and Michael Haller. 2020. Simo: Interactions with distant displays by smartphones with simultaneous face and world tracking. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1--12.
    [3]
    Alan Bränzel, Christian Holz, Daniel Hoffmann, Dominik Schmidt, Marius Knaust, Patrick Lühne, René Meusel, Stephan Richter, and Patrick Baudisch. 2013. GravitySpace: tracking users and their poses in a smart room using a pressure-sensing floor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 725--734.
    [4]
    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.
    [5]
    Ke-Yu Chen, Shwetak N Patel, and Sean Keller. 2016. Finexus: Tracking precise motions of multiple fingertips using magnetic sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1504--1514.
    [6]
    Mahmoud El-Gohary and James McNames. 2012. Shoulder and elbow joint angle tracking with inertial sensors. IEEE Transactions on Biomedical Engineering 59, 9 (2012), 2635--2641.
    [7]
    Halfbrick. 2021. Fruit Ninja VR. https://www.halfbrick.com/games/fruit-ninja-vr
    [8]
    Daniel Halperin, Wenjun Hu, Anmol Sheth, and David Wetherall. 2011. Tool release: Gathering 802.11 n traces with channel state information. ACM SIGCOMM Computer Communication Review 41, 1 (2011), 53--53.
    [9]
    Ferid Harabi, Ali Gharsallah, and Sylvie Marcos. 2009. Three-dimensional antennas array for the estimation of direction of arrival. IET microwaves, antennas & propagation 3, 5 (2009), 843--849.
    [10]
    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.
    [11]
    Dun-Yu Hsiao, Min Sun, Christy Ballweber, Seth Cooper, and Zoran Popović. 2016. Proactive sensing for improving hand pose estimation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2348--2352.
    [12]
    Stephen S Intille, Ling Bao, Emmanuel Munguia Tapia, and John Rondoni. 2004. Acquiring in situ training data for context-aware ubiquitous computing applications. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1--8.
    [13]
    Wenjun Jiang, Hongfei Xue, Chenglin Miao, Shiyang Wang, Sen Lin, Chong Tian, Srinivasan Murali, Haochen Hu, Zhi Sun, and Lu Su. 2020. Towards 3D human pose construction using wifi. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
    [14]
    Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7122--7131.
    [15]
    Cagdas Karatas, Luyang Liu, Hongyu Li, Jian Liu, Yan Wang, Sheng Tan, Jie Yang, Yingying Chen, Marco Gruteser, and Richard Martin. 2016. Leveraging wearables for steering and driver tracking. In IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. IEEE, 1--9.
    [16]
    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
    [17]
    Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 269--282.
    [18]
    Jooshik Lee, Iickho Song, Hyoungmoon Kwon, and Sung Ro Lee. 2003. Low-complexity estimation of 2D DOA for coherently distributed sources. Signal processing 83, 8 (2003), 1789--1802.
    [19]
    Hanchuan Li, Peijin Zhang, Samer Al Moubayed, Shwetak N Patel, and Alanson P Sample. 2016. Id-match: A hybrid computer vision and rfid system for recognizing individuals in groups. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4933--4944.
    [20]
    Jianfeng Li, Penghui Ma, Xiaofei Zhang, and Gaofeng Zhao. 2020. Improved DFT algorithm for 2D DOA estimation based on 1D nested array motion. IEEE Communications Letters 24, 9 (2020), 1953--1956.
    [21]
    Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 331--344.
    [22]
    Xiang Li, Shengjie Li, Daqing Zhang, Jie Xiong, Yasha Wang, and Hong Mei. 2016. Dynamic-music: accurate device-free indoor localization. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 196--207.
    [23]
    Hongbo Liu, Yu Gan, Jie Yang, Simon Sidhom, Yan Wang, Yingying Chen, and Fan Ye. 2012. Push the limit of WiFi based localization for smartphones. In Proceedings of the 18th annual international conference on Mobile computing and networking. 305--316.
    [24]
    Jian Liu, Yingying Chen, Yan Wang, Xu Chen, Jerry Cheng, and Jie Yang. 2018. Monitoring vital signs and postures during sleep using WiFi signals. IEEE Internet of Things Journal 5, 3 (2018), 2071--2084.
    [25]
    Jian Liu, Yan Wang, Yingying Chen, Jie Yang, Xu Chen, and Jerry Cheng. 2015. Tracking vital signs during sleep leveraging off-the-shelf wifi. In Proceedings of the 16th ACM International Symposium on Mobile Ad Hoc Networking and Computing. 267--276.
    [26]
    Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--14.
    [27]
    Microsoft. 2021. Kinect 2 for Windows. https://developer.microsoft.com/en-us/windows/kinect/
    [28]
    Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3225--3234.
    [29]
    George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy. 2017. Towards accurate multi-person pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4903--4911.
    [30]
    Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 2019. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7753--7762.
    [31]
    Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. 27--38.
    [32]
    Kun Qian, Chenshu Wu, Zimu Zhou, Yue Zheng, Zheng Yang, and Yunhao Liu. 2017. Inferring motion direction using commodity wi-fi for interactive exergames. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1961--1972.
    [33]
    Yanzhi Ren, Yingying Chen, Mooi Choo Chuah, and Jie Yang. 2013. Smartphone based user verification leveraging gait recognition for mobile healthcare systems. In 2013 IEEE international conference on sensing, communications and networking (SECON). IEEE, 149--157.
    [34]
    Yanzhi Ren, Yingying Chen, Mooi Choo Chuah, and Jie Yang. 2014. User verification leveraging gait recognition for smartphone enabled mobile healthcare systems. IEEE Transactions on Mobile Computing 14, 9 (2014), 1961--1974.
    [35]
    Yili Ren, Sheng Tan, Linghan Zhang, Zi Wang, Zhi Wang, and Jie Yang. 2020. Liquid Level Sensing Using Commodity WiFi in a Smart Home Environment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.
    [36]
    Yili Ren, Zi Wang, Sheng Tan, Yingying Chen, and Jie Yang. 2021. Tracking free-form activity using wifi signals. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 816--818.
    [37]
    Yili Ren, Zi Wang, Sheng Tan, Yingying Chen, and Jie Yang. 2021. Winect: 3D Human Pose Tracking for Free-form Activity Using Commodity WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--29.
    [38]
    Ralph Schmidt. 1986. Multiple emitter location and signal parameter estimation. IEEE transactions on antennas and propagation 34, 3 (1986), 276--280.
    [39]
    Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a smartwatch and i can track my user's arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. 85--96.
    [40]
    Leonid Sigal, Alexandru O Balan, and Michael J Black. 2010. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International journal of computer vision 87, 1-2 (2010), 4.
    [41]
    Sheng Tan, Yili Ren, Jie Yang, and Yingying Chen. 2022. Commodity WiFi Sensing in 10 Years: Status, Challenges, and Opportunities. IEEE Internet of Things Journal (2022).
    [42]
    Sheng Tan and Jie Yang. 2016. WiFinger: Leveraging commodity WiFi for fine-grained finger gesture recognition. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. 201--210.
    [43]
    Sheng Tan, Jie Yang, and Yingying Chen. 2020. Enabling fine-grained finger gesture recognition on commodity wifi devices. IEEE Transactions on Mobile Computing (2020).
    [44]
    Sheng Tan, Linghan Zhang, Zi Wang, and Jie Yang. 2019. MultiTrack: Multi-user tracking and activity recognition using commodity WiFi. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.
    [45]
    Sheng Tan, Linghan Zhang, and Jie Yang. 2018. Sensing fruit ripeness using wireless signals. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1--9.
    [46]
    Maksym Tatariants. 2021. Human Pose Estimation Technology 2021 Guide. https://mobidev.biz/blog/human-pose-estimation-ai-personal-fitness-coach
    [47]
    Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bernd Eberhardt. 2011. Motion reconstruction using sparse accelerometer data. ACM Transactions on Graphics (ToG) 30, 3 (2011), 1--12.
    [48]
    Ultraleap. 2021. Leap Motion Controller. https://www.ultraleap.com/product/leap-motion-controller/
    [49]
    Elise Klæbo Vonstad, Xiaomeng Su, Beatrix Vereijken, Kerstin Bach, and Jan Harald Nilsen. 2020. Comparison of a Deep Learning-Based Pose Estimation System to Marker-Based and Kinect Systems in Exergaming for Balance Training. Sensors 20, 23 (2020), 6940.
    [50]
    Chuyu Wang, Jian Liu, Yingying Chen, Lei Xie, Hong Bo Liu, and Sanclu Lu. 2018. RF-kinect: A wearable RFID-based approach towards 3D body movement tracking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--28.
    [51]
    Fei Wang, Sanping Zhou, Stanislav Panev, Jinsong Han, and Dong Huang. 2019. Person-in-WiFi: Fine-grained person perception using WiFi. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5452--5461.
    [52]
    Xianpeng Wang, Mengxing Huang, and Liangtian Wan. 2021. Joint 2D-DOD and 2D-DOA estimation for coprime EMVS-MIMO radar. Circuits, Systems, and Signal Processing 40, 6 (2021), 2950--2966.
    [53]
    Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of the 20th annual international conference on Mobile computing and networking. 617--628.
    [54]
    Yinsheng Wei and Xiaojiang Guo. 2014. Pair-matching method by signal covariance matrices for 2D-DOA estimation. IEEE Antennas and Wireless Propagation Letters 13 (2014), 1199--1202.
    [55]
    Erwin Wu, Ye Yuan, Hui-Shyong Yeo, Aaron Quigley, Hideki Koike, and Kris M Kitani. 2020. Back-Hand-Pose: 3D Hand Pose Estimation for a Wrist-worn Camera via Dorsum Deformation Network. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 1147--1160.
    [56]
    Jie Yang and Yingying Chen. 2008. A theoretical analysis of wireless localization using RF-based fingerprint matching. In 2008 IEEE International Symposium on Parallel and Distributed Processing. IEEE, 1--6.
    [57]
    Jie Yang and Yingying Chen. 2009. Indoor localization using improved rss-based lateration methods. In GLOBECOM 2009 IEEE Global Telecommunications Conference. IEEE, 1--6.
    [58]
    Youwei Zeng, Dan Wu, Jie Xiong, Jinyi Liu, Zhaopeng Liu, and Daqing Zhang. 2020. MultiSense: Enabling multi-person respiration sensing with commodity wifi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1--29.
    [59]
    Feng Zhang, Chenshu Wu, Beibei Wang, and KJ Ray Liu. 2020. mmEye: Super-resolution millimeter wave imaging. IEEE Internet of Things Journal 8, 8 (2020), 6995--7008.
    [60]
    Yang Zhang, Chouchang Yang, Scott E Hudson, Chris Harrison, and Alanson Sample. 2018. Wall++ room-scale interactive and context-aware sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--15.
    [61]
    Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia 19, 2 (2012), 4--10.
    [62]
    Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356--7365.
    [63]
    Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-based 3D skeletons. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 267--281.
    [64]
    Anastasiya Zharovskikh. 2020. Pose Estimation to Empower Your Business. https://indatalabs.com/blog/pose-estimation
    [65]
    Xiuyuan Zheng, Hongbo Liu, Jie Yang, Yingying Chen, Richard P Martin, and Xiaoyan Li. 2013. A study of localization accuracy using multiple frequencies and powers. IEEE Transactions on Parallel and Distributed Systems 25, 8 (2013), 1955--1965.

    Cited By

    View all
    • (2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/3677122Online publication date: 16-Jul-2024
    • (2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
    • (2024)WiProfile: Unlocking Diffraction Effects for Sub-Centimeter Target Profiling Using Commodity WiFi DevicesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649355(185-199)Online publication date: 29-May-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 2
    July 2022
    1551 pages
    EISSN:2474-9567
    DOI:10.1145/3547347
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2022
    Published in IMWUT Volume 6, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Channel State Information (CSI)
    2. Deep Learning
    3. Human Pose Estimation
    4. WiFi Sensing

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1,128
    • Downloads (Last 6 weeks)93

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/3677122Online publication date: 16-Jul-2024
    • (2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
    • (2024)WiProfile: Unlocking Diffraction Effects for Sub-Centimeter Target Profiling Using Commodity WiFi DevicesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649355(185-199)Online publication date: 29-May-2024
    • (2024)MobiRFPose: Portable RF-Based 3D Human Pose CameraIEEE Transactions on Multimedia10.1109/TMM.2023.331497926(3715-3727)Online publication date: 1-Jan-2024
    • (2024)Fall-Attention: An Attention-Based Fall Detection Method for Adjoint ActivitiesIEEE Transactions on Mobile Computing10.1109/TMC.2023.334412523:7(7895-7909)Online publication date: Jul-2024
    • (2024)WiFi-Based Human Sensing With Deep Learning: Recent Advances, Challenges, and OpportunitiesIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34115295(3595-3623)Online publication date: 2024
    • (2024)MDST: 2-D Human Pose Estimation for SISO UWB Radar Based on Micro-Doppler Signature via Cascade and Parallel Swin TransformerIEEE Sensors Journal10.1109/JSEN.2024.340186124:13(21730-21749)Online publication date: 1-Jul-2024
    • (2024)Deep learning for 3D human pose estimation and mesh recovery: A surveyNeurocomputing10.1016/j.neucom.2024.128049596(128049)Online publication date: Sep-2024
    • (2024)Wireless sensing applications with Wi-Fi Channel State Information, preprocessing techniques, and detection algorithms: A surveyComputer Communications10.1016/j.comcom.2024.06.011224(254-274)Online publication date: Aug-2024
    • (2023)Person re-identification in 3D spaceProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620529(5217-5234)Online publication date: 9-Aug-2023
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media