Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Public Access

Winect: 3D Human Pose Tracking for Free-form Activity Using Commodity WiFi

Published: 30 December 2021 Publication History

Abstract

WiFi human sensing has become increasingly attractive in enabling emerging human-computer interaction applications. The corresponding technique has gradually evolved from the classification of multiple activity types to more fine-grained tracking of 3D human poses. However, existing WiFi-based 3D human pose tracking is limited to a set of predefined activities. In this work, we present Winect, a 3D human pose tracking system for free-form activity using commodity WiFi devices. Our system tracks free-form activity by estimating a 3D skeleton pose that consists of a set of joints of the human body. In particular, we combine signal separation and joint movement modeling to achieve free-form activity tracking. Our system first identifies the moving limbs by leveraging the two-dimensional angle of arrival of the signals reflected off the human body and separates the entangled signals for each limb. Then, it tracks each limb and constructs a 3D skeleton of the body by modeling the inherent relationship between the movements of the limb and the corresponding joints. Our evaluation results show that Winect is environment-independent and achieves centimeter-level accuracy for free-form activity tracking under various challenging environments including the none-line-of-sight (NLoS) scenarios.

References

[1]
Frederic Abrard and Yannick Deville. 2005. A time-frequency blind signal separation method applicable to underdetermined mixtures of dependent sources. Signal processing 85, 7 (2005), 1389--1403.
[2]
Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--13.
[3]
Kamran Ali, Alex X Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke recognition using wifi signals. In Proceedings of the 21st annual international conference on mobile computing and networking. 90--102.
[4]
Michael Berouti, Richard Schwartz, and John Makhoul. 1979. Enhancement of speech corrupted by acoustic noise. In ICASSP'79. IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 4. IEEE, 208--211.
[5]
Steven Boll. 1979. Suppression of acoustic noise in speech using spectral subtraction. IEEE Transactions on acoustics, speech, and signal processing 27, 2 (1979), 113--120.
[6]
Donna C Boone and Stanley P Azen. 1979. Normal range of motion of joints in male subjects. JBJS 61, 5 (1979), 756--759.
[7]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.
[8]
André Ebert, Marie Kiermeier, Chadly Marouane, and Claudia Linnhoff-Popien. 2017. Sensx: About sensing and assessment of complex human motion. In 2017 IEEE 14th International Conference on Networking, Sensing and Control (ICNSC). IEEE, 327--332.
[9]
Mahmoud El-Gohary and James McNames. 2012. Shoulder and elbow joint angle tracking with inertial sensors. IEEE Transactions on Biomedical Engineering 59, 9 (2012), 2635--2641.
[10]
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, Vol. 96. 226--231.
[11]
Richard L Gajdosik and Richard W Bohannon. 1987. Clinical measurement of range of motion: review of goniometry emphasizing reliability and validity. Physical therapy 67, 12 (1987), 1867--1872.
[12]
David E Golberg. 1989. Genetic algorithms in search, optimization, and machine learning. Addion wesley 1989, 102 (1989), 36.
[13]
Google. 2021. Tilt Brush. https://www.tiltbrush.com/
[14]
Halfbrick. 2021. Fruit Ninja VR. https://www.halfbrick.com/games/fruit-ninja-vr
[15]
Daniel Halperin, Wenjun Hu, Anmol Sheth, and David Wetherall. 2011. Tool release: Gathering 802.11 n traces with channel state information. ACM SIGCOMM Computer Communication Review 41, 1 (2011), 53--53.
[16]
Ferid Harabi, Ali Gharsallah, and Sylvie Marcos. 2009. Three-dimensional antennas array for the estimation of direction of arrival. IET microwaves, antennas & propagation 3, 5 (2009), 843--849.
[17]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.
[18]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[19]
HTC. 2021. HTC VIVE. https://www.vive.com/us/
[20]
Wenjun Jiang, Hongfei Xue, Chenglin Miao, Shiyang Wang, Sen Lin, Chong Tian, Srinivasan Murali, Haochen Hu, Zhi Sun, and Lu Su. 2020. Towards 3D human pose construction using wifi. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
[21]
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7122--7131.
[22]
Cagdas Karatas, Luyang Liu, Hongyu Li, Jian Liu, Yan Wang, Sheng Tan, Jie Yang, Yingying Chen, Marco Gruteser, and Richard Martin. 2016. Leveraging wearables for steering and driver tracking. In IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. IEEE, 1--9.
[23]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[24]
Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 269--282.
[25]
Chenning Li, Zheng Liu, Yuguang Yao, Zhichao Cao, Mi Zhang, and Yunhao Liu. 2020. Wi-fi see it all: generative adversarial network-augmented versatile wi-fi imaging. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 436--448.
[26]
Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 331--344.
[27]
Xiang Li, Shengjie Li, Daqing Zhang, Jie Xiong, Yasha Wang, and Hong Mei. 2016. Dynamic-music: accurate device-free indoor localization. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 196--207.
[28]
Hongbo Liu, Yu Gan, Jie Yang, Simon Sidhom, Yan Wang, Yingying Chen, and Fan Ye. 2012. Push the limit of WiFi based localization for smartphones. In Proceedings of the 18th annual international conference on Mobile computing and networking. 305--316.
[29]
Jian Liu, Yan Wang, Yingying Chen, Jie Yang, Xu Chen, and Jerry Cheng. 2015. Tracking vital signs during sleep leveraging off-the-shelf wifi. In Proceedings of the 16th ACM International Symposium on Mobile Ad Hoc Networking and Computing. 267--276.
[30]
Yang Liu, Zhenjiang Li, Zhidan Liu, and Kaishun Wu. 2019. Real-time arm skeleton tracking and gesture inference tolerant to missing wearable sensors. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. 287--299.
[31]
Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--14.
[32]
Microsoft. 2021. Kinect 2 for Windows. https://developer.microsoft.com/en-us/windows/kinect/
[33]
Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3225--3234.
[34]
Oculus. 2021. Oculus Rift. https://www.oculus.com/
[35]
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy. 2017. Towards accurate multi-person pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4903--4911.
[36]
Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 2019. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7753--7762.
[37]
Joel C Perry and Jacob Rosen. 2006. Design of a 7 degree-of-freedom upper-limb powered exoskeleton. In The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006. IEEE, 805--810.
[38]
Primex. 2018. Beyond Voice: Gesture-Based Controls for the Smart Home. https://primex.com/beyond-voice-gesture-based-controls-for-the-smart-home/
[39]
Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. 27--38.
[40]
Kun Qian, Chenshu Wu, Yi Zhang, Guidong Zhang, Zheng Yang, and Yunhao Liu. 2018. Widar2. 0: Passive human tracking with a single wi-fi link. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 350--361.
[41]
Kun Qian, Chenshu Wu, Zimu Zhou, Yue Zheng, Zheng Yang, and Yunhao Liu. 2017. Inferring motion direction using commodity wi-fi for interactive exergames. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1961--1972.
[42]
Nancy Berryman Reese and William D Bandy. 2016. Joint range of motion and muscle length testing-E-book. Elsevier Health Sciences.
[43]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015), 91--99.
[44]
Yili Ren, Sheng Tan, Linghan Zhang, Zi Wang, Zhi Wang, and Jie Yang. 2020. Liquid Level Sensing Using Commodity WiFi in a Smart Home Environment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.
[45]
Qaiser Riaz, Guanhong Tao, Björn Krüger, and Andreas Weber. 2015. Motion reconstruction using very few accelerometers and ground contacts. Graphical Models 79 (2015), 23--38.
[46]
V David Sánchez A. 2002. Frontiers of research in BSS/ICA. Neurocomputing 49, 1--4 (2002), 7--23.
[47]
Ralph Schmidt. 1986. Multiple emitter location and signal parameter estimation. IEEE transactions on antennas and propagation 34, 3 (1986), 276--280.
[48]
Chenguang Shen, Bo-Jhang Ho, and Mani Srivastava. 2017. Milift: Efficient smartwatch-based workout tracking using automatic segmentation. IEEE Transactions on Mobile Computing 17, 7 (2017), 1609--1622.
[49]
Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 429--444.
[50]
Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a smartwatch and i can track my user's arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. 85--96.
[51]
Leonid Sigal, Alexandru O Balan, and Michael J Black. 2010. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International journal of computer vision 87, 1--2 (2010), 4.
[52]
Sheng Tan and Jie Yang. 2016. WiFinger: Leveraging commodity WiFi for fine-grained finger gesture recognition. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. 201--210.
[53]
Sheng Tan, Jie Yang, and Yingying Chen. 2020. Enabling Fine-grained Finger Gesture Recognition on Commodity WiFi Devices. IEEE Transactions on Mobile Computing (2020).
[54]
Sheng Tan, Linghan Zhang, Zi Wang, and Jie Yang. 2019. MultiTrack: Multi-user tracking and activity recognition using commodity WiFi. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.
[55]
Sheng Tan, Linghan Zhang, and Jie Yang. 2018. Sensing fruit ripeness using wireless signals. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1--9.
[56]
Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bernd Eberhardt. 2011. Motion reconstruction using sparse accelerometer data. ACM Transactions on Graphics (ToG) 30, 3 (2011), 1--12.
[57]
Kevin C. Tofel. 2019. We are heading towards a quieter, gesture-based smart home. https://staceyoniot.com/shhh-were-heading-towards-a-quieter-gesture-based-smart-home/
[58]
Ultraleap. 2021. Leap Motion Controller. https://www.ultraleap.com/product/leap-motion-controller/
[59]
Visualise. 2021. Virtual Reality in Healthcare. https://visualise.com/virtual-reality/virtual-reality-healthcare
[60]
Binny Vyas. 2021. Top 5 use cases of extended reality in the healthcare sector. https://www.softwebsolutions.com/resources/extended-reality-in-healthcare-sector.html
[61]
Chen Wang, Xiuyuan Zheng, Yingying Chen, and Jie Yang. 2016. Locating rogue access point using fine-grained channel information. IEEE Transactions on Mobile Computing 16, 9 (2016), 2560--2573.
[62]
Fei Wang, Sanping Zhou, Stanislav Panev, Jinsong Han, and Dong Huang. 2019. Person-in-WiFi: Fine-grained person perception using WiFi. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5452--5461.
[63]
Wei Wang, Alex X Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2015. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of the 21st annual international conference on mobile computing and networking. 65--76.
[64]
Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of the 20th annual international conference on Mobile computing and networking. 617--628.
[65]
Dan Wu, Ruiyang Gao, Youwei Zeng, Jinyi Liu, Leye Wang, Tao Gu, and Daqing Zhang. 2020. Fingerdraw: Sub-wavelength level finger motion tracking with WiFi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--27.
[66]
Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging multi-dimensionality for passive indoor Wi-Fi tracking. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.
[67]
Yaxiong Xie, Yanbo Zhang, Jansen Christian Liando, and Mo Li. 2018. Swan: Stitched wi-fi antennas. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 51--66.
[68]
Jie Yang and Yingying Chen. 2009. Indoor localization using improved rss-based lateration methods. In GLOBECOM 2009 IEEE Global Telecommunications Conference. IEEE, 1--6.
[69]
Nan Yu, Wei Wang, Alex X Liu, and Lingtao Kong. 2018. Qgesture: Quantifying gesture distance and direction with wifi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--23.
[70]
Vicente Zarzoso and Pierre Comon. 2009. Robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size. IEEE Transactions on neural networks 21, 2 (2009), 248--261.
[71]
Youwei Zeng, Dan Wu, Jie Xiong, Jinyi Liu, Zhaopeng Liu, and Daqing Zhang. 2020. MultiSense: Enabling multi-person respiration sensing with commodity wifi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1--29.
[72]
Youwei Zeng, Dan Wu, Jie Xiong, Enze Yi, Ruiyang Gao, and Daqing Zhang. 2019. FarSense: Pushing the range limit of WiFi-based respiration sensing with CSI ratio of two antennas. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--26.
[73]
Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia 19, 2 (2012), 4--10.
[74]
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356--7365.
[75]
Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-based 3D skeletons. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 267--281.
[76]
Xiuyuan Zheng, Hongbo Liu, Jie Yang, Yingying Chen, Richard P Martin, and Xiaoyan Li. 2013. A study of localization accuracy using multiple frequencies and powers. IEEE Transactions on Parallel and Distributed Systems 25, 8 (2013), 1955--1965.

Cited By

View all
  • (2024)SpaceBeat: Identity-aware Multi-person Vital Signs Monitoring Using Commodity WiFiProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785908:3(1-23)Online publication date: 9-Sep-2024
  • (2024)NeuroCamTags: Long-Range, Battery-free, Wireless Sensing with Neuromorphic CamerasProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785298:3(1-25)Online publication date: 9-Sep-2024
  • (2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
  • Show More Cited By

Index Terms

  1. Winect: 3D Human Pose Tracking for Free-form Activity Using Commodity WiFi

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 5, Issue 4
    Dec 2021
    1307 pages
    EISSN:2474-9567
    DOI:10.1145/3508492
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 December 2021
    Published in IMWUT Volume 5, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3D Human Skeleton
    2. Channel State Information (CSI)
    3. Commodity WiFi
    4. Free-form Activity
    5. Human Pose Estimation
    6. WiFi Sensing

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)549
    • Downloads (Last 6 weeks)64
    Reflects downloads up to 15 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)SpaceBeat: Identity-aware Multi-person Vital Signs Monitoring Using Commodity WiFiProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785908:3(1-23)Online publication date: 9-Sep-2024
    • (2024)NeuroCamTags: Long-Range, Battery-free, Wireless Sensing with Neuromorphic CamerasProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785298:3(1-25)Online publication date: 9-Sep-2024
    • (2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
    • (2024)EVLeSen: In-Vehicle Sensing with EV-Leaked SignalProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649389(679-693)Online publication date: 29-May-2024
    • (2024)WaffleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314587:4(1-29)Online publication date: 12-Jan-2024
    • (2024)LoCalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314367:4(1-27)Online publication date: 12-Jan-2024
    • (2024)PyroSenseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314357:4(1-32)Online publication date: 12-Jan-2024
    • (2024)CAvatarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314247:4(1-24)Online publication date: 12-Jan-2024
    • (2024)Wall MattersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314177:4(1-22)Online publication date: 12-Jan-2024
    • (2024)WiFi-Based Human Sensing With Deep Learning: Recent Advances, Challenges, and OpportunitiesIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34115295(3595-3623)Online publication date: 2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media