Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Seeing through the Tactile: 3D Human Shape Estimation from Temporal In-Bed Pressure Images

Published: 15 May 2024 Publication History

Abstract

Humans spend about one-third of their lives resting. Reconstructing human dynamics in in-bed scenarios is of considerable significance in sleep studies, bedsore monitoring, and biomedical factor extractions. However, the mainstream human pose and shape estimation methods mainly focus on visual cues, facing serious issues in non-line-of-sight environments. Since in-bed scenarios contain complicated human-environment contact, pressure-sensing bedsheets provide a non-invasive and privacy-preserving approach to capture the pressure distribution on the contact surface, and have shown prospects in many downstream tasks. However, few studies focus on in-bed human mesh recovery. To explore the potential of reconstructing human meshes from the sensed pressure distribution, we first build a high-quality temporal human in-bed pose dataset, TIP, with 152K multi-modality synchronized images. We then propose a label generation pipeline for in-bed scenarios to generate reliable 3D mesh labels with a SMPLify-based optimizer. Finally, we present PIMesh, a simple yet effective temporal human shape estimator to directly generate human meshes from pressure image sequences. We conduct various experiments to evaluate PIMesh's performance, showing that PIMesh archives 79.17mm joint position errors on our TIP dataset. The results demonstrate that the pressure-sensing bedsheet could be a promising alternative for long-term in-bed human shape estimation.

References

[1]
Felix Achilles, Alexandru-Eugen Ichim, Huseyin Coskun, Federico Tombari, Soheyl Noachtar, and Nassir Navab. 2016. Patient MoCap: Human pose estimation under blanket occlusion for hospital monitoring applications. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part 119. Springer, 491--499.
[2]
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. Scape: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers. 408--416.
[3]
Anurag Arnab, Carl Doersch, and Andrew Zisserman. 2019. Exploiting temporal context for 3D human pose estimation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3395--3404.
[4]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
[5]
Joyce Black, Mona Mylene Baharestani, Janet Cuddigan, Becky Dorner, Laura Edsberg, Diane Langemo, Mary Ellen Posthauer, Catherine Ratliff, George Taler, et al. 2007. National Pressure Ulcer Advisory Panel's updated pressure ulcer staging system. Advances in skin & wound care 20, 5 (2007), 269--274.
[6]
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. 2016. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. Springer, 561--578.
[7]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.
[8]
Leslie Casas, Nassir Navab, and Stefanie Demirci. 2019. Patient 3D body pose estimation from pressure imaging. International journal of computer assisted radiology and surgery 14 (2019), 517--524.
[9]
Liqiong Chang, Jiaqi Lu, Ju Wang, Xiaojiang Chen, Dingyi Fang, Zhanyong Tang, Petteri Nurmi, and Zheng Wang. 2018. SleepGuard: Capturing rich sleep information using smartwatch sensing data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--34.
[10]
Kenny Chen, Paolo Gabriel, Abdulwahab Alasfour, Chenghao Gong, Werner K Doyle, Orrin Devinsky, Daniel Friedman, Patricia Dugan, Lucia Melloni, Thomas Thesen, et al. 2018. Patient-specific pose estimation in clinical environments. IEEE journal of translational engineering in health and medicine 6 (2018), 1--11.
[11]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
[12]
Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. 2021. Beyond static features for temporally consistent 3d human pose and shape from a video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1964--1973.
[13]
Yunho Choi, Dong-Hyeok Park, Sungha Lee, Isaac Han, Ecehan Akan, Hyeon-Chang Jeon, Yiyue Luo, SeungJun Kim, Wojciech Matusik, Daniela Rus, et al. 2023. Seamless-walk: natural and comfortable virtual reality locomotion method with a high-resolution tactile sensor. Virtual Reality (2023), 1--15.
[14]
Henry M Clever, Zackory Erickson, Ariel Kapusta, Greg Turk, Karen Liu, and Charles C Kemp. 2020. Bodies at rest: 3d human pose and shape estimation from a pressure image using synthetic data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6215--6224.
[15]
Henry M Clever, Patrick L Grady, Greg Turk, and Charles C Kemp. 2022. Bodypressure-inferring body pose and contact pressure from a depth image. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 1 (2022), 137--153.
[16]
Henry M Clever, Ariel Kapusta, Daehyung Park, Zackory Erickson, Yash Chitalia, and Charles C Kemp. 2018. 3d human pose estimation on a configurable bed from a pressure image. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 54--61.
[17]
Vandad Davoodnia and Ali Etemad. 2023. Human Pose Estimation from Ambiguous Pressure Recordings with Spatio-Temporal Masked Transformers. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1--5.
[18]
Vandad Davoodnia, Saeed Ghorbani, and Ali Etemad. 2021. In-bed pressure-based pose estimation using image space representation learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3965--3969.
[19]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.
[20]
Han Ding, Zhenbin Chen, Cui Zhao, Fei Wang, Ge Wang, Wei Xi, and Jizhong Zhao. 2023. MI-Mesh: 3D Human Mesh Construction by Fusing Image and Millimeter Wave. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 1 (2023), 1--24.
[21]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[22]
Stuart Geman and D. McClure. 1987. Statistical methods for tomographic image reconstruction. Bull Int Stat Inst LII-4 (01 1987).
[23]
John C Gower. 1975. Generalized procrustes analysis. Psychometrika 40 (1975), 33--51.
[24]
Robert Grimm, Sebastian Bauer, Johann Sukkau, Joachim Hornegger, and Günther Greiner. 2012. Markerless estimation of patient orientation, posture and pose using range and pressure imaging: For automatic patient setup and scanner initialization in tomographic imaging. International journal of computer assisted radiology and surgery 7 (2012), 921--929.
[25]
Tatsuya Harada, Tomomasa Sato, and Taketoshi Mori. 2001. Pressure distribution image based human motion tracking system using skeleton and surface integration model. In Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), Vol. 4. IEEE, 3201--3207.
[26]
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. 2019. Resolving 3D human pose ambiguities with 3D scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision. 2282--2292.
[27]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000--16009.
[28]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[29]
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016).
[30]
Zawar Hussain, Quan Z Sheng, Wei Emma Zhang, Jorge Ortiz, and Seyedamin Pouriyeh. 2022. Non-invasive techniques for monitoring different aspects of sleep: A comprehensive review. ACM Transactions on Computing for Healthcare (HEALTH) 3, 2 (2022), 1--26.
[31]
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. YOLO by Ultralytics. https://github.com/ultralytics/ultralytics
[32]
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7122--7131.
[33]
Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. 2019. Learning 3d human dynamics from video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5614--5623.
[34]
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. 2020. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5253--5263.
[35]
Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision. 2252--2261.
[36]
Masamichi Kosuge, Yu Ishihara, and Masaki Takahashi. 2021. Body pressure prediction for pressure ulcer prevention in a bed head elevation operation. Advanced Robotics 35, 3-4 (2021), 181--193.
[37]
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016).
[38]
Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler. 2017. Unite the people: Closing the loop between 3d and 2d human representations. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6050--6059.
[39]
Jiefeng Li, Siyuan Bian, Qi Liu, Jiasheng Tang, Fan Wang, and Cewu Lu. 2023. NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12933--12942.
[40]
Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. 2021. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3383--3393.
[41]
Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. 2022. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference on Computer Vision. Springer, 590--606.
[42]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740--755.
[43]
Jason J Liu, Ming-Chun Huang, Wenyao Xu, and Majid Sarrafzadeh. 2014. Bodypart localization for pressure ulcer prevention. In 2014 36th annual international conference of the ieee engineering in medicine and biology society. IEEE, 766--769.
[44]
Shuangjun Liu, Xiaofei Huang, Nihang Fu, Cheng Li, Zhongnan Su, and Sarah Ostadabbas. 2022. Simultaneously-collected multimodal lying pose dataset: Enabling in-bed human pose monitoring. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 1 (2022), 1106--1118.
[45]
Shuangjun Liu and Sarah Ostadabbas. 2019. Seeing under the cover: A physics guided learning approach for in-bed pose estimation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 236--245.
[46]
Shuangjun Liu, Yu Yin, and Sarah Ostadabbas. 2019. In-bed pose estimation: Deep learning with shallow 16. IEEE journal of translational engineering in health and medicine 7 (2019), 1--12.
[47]
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Transactions on Graphics 34, 6 (2015).
[48]
Yiyue Luo. 2023. Intelligent Textiles for Physical Human-Environment Interactions. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1--5.
[49]
Yiyue Luo, Yunzhu Li, Michael Foshey, Wan Shou, Pratyusha Sharma, Tomás Palacios, Antonio Torralba, and Wojciech Matusik. 2021. Intelligent carpet: Inferring 3d human pose from tactile signals. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11255--11265.
[50]
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. 2019. AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision. 5442--5451.
[51]
Lea Muller, Ahmed AA Osman, Siyu Tang, Chun-Hao P Huang, and Michael J Black. 2021. On self-contact and human pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9990--9999.
[52]
Debashish Nayak, K Srinivasan, Sadasivan Jagdish, Roma Rattan, and Vinayaka S Chatram. 2008. Bedsores:"top to bottom" and "bottom to top". Indian journal of surgery 70 (2008), 161--168.
[53]
Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. 2018. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In 2018 international conference on 3D vision (3DV). IEEE, 484--494.
[54]
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. 2019. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10975--10985.
[55]
M Baran Pouyan, Javad Birjandtalab, Mehrdad Heydarzadeh, Mehrdad Nourani, and Sarah Ostadabbas. 2017. A pressure map dataset for posture and subject analytics. In 2017 IEEE EMBS international conference on biomedical & health informatics (BHI). IEEE, 65--68.
[56]
William H Press and Saul A Teukolsky. 1990. Savitzky-Golay smoothing filters. Computers in Physics 4, 6 (1990), 669--672.
[57]
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. 2021. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision. 11488--11499.
[58]
Marilyn Schallom, Betsy Dykeman, Norma Metheny, John Kirby, and Janet Pierce. 2015. Head-of-bed elevation and early outcomes of gastric reflux, aspiration and pressure ulcers: a feasibility study. American Journal of Critical Care 24, 1 (2015), 57--66.
[59]
Xiaolong Shen, Zongxin Yang, Xiaohan Wang, Jianxin Ma, Chang Zhou, and Yi Yang. 2023. Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8887--8896.
[60]
Mathias Sundholm, Jingyuan Cheng, Bo Zhou, Akash Sethi, and Paul Lukowicz. 2014. Smart-mat: Recognizing and counting gym exercises with low-cost resistive pressure sensing matrix. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. 373--382.
[61]
Matthias Teschner, Stefan Kimmerle, Bruno Heidelberger, Gabriel Zachmann, Laks Raghupathi, Arnulph Fuhrmann, M-P Cani, François Faure, Nadia Magnenat-Thalmann, Wolfgang Strasser, et al. 2005. Collision detection for deformable objects. In Computer graphics forum, Vol. 24. Wiley Online Library, 61--81.
[62]
Shashank Tripathi, Lea Müller, Chun-Hao P Huang, Omid Taheri, Michael J Black, and Dimitrios Tzionas. 2023. 3D human pose estimation via intuitive physics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4713--4725.
[63]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[64]
Wen-Li Wei, Jen-Chun Lin, Tyng-Luh Liu, and Hong-Yuan Mark Liao. 2022. Capturing humans in motion: Temporal-attentive 3D human pose and shape estimation from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13211--13220.
[65]
Ziyu Wu, Quan Wan, Mingjie Zhao, Yi Ke, Yiran Fang, Zhen Liang, Fangting Xie, and Jingyuan Cheng. 2023. MassNet: A Deep Learning Approach for Body Weight Extraction from A Single Pressure Image. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 180--189.
[66]
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems 35 (2022), 38571--38584.
[67]
Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. 2023. Generating holistic 3d human motion from speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 469--480.
[68]
Yu Yin, Joseph P Robinson, and Yun Fu. 2022. Multimodal in-bed pose and shape estimation under the blankets. In Proceedings of the 30th ACM International Conference on Multimedia. 2411--2419.
[69]
Shichao Yue, Yuzhe Yang, Hao Wang, Hariharan Rahul, and Dina Katabi. 2020. BodyCompass: Monitoring sleep posture with wireless signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1--25.
[70]
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. 2021. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11446--11456.
[71]
Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence 22, 11 (2000), 1330--1334.
[72]
Bo Zhou, Jingyuan Cheng, Mathias Sundholm, and Paul Lukowicz. 2014. From smart clothing to smart table cloth: Design and implementation of a large scale, textile pressure matrix sensor. In Architecture of Computing Systems-ARCS 2014: 27th International Conference, Lübeck, Germany, February 25-28, 2014. Proceedings 27. Springer, 159--170.
[73]
Bo Zhou, Sungho Suh, Vitor Fortes Rey, Carlos Andres Velez Altamirano, and Paul Lukowicz. 2022. Quali-mat: Evaluating the quality of execution in body-weight exercises with a pressure sensitive sports mat. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--45.

Index Terms

  1. Seeing through the Tactile: 3D Human Shape Estimation from Temporal In-Bed Pressure Images

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 8, Issue 2
    June 2024
    1330 pages
    EISSN:2474-9567
    DOI:10.1145/3665317
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 May 2024
    Published in IMWUT Volume 8, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. datasets
    2. deep learning
    3. human mesh recovery
    4. pressure-sensing mattress
    5. smart textile

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • ational Natural Science Foundation of China

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 339
      Total Downloads
    • Downloads (Last 12 months)339
    • Downloads (Last 6 weeks)27
    Reflects downloads up to 12 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media