research-article

Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks

Authors:

Jonathan Tompson,

Ken PerlinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 5

Article No.: 169, Pages 1 - 10

https://doi.org/10.1145/2629500

Published: 23 September 2014 Publication History

Abstract

We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and finally an inverse kinematics stage for stable real-time pose recovery. As one possible application of this pipeline, we show state-of-the-art results for real-time puppeteering of a skinned hand-model.

Supplementary Material

MP4 File (a169-sidebyside.mp4)

Download
13.80 MB

References

[1]

3GEAR. 2014. 3gear sytems hand-tracking development platform. http://www.threegear.com/.

[2]

B. Allen, B. Curless, and Z. Popovic. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22, 3, 587--594.

Digital Library

[3]

L. Ballan, A. Taneja, J. Gall, L. Van Gool, and M. Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Proceedings of the 12^th European Conference on Computer Vision. 640--653.

Digital Library

[4]

Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222--1239.

Digital Library

[5]

D. A. Butler, S. Izadi, O. Hilliges, D. Molyneaux, S. Hodges, and D. Kim. 2012. Shake'n'sense: Reducing interference for overlapping structured light depth cameras. In Proceedings of the ACM Annual Conference on Human Factors in Computing Systems. 1933--1936.

Digital Library

[6]

R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A matlab-like environment for machine learning. http://ronan.collobert.com/pub/matos/2011_torch7_nipsw.pdf.

[7]

C. Couprie, C. Farabet, L. Najman, and Y. Lecun. 2013. Indoor semantic segmentation using depth information. In Proceedings of the International Conference on Learning Representations.

[8]

A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly. 2007. Vision-based hand pose estimation: A review. Comput. Vis. Image Understand. 108, 1--2, 52--73.

Digital Library

[9]

C. Farabet, C. Couprie, L. Najman, and Y. Lecun. 2013. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8, 1915--1929.

Digital Library

[10]

B. K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer. 4, 4, 629--642.

[11]

K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. Lecun. 2009. What is the best multi-stage architecture for object recognition&quest; In Proceedings of the 12^th IEEE International Conference on Computer Vision. 2146--2153.

[12]

M. Jiu, C. Wolf, G. W. Taylor, and A. Baskurt. 2013. Human body part estimation from depth images via spatially-constrained deep learning. Pattern Recogn. Lett. (to appear).

[13]

C. Keskin, F. Kirac, Y. Kara, and L. Akarun. 2011. Real time hand pose estimation using depth sensors. In Proceedings of the IEEE International Computer Vision Workshops. 1228--1234.

[14]

C. Keskin, F. Kirac, Y. E. Kara, and L. Akarun. 2012. Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In Proceedings of the 12^th European Conference on Computer Vision. Vol. 6. Springer, 852--863.

Digital Library

[15]

A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Neural Information Processing Systems Conference. P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., 1106--1114.

[16]

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11, 2278--2324.

[17]

Y. Lecun, F. J. Huang, and L. Bottou. 2004. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2. 97--104.

Digital Library

[18]

H. Li, R. W. Sumner, and M. Pauly. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421--1430.

Digital Library

[19]

H. Li, J. Yu, Y. Ye, and C. Bregler. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4.

Digital Library

[20]

S. Melax, L. Keselman, and S. Orsten. 2013. Dynamics based 3D skeletal hand tracking. In Proceedings of the ACM Symposium on Interactive 3D Graphics and Games.

Digital Library

[21]

J. Nagi, F. Ducatelle, G. Di Caro, D. Ciresan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and L. Gambardella. 2011. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications. 342--347.

[22]

S. J. Nowlan and J. C. Platt. 1995. A convolutional neural network hand tracker. In Proceedings of the Neural Information Processing Systems Conference. 901--908.

[23]

I. Oikonomidis, N. Kyriazis, and A. Argyros. 2011. Efficient model-based 3D tracking of hand articulations using kinect. In Proceedings of the British Machine Vision Conference.

[24]

M. Osadchy, Y. Lecun, M. L. Miller, and P. Perona. 2005. Synergistic face detection and pose estimation with energy-based model. In Proceedings of the Neural Information Processing Systems Conference. 1017--1024.

[25]

J. M. Rehg and T. Kanade. 1994. Visual tracking of high dof articulated structures: An application to human hand tracking. In Proceedings of the 3^rd European Conference on Computer Vision. 35--46.

Digital Library

[26]

M. Saric. 2011. Libhand: A library for hand articulation. http://www.libhand. org/.

[27]

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. 2011. Real-time human pose recognition in parts from single depth images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1297--1304.

Digital Library

[28]

M. Stein, J. Tompson, X. Xiao, C. Hendeee, H. Ishii, and K. Perlin. 2012. Arcade: A system for augmenting gesture-based computer graphic presentations. In Proceedings of the ACM SIGGRAPH Computer Animation Festival. 77--77.

Digital Library

[29]

G. W. Taylor, I. Spiro, C. Bregler, and R. Fergus. 2011. Learning invariance through imitation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2729--2736.

Digital Library

[30]

P. Tseng. 1995. Fortified-descent simplicial search method: A general approach. SIAM J. Optim. 10, 1, 269--288.

Digital Library

[31]

R. Wang, S. Paris, and J. Popovic. 2011. 6d hands: Markerless tracking for computer aided design. In Proceedings of the 24^th Annual ACM Symposium on User Interface Software and Technology. 549--558.

Digital Library

[32]

R. Y. Wang. and J. Popovic. 2009. Real-time hand-tracking with a color glove. ACM Trans. Graph. 28, 3.

Digital Library

[33]

T. Weise, H. Li, L. Van Gool, and M. Pauly. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation.

Digital Library

[34]

T. Yasuda, K. Ohkura, and Y. Matsumura. 2010. Extended PSO with partial randomization for large scale multimodal problems. In Proceedings of the World Automation Congress. 1--6.

[35]

W. Zhao, J. Chai, and Y.-Q. Xu. 2012. Combining marker-based MOCAP and RGB-d camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 33--42.

Digital Library

Cited By

Yu SWang YChen LZhang XLi J(2024)3D hand pose and mesh estimation via a generic Topology-aware Transformer modelFrontiers in Neurorobotics10.3389/fnbot.2024.139565218Online publication date: 3-May-2024
https://doi.org/10.3389/fnbot.2024.1395652
Zhao WHu HZhou WLi LLi H(2024)Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB VideoACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363970720:6(1-18)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3639707
Han WLiu Z(2024)A lightweight real-time 3D hand gesture tracking solution for mobile devicesSeventh International Conference on Computer Graphics and Virtuality (ICCGV 2024)10.1117/12.3029630(19)Online publication date: 13-May-2024
https://doi.org/10.1117/12.3029630
Show More Cited By

Index Terms

Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
1. Computing methodologies
  1. Computer graphics
    1. Animation
    2. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Accurate, Robust, and Flexible Real-time Hand Tracking
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. ...
Tracking of deformable human hand in real time as continuous input for gesture-based interaction
IUI '07: Proceedings of the 12th international conference on Intelligent user interfaces

Gesture input is a natural and effective interactive model. The tracking of deformable hand gesture is a very important task in gesture-based interaction. A novel real-time tracking approach is proposed to capture hand motion with single camera. It ...
Continuous recovery for real time pan tilt zoom localization and mapping
AVSS '11: Proceedings of the 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance

We propose a method for real time recovering from tracking failure in monocular localization and mapping with a Pan Tilt Zoom camera (PTZ). The method automatically detects and seamlessly recovers from tracking failure while preserving map integrity. By ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 5

August 2014

152 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2672594

Editor:
Julie Dorsey
Yale University

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 September 2014

Accepted: 01 March 2014

Revised: 01 January 2014

Received: 01 August 2013

Published in TOG Volume 33, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

572
Total Citations
View Citations
3,265
Total Downloads

Downloads (Last 12 months)232
Downloads (Last 6 weeks)17

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu SWang YChen LZhang XLi J(2024)3D hand pose and mesh estimation via a generic Topology-aware Transformer modelFrontiers in Neurorobotics10.3389/fnbot.2024.139565218Online publication date: 3-May-2024
https://doi.org/10.3389/fnbot.2024.1395652
Zhao WHu HZhou WLi LLi H(2024)Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB VideoACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363970720:6(1-18)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3639707
Han WLiu Z(2024)A lightweight real-time 3D hand gesture tracking solution for mobile devicesSeventh International Conference on Computer Graphics and Virtuality (ICCGV 2024)10.1117/12.3029630(19)Online publication date: 13-May-2024
https://doi.org/10.1117/12.3029630
Bernal ESharma RYenneti SMackey IMalave JWalvoord DBrower B(2024)Task-oriented synthetic-to-real image translation for data-efficient learningSynthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications II10.1117/12.3013814(32)Online publication date: 7-Jun-2024
https://doi.org/10.1117/12.3013814
Georgievskaya ATlyachev TKiselev KHillebrand GChekanov KDanko DGolodyaev AMajmudar G(2024) Predicting human chronological age via AI analysis of dorsal hand versus facial images: A study in a cohort of Indian females Experimental Dermatology10.1111/exd.1504533:3Online publication date: 20-Mar-2024
https://doi.org/10.1111/exd.15045
Batch AJi YFan MZhao JElmqvist N(2024)uxSense: Supporting User Experience Analysis with Visualization and Computer VisionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324158130:7(3841-3856)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3241581
Jiang JLi JZhang BDeng XShi B(2024)EvHandPose: Event-Based 3D Hand Pose Estimation With Sparse SupervisionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338064846:9(6416-6430)Online publication date: Sep-2024
https://doi.org/10.1109/TPAMI.2024.3380648
Pang WGao QZhao YJu ZHu J(2024)BaSICNet: Lightweight 3D Hand Pose Estimation Network Based on Biomechanical Structure Information for Dexterous Manipulator TeleoperationIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2022.3230707(1-1)Online publication date: 2024
https://doi.org/10.1109/TCDS.2022.3230707
Wang YSun WRao R(2024)Accurate and Real-Time Variant Hand Pose Estimation Based on Gray Code Bounding Box RepresentationIEEE Sensors Journal10.1109/JSEN.2024.338905524:11(18043-18053)Online publication date: 1-Jun-2024
https://doi.org/10.1109/JSEN.2024.3389055
Gao YRen PShu MChu RLi JJin JLi W(2024)SO-Net: Model-Agnostic Sequential Hand Pose Optimization FrameworkICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10445741(3755-3759)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10445741
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents