research-article

VideoMocap: modeling physically realistic human motion from monocular video sequences

Authors:

Jinxiang ChaiAuthors Info & Claims

SIGGRAPH '10: ACM SIGGRAPH 2010 papers

Article No.: 42, Pages 1 - 10

https://doi.org/10.1145/1833349.1778779

Published: 26 July 2010 Publication History

Abstract

This paper presents a video-based motion modeling technique for capturing physically realistic human motion from monocular video sequences. We formulate the video-based motion modeling process in an image-based keyframe animation framework. The system first computes camera parameters, human skeletal size, and a small number of 3D key poses from video and then uses 2D image measurements at intermediate frames to automatically calculate the "in between" poses. During reconstruction, we leverage Newtonian physics, contact constraints, and 2D image measurements to simultaneously reconstruct full-body poses, joint torques, and contact forces. We have demonstrated the power and effectiveness of our system by generating a wide variety of physically realistic human actions from uncalibrated monocular video sequences such as sports video footage.

Supplementary Material

Supplemental material. (042.zip)

Download
58.60 MB

MP4 File (tp051-10.mp4)

Download
61.84 MB

References

[1]

Agarwal, A., and Triggs, B. 2006. Recovering 3D human pose from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). 28(1):44--58.

Digital Library

[2]

Bazaraa, M. S., Sherali, H. D., and Shetty, C. M. 1993. Nonlinear Programming: Theory and Algorithms. John Wiley and Sons Ltd. 2nd Edition.

[3]

Bregler, C., Malik, J., and Pullen, K. 2004. Twist Based Acquisition and Tracking of Animal and Human Kinematics. International Journal of Computer Vision. 56(3):179--194.

Digital Library

[4]

Brubaker, M. A., and Fleet, D. J. 2008. The Kneed Walker for human pose tracking. In Proceedings of IEEE CVPR. 1--8.

[5]

Chai, J., and Hodgins, J. 2005. Performance Animation from Low-dimensional Control Signals. In ACM Transactions on Graphics. 24(3):686--696.

Digital Library

[6]

Chen, Y.-L., and Chai, J. 2009. Simultaneous Reconstruction of 3D Human Skeleton and Motion from Monocular Video Sequences. Proceedings of The Ninth Asian Conference on Computer Vision.

[7]

Cohen, M. F. 1992. Interactive Spacetime Control for Animation. In Proceedings of ACM SIGGRAPH 1992. 293--302.

Digital Library

[8]

Comaniciu, D., and Meer, P. 2002. Mean Shift: A Robust Approach Toward Feature Space Analysis. In IEEE Trans. Pattern Analysis and Machine Intelligence. 24(5):603--619.

Digital Library

[9]

Cowley, A., and Taylor, C. J., 2001. Videomocap: A video based motion capture system. http://www.cis.upenn.edu/cjtaylor/RESEARCH/projects/Johansson/VideoMoCap.html.

[10]

DiFranco, D. E., Cham, T.-J., and Rehg, J. M. 2001. Reconstruction of 3D figure motion from 2D correspondences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1:307--314.

[11]

Elgammal, A., and Lee, C. 2004. Inferring 3D body pose from silhouettes using activity manifold learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2: 681--688.

Digital Library

[12]

Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A. 1986. Robust Statistics: The Approach Based on Influence Functions. Wiley.

[13]

Howe, N., Leventon, M., and Freeman, W. 1999. Bayesian Reconstruction of 3D Human Motion from Single-camera Video. In Advances in Neural Information Processing Systems 12. 820--826.

[14]

Huber, P. J. 1981. Robust Statistics. Wiley.

[15]

Kanaujia, C. S. A., and Metaxas, D. 2007. BM³E: Discriminative density propagation for visual tracking. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). 29(11):2030--2044.

Digital Library

[16]

Liu, K., Hertzmann, A., and Popović, Z. 2005. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization. In ACM Transactions on Graphics. 23(3):1071--1081.

Digital Library

[17]

Lourakis, M. I. A. 2009. levmar: Levenberg marquardt nonlinear least squares algorithms in c/c++. In http://www.ics.forth.gr/lourakis/levmar/.

[18]

Loy, G., Eriksson, M., Sullivan, J., and Carlsson, S. 2004. Monocular 3D Reconstruction of Human motion in Long Action Sequences. In European Conference on Computer Vision. 442--455.

[19]

MatchMover, 2008. http://www.realviz.com/.

[20]

Pavlović, V., Rehg, J. M., and MacCormick, J. 2000. Learning Switching Linear Models of Human Motion. In Advances in Neural Information Processing Systems 13, 981--987.

[21]

Pollard, N., and Reitsma, P. 2001. Animation of Human-like Characters: Dynamic Motion Filtering with A Physically Plausible Contact Model. In In Yale Workshop on Adaptive and Learning Systems.

[22]

Popović, Z., and Witkin, A. P. 1999. Physically Based Motion Transformation. In Proceedings of ACM SIGGRAPH 1999. 11--20.

Digital Library

[23]

Rosales, R., and Sclaroff, S. 2000. Specialized Mappings and the Estimation of Human Body Pose from a Single Image. In Proceedings of the Workshop on Human Motion. 19--24.

Digital Library

[24]

Safonova, A., Hodgins, J., and Pollard, N. 2004. Synthesizing Physically Realistic Human Motion in Low-Dimensional, Behavior-Specific Spaces. In ACM Transactions on Graphics. 23(3):514--521.

Digital Library

[25]

Sidenbladh, H., Black, M. J., and Sigal, L. 2002. Implicit Probabilistic Models of Human Motion for Synthesis and Tracking. In European Conference on Computer Vision. 784--800.

Digital Library

[26]

Sminchisescu, C., and Jepson, A. 2004. Generative Modeling for Continuous Non-Linearly Embedded Visual Inference. In ICML, 759--766.

Digital Library

[27]

Sulejmanpasic, A., and Popović, J. 2005. Adaptation of Performed Ballistic Motion. In ACM Transactions on Graphics. 24(1):165--179.

Digital Library

[28]

Taylor, C. J. 2000. Reconstruction of Articulated Objects from Point Correspondences in a Single Uncalibrated Image. In Computer Vision and Image Understanding. 80(3):349--363.

Digital Library

[29]

Urtasun, R., Fleet, D. J., Hertzmann, A., and Fua., P. 2005. Priors for people tracking from small training sets. In IEEE International Conference on Computer Vision, 403--410.

Digital Library

[30]

Vicon Systems, 2009. http://www.vicon.com.

[31]

Vondrak, M., Sigal, L., and Jenkins, O. C. 2008. Physical simulation for probabilistic motion tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 1--8.

[32]

Wei, X., and Chai, J. 2008. Interactive Tracking of 2D Generic Objects with Spacetime Optimization. In Proceedings of European Conference on Computer Vision. 1:657--670.

Digital Library

[33]

Wei, X., and Chai, J. 2009. Modeling 3D Human Poses from Uncalibrated Monocular Images. Proceedings of IEEE Conference on Computer Vision.

[34]

Witkin, A., and Kass, M. 1988. Spacetime Constraints. In Proceedings of ACM SIGGRAPH 1998. 159--168.

Digital Library

Cited By

Zhang HYuan YMakoviychuk VGuo YFidler SPeng XFatahalian K(2023)Learning Physically Simulated Tennis Skills from Broadcast VideosACM Transactions on Graphics10.1145/359240842:4(1-14)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592408
Kim JLee S(2023)Potentially Visible Hidden-Volume Rendering for Multi-View WarpingACM Transactions on Graphics10.1145/359210842:4(1-11)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592108
Cao DRoetzer PBernard F(2023)Unsupervised Learning of Robust Spectral Shape MatchingACM Transactions on Graphics10.1145/359210742:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592107
Show More Cited By

Index Terms

VideoMocap: modeling physically realistic human motion from monocular video sequences
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Video summarization
      2. Image and video acquisition
        3D imaging
        Motion capture
  2. Computer graphics
    1. Animation
      1. Motion capture
      2. Motion processing
    2. Shape modeling
2. Theory of computation
  1. Randomness, geometry and discrete structures
    1. Computational geometry

Recommendations

VideoMocap: modeling physically realistic human motion from monocular video sequences

This paper presents a video-based motion modeling technique for capturing physically realistic human motion from monocular video sequences. We formulate the video-based motion modeling process in an image-based keyframe animation framework. The system ...
Realtime human motion control with a small number of inertial sensors
I3D '11: Symposium on Interactive 3D Graphics and Games

This paper introduces an approach to performance animation that employs a small number of motion sensors to create an easy-to-use system for an interactive control of a full-body human character. Our key idea is to construct a series of online local ...
Physics-based character animation with cascadeur
SIGGRAPH '19: ACM SIGGRAPH 2019 Studio

In this workshop we will create a realistic acrobatic 3D fighting animation using the animation software Cascadeur. We will learn the key features of physics-based character animation and will immediately apply the learned knowledge by creating an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '10: ACM SIGGRAPH 2010 papers

July 2010

984 pages

ISBN:9781450302104

DOI:10.1145/1833349

Conference Chair:
Tony DeRose,
Editor:
Hugues Hoppe
ACM Transactions on Graphics

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGGRAPH '10

Sponsor:

SIGGRAPH

SIGGRAPH '10: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 26 - 30, 2010

California, Los Angeles

Acceptance Rates

SIGGRAPH '10 Paper Acceptance Rate 103 of 390 submissions, 26%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
2,234
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HYuan YMakoviychuk VGuo YFidler SPeng XFatahalian K(2023)Learning Physically Simulated Tennis Skills from Broadcast VideosACM Transactions on Graphics10.1145/359240842:4(1-14)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592408
Kim JLee S(2023)Potentially Visible Hidden-Volume Rendering for Multi-View WarpingACM Transactions on Graphics10.1145/359210842:4(1-11)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592108
Cao DRoetzer PBernard F(2023)Unsupervised Learning of Robust Spectral Shape MatchingACM Transactions on Graphics10.1145/359210742:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592107
Shen YSaito SWang ZMaury OWu CHodgins JZheng YNam G(2023)CT2Hair: High-Fidelity 3D Hair Modeling using Computed TomographyACM Transactions on Graphics10.1145/359210642:4(1-13)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592106
Wang YGuo MSolomon J(2023)Variational quasi-harmonic maps for computing diffeomorphismsACM Transactions on Graphics10.1145/359210542:4(1-26)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592105
Yi XZhou YHabermann MShimada SGolyanik VTheobalt CXu F(2022)Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01282(13157-13168)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.01282
Huang BPan LYang YJu JWang Y(2022)Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00631(6407-6416)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.00631
Yu RPark HLee J(2021)Human dynamics from monocular video with dynamic camera movementsACM Transactions on Graphics10.1145/3478513.348050440:6(1-14)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480504
Yuan YWei SSimon TKitani KSaragih J(2021)SimPoE: Simulated Character Control for 3D Human Pose Estimation2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.00708(7155-7165)Online publication date: Jun-2021
https://doi.org/10.1109/CVPR46437.2021.00708
Li JVillegas RCeylan DYang JKuang ZLi HZhao Y(2021)Task-Generic Hierarchical Human Motion Prior using VAEs2021 International Conference on 3D Vision (3DV)10.1109/3DV53792.2021.00086(771-781)Online publication date: Dec-2021
https://doi.org/10.1109/3DV53792.2021.00086
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents