research-article

MovieReshape: tracking and reshaping of humans in videos

Authors:

Thorsten Thormählen,

Hans-Peter Seidel,

Christian TheobaltAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 29, Issue 6

Article No.: 148, Pages 1 - 10

https://doi.org/10.1145/1882261.1866174

Published: 15 December 2010 Publication History

Abstract

We present a system for quick and easy manipulation of the body shape and proportions of a human actor in arbitrary video footage. The approach is based on a morphable model of 3D human shape and pose that was learned from laser scans of real people. The algorithm commences by spatio-temporally fitting the pose and shape of this model to the actor in either single-view or multi-view video footage. Once the model has been fitted, semantically meaningful attributes of body shape, such as height, weight or waist girth, can be interactively modified by the user. The changed proportions of the virtual human model are then applied to the actor in all video frames by performing an image-based warping. By this means, we can now conveniently perform spatio-temporal reshaping of human actors in video footage which we show on a variety of video sequences.

Supplementary Material

Supplemental material. (148-152-0286-auxiliary.zip)

Supplementary video for paper "MovieReshape: Tracking and Reshaping of Humans in Videos" AVI, 960 x 540, 25.00 fps, video: XviD, audio: MPEG-1 Layer 3 (stereo, 48000 Hz)

Download
83.43 MB

References

[1]

Agarwal, A., and Triggs, B. 2006. Recovering 3d human pose from monocular images. IEEE Trans. PAMI 28, 1, 44--58.

Digital Library

[2]

Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In Proc. ACM SIGGRAPH '03, 587--594.

Digital Library

[3]

Allen, B., Curless, B., Popović, Z., and Hertzmann, A. 2006. Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis. In Proc. SCA, 147--156.

Digital Library

[4]

Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. SCAPE: Shape completion and animation of people. In ACM TOG (Proc. SIGGRAPH '05).

Digital Library

[5]

Barrett, W. A., and Cheney, A. S. 2002. Object-based image editing. In Proc. ACM SIGGRAPH '02, ACM, 777--784.

Digital Library

[6]

Bennett, E. P., and McMillan, L. 2003. Proscenium: a framework for spatio-temporal video editing. In Proc. ACM MULTIMEDIA '03, 177--184.

Digital Library

[7]

Bǎlan, A. O., Sigal, L., Black, M. J., Davis, J. E., and Haussecker, H. W. 2007. Detailed human shape and pose from images. In Proc. IEEE CVPR.

[8]

Davis, J., Agrawala, M., Chuang, E., Popović, Z., and Salesin, D. 2003. A sketching interface for articulated figure animation. In Proc. SCA, 320--328.

Digital Library

[9]

de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. In ACM TOG (Proc. SIGGRAPH '08).

Digital Library

[10]

Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., and Seidel, H.-P. 2009. Motion capture using simultaneous skeleton tracking and surface estimation. In Proc. IEEE CVPR.

[11]

Guan, P., Weiss, A., Bǎlan, A. O., and Black, M. J. 2009. Estimating human shape and pose from a single image. In Proc. IEEE ICCV.

[12]

Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., and Seidel, H.-P. 2009. A statistical model of human pose and body shape. In CGF (Proc. Eurographics 2008), vol. 2.

[13]

Hasler, N., Ackermann, H., Rosenhahn, B., Thormählen, T., and Seidel, H.-P. 2010. Multilinear pose and body shape estimation of dressed subjects from image sets. In Proc. IEEE CVPR.

[14]

Hornung, A., Dekkers, E., and Kobbelt, L. 2007. Character animation from 2d pictures and 3d motion data. ACM TOG 26, 1, 1.

Digital Library

[15]

Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. In Proc. ACM SIGGRAPH Asia '09, 1--10.

Digital Library

[16]

Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. ACM TOG 27, 3, 1--9.

Digital Library

[17]

Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM TOG 24, 3, 595--600.

Digital Library

[18]

Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. In Proc. ACM SIGGRAPH '05, 519--526.

Digital Library

[19]

Müller, M., Heidelberger, B., Teschner, M., and Gross, M. 2005. Meshless deformations based on shape matching. ACM TOG 24, 3, 471--478.

Digital Library

[20]

Parameswaran, V., and Chellappa, R. 2004. View independent human body pose estimation from a single perspective image. In Proc. IEEE CVPR, II: 16--22.

Digital Library

[21]

Poppe, R. 2007. Vision-based human motion analysis: An overview. CVIU 108, 1--2, 4--18.

Digital Library

[22]

Ritschel, T., Okabe, M., Thormählen, T., and Seidel, H.-P. 2009. Interactive reflection editing. ACM TOG (Proc. SIGGRAPH Asia '09) 28, 5.

Digital Library

[23]

Rosales, R., and Sclaroff, S. 2006. Combining generative and discriminative models in a framework for articulated pose estimation. Int. J. Comput. Vision 67, 3, 251--276.

Digital Library

[24]

Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM TOG (Proc. SIGGRAPH '08) 27, 3, 1--9.

Digital Library

[25]

Schaefer, S., McPhail, T., and Warren, J. 2006. Image deformation using moving least squares. ACM TOG 25, 3, 533--540.

Digital Library

[26]

Scholz, V., and Magnor, M. 2006. Texture replacement of garments in monocular video sequences. In Proc. EGSR, 305--312.

Digital Library

[27]

Scholz, V., El-Abed, S., Seidel, H.-P., and Magnor, M. A. 2009. Editing object behaviour in video sequences. CGF 28, 6, 1632--1643.

[28]

Seo, H., and Magnenat-Thalmann, N. 2004. An example-based approach to human body manipulation. Graph. Models 66, 1, 1--23.

Digital Library

[29]

Sigal, L., Balan, A. O., and Black, M. J. 2007. Combined discriminative and generative articulated pose and nonrigid shape estimation. In Proc. NIPS.

[30]

Stolfi, J. 1991. Oriented Projective Geometry: A Framework for Geometric Computation. Academic Press.

Digital Library

[31]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM TOG 24, 3, 426--433.

Digital Library

[32]

Vlasic, D., Baran, I., Matusik, W., and Popović, J. 2008. Articulated mesh animation from multi-view silhouettes. ACM TOG (Proc. SIGGRAPH '08).

Digital Library

[33]

Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. In Proc. ACM SIGGRAPH '05, ACM, 585--594.

Digital Library

[34]

Wang, J., Drucker, S. M., Agrawala, M., and Cohen, M. F. 2006. The cartoon animation filter. ACM TOG (Proc. SIGGRAPH '06), 1169--1173.

Digital Library

[35]

Wang, H., Xu, N., Raskar, R., and Ahuja, N. 2007. Videoshop: A new framework for spatio-temporal video editing in gradient domain. Graph. Models 69, 1, 57--70.

Digital Library

[36]

Wei, X., and Chai, J. 2010. Videomocap: modeling physically realistic human motion from monocular video sequences. ACM TOG (Proc. SIGGRAPH '10) 29, 4.

Digital Library

[37]

Zhou, S., Fu, H., Liu, L., Cohen-Or, D., and Han, X. 2010. Parametric reshaping of human bodies in images. ACM TOG (Proc. SIGGRAPH '10) 29, 4.

Digital Library

Cited By

Jia ZZhang ZWang LTan T(2024)Human Image Generation: A Comprehensive SurveyACM Computing Surveys10.1145/366586956:11(1-39)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3665869
Yang YZhang HFernández AAlemany SChen SZhang G(2024)Digitalization of 3-D Human Bodies: A SurveyIEEE Transactions on Consumer Electronics10.1109/TCE.2024.336361670:1(3152-3166)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3363616
Ye QHuang RWang ZLyu YLiu HSun Y(2024)Measurements-to-body: 3D human body reshaping based on anthropometric measurementsThe Journal of The Textile Institute10.1080/00405000.2024.2343120(1-14)Online publication date: 6-May-2024
https://doi.org/10.1080/00405000.2024.2343120
Show More Cited By

Recommendations

MovieReshape: tracking and reshaping of humans in videos
SIGGRAPH ASIA '10: ACM SIGGRAPH Asia 2010 papers

We present a system for quick and easy manipulation of the body shape and proportions of a human actor in arbitrary video footage. The approach is based on a morphable model of 3D human shape and pose that was learned from laser scans of real people. ...
Keyframe-based tracking for rotoscoping and animation
SIGGRAPH '04: ACM SIGGRAPH 2004 Papers

We describe a new approach to rotoscoping --- the process of tracking contours in a video sequence --- that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these ...
Trapunta Attori: AMV Creation Support System Focusing on Characters in a Video
UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

Anime music videos (AMVs) are fan-made music videos that are created by editing and synthesizing anime footage and music. Anime fans create AMVs and share them on the Internet and with their friends. However, AMV creation takes time and effort. It is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 29, Issue 6

December 2010

480 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1882261

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 December 2010

Published in TOG Volume 29, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

82
Total Citations
View Citations
1,059
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)3

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jia ZZhang ZWang LTan T(2024)Human Image Generation: A Comprehensive SurveyACM Computing Surveys10.1145/366586956:11(1-39)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3665869
Yang YZhang HFernández AAlemany SChen SZhang G(2024)Digitalization of 3-D Human Bodies: A SurveyIEEE Transactions on Consumer Electronics10.1109/TCE.2024.336361670:1(3152-3166)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3363616
Ye QHuang RWang ZLyu YLiu HSun Y(2024)Measurements-to-body: 3D human body reshaping based on anthropometric measurementsThe Journal of The Textile Institute10.1080/00405000.2024.2343120(1-14)Online publication date: 6-May-2024
https://doi.org/10.1080/00405000.2024.2343120
Li RLi DZhang M(2023)3D Human Reconstruction from A Single Depth Image2023 International Conference on Intelligent Perception and Computer Vision (CIPCV)10.1109/CIPCV58883.2023.00011(33-39)Online publication date: May-2023
https://doi.org/10.1109/CIPCV58883.2023.00011
Zheng QDing PShen YWang JZhang Z(2023)Research on 3D Driveable Digital Human Generation System2023 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)10.1109/BMSB58369.2023.10211615(1-5)Online publication date: 14-Jun-2023
https://doi.org/10.1109/BMSB58369.2023.10211615
Luo XHu YGao ZGuo HSu Y(2023)Automated measurement of livestock body based on pose normalisation using statistical shape modelBiosystems Engineering10.1016/j.biosystemseng.2023.01.016227(36-51)Online publication date: Mar-2023
https://doi.org/10.1016/j.biosystemseng.2023.01.016
Hemati ABastanfard A(2023)Adjustable method based on body parts for improving the accuracy of 3D reconstruction in visually important body parts from silhouettesMultimedia Tools and Applications10.1007/s11042-023-16170-783:7(21583-21613)Online publication date: 27-Jul-2023
https://doi.org/10.1007/s11042-023-16170-7
Li XLi GLi TLv JMitrouchev P(2023)Remodeling of mannequins based on automatic binding of mesh to anthropometric parametersThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02738-139:12(6435-6458)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00371-022-02738-1
Xiang DBagautdinov TStuyck TPrada FRomero JXu WSaito SGuo JSmith BShiratori TSheikh YHodgins JWu C(2022)Dressing AvatarsACM Transactions on Graphics10.1145/3550454.355545641:6(1-15)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555456
Cai JWohn D(2021)After Violation But Before Sanction: Understanding Volunteer Moderators' Profiling Processes Toward Violators in Live Streaming CommunitiesProceedings of the ACM on Human-Computer Interaction10.1145/34795545:CSCW2(1-25)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479554
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents