article

Keyframe-based tracking for rotoscoping and animation

Authors:

Aseem Agarwala,

Aaron Hertzmann,

David H. Salesin,

Steven M. SeitzAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 23, Issue 3

Pages 584 - 591

https://doi.org/10.1145/1015706.1015764

Published: 01 August 2004 Publication History

Abstract

We describe a new approach to rotoscoping --- the process of tracking contours in a video sequence --- that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these curves are used as keyframes by a computer-vision-based tracking algorithm. The user may interactively refine the curves and then restart the tracking algorithm. Combining computer vision with user interaction allows our system to track any sequence with significantly less effort than interpolation-based systems --- and with better reliability than "pure" computer vision systems. Our tracking algorithm is cast as a spacetime optimization problem that solves for time-varying curve shapes based on an input video sequence and user-specified constraints. We demonstrate our system with several rotoscoped examples. Additionally, we show how these rotoscoped contours can be used to help create cartoon animation by attaching user-drawn strokes to the tracked contours.

Supplementary Material

MOV File (pps048.mov)

Download
11.77 KB

References

[1]

AGARWALA, A. 2002. SnakeToonz: A Semi-Automatic Approach to Creating Cel Animation from Video. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 139--146.

Digital Library

[2]

BEIER, T., AND NEELY, S. 1992. Feature-based image metamorphosis. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 35--42.

Digital Library

[3]

BERGEN, J. R., ANANDAN, P., HANNA, K. J., AND HINGORANI, R. 1992. Hierarchical model-based motion estimation. In European Conference on Computer Vision, 237--252.

Digital Library

[4]

BLACK, M. J., AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75--104.

Digital Library

[5]

BLAKE, A., AND ISARD, M. 1998. Active Contours, Springer-Verlag.

[6]

BURTNYK, N., AND WEIN, M. 1976. Interactive skeleton techniques for enhancing motion dynamics in key frame animation. CACM 19 (Oct.), 564--569.

Digital Library

[7]

CHUANG, Y.-Y., AGARWALA, A., CURLESS, B., SALESIN, D. H., AND SZELISKI, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243--248.

Digital Library

[8]

COHEN, M. F. 1992. Interactive spacetime control for animation. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 293--302.

Digital Library

[9]

FLEISCHER, M., 1917. Method of Producing Moving Picture Cartoons. US Patent no. 1,242,674.

[10]

GEIGER, D., GUPTA, A., COSTA, L., AND VLONTZOS, J. 1995. Dynamic programming for detecting, tracking and matching deformable contours. IEEE Transactions On Pattern Analysis and Machine Intelligence 17, 3, 294--302.

Digital Library

[11]

GOLDMAN, D., 2003. Computer graphics supervisor, Industrial Light & Magic, personal communication.

[12]

HALL, J., GREENHILL, D., AND JONES, G. 1997. Segmenting film sequences using active surfaces. In International Conference on Image Processing (ICIP), 751--754.

Digital Library

[13]

HOCH, M., AND LITWINOWICZ, P. C. 1996. A semi-automatic system for edge tracking with snakes. The Visual Computer 12, 2, 75--83.

[14]

HSU, S. C., AND LEE, I. H. H. 1994. Drawing and animation using skeletal strokes. In Proceedings of SIGGRAPH 94, 109--118.

Digital Library

[15]

IRANI, M. 2002. Multi-Frame Correspondence Estimation Using Subspace Constraints. International Journal of Computer Vision 48, 3, 173--194.

Digital Library

[16]

KALNINS, R. D., MARKOSIAN, L., MEIER, B. J., KOWALSKI, M. A., LEE, J. C., DAVIDSON, P. L., WEBB, M., HUGHES, J. F., AND FINKELSTEIN, A. 2002. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics 21, 3, 755--762.

Digital Library

[17]

KASS, M., WITKIN, A., AND TERZOPOULOS, D. 1987. Snakes: Active contour models. International Journal of Computer Vision 1, 4, 321--331.

[18]

KORT, A. 2002. Computer aided inbetweening. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 125--132.

Digital Library

[19]

LINKLATER, R., 2001. Waking Life. 20th Century Fox.

[20]

LITWINOWICZ, P., AND WILLIAMS, L. 1994. Animating images with drawings. In Proceedings of SIGGRAPH 94, 409--412.

Digital Library

[21]

LITWINOWICZ, P. 1997. Processing images and video for an impressionist effect. In Proceedings of SIGGRAPH 97, 407--414.

Digital Library

[22]

LUCAS, B. D., AND KANADE, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI'81), 674--679.

[23]

LUO, H., AND ELEFTHERIADIS, A. 1999. Spatial temporal active contour interpolation for semi-automatic video object generation. In International Conference on Image Processing (ICIP), 944--948.

[24]

MEIER, B. J. 1996. Painterly rendering for animation. In Proceedings of SIGGRAPH 96, 477--484.

Digital Library

[25]

MITSUNAGA, T., YOKOYAMA, T., AND TOTSUKA, T. 1995. Autokey: Human assisted key extraction. In Proceedings of SIGGRAPH 95, 265--272.

Digital Library

[26]

MORTENSEN, E. N., AND BARRETT, W. A. 1995. Intelligent scissors for image composition. In Proceedings of SIGGRAPH 95, 191--198.

Digital Library

[27]

MORTENSEN, E. N., 1999. Vision-assisted image editing. Computer Graphics 33, 4 (Nov.), 55--57.

Digital Library

[28]

NOCEDAL, J., AND WRIGHT, S. J. 1999. Numerical Optimization. Springer.

[29]

SEDERBERG, T. W., AND GREENWOOD, E. 1992. A physically based approach to 2d shape blending. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 25--34.

Digital Library

[30]

SEDERBERG, T. W., GAO, P., WANG, G., AND MU, H. 1993. 2d shape blending: An intrinsic solution to the vertex path problem. In Proceedings of SIGGRAPH 93, 15--18.

Digital Library

[31]

SMITH, A. R., AND BLINN, J. F. 1996. Blue screen matting. In Proceedings of SIGGRAPH 96, 259--268.

Digital Library

[32]

STEIHAUG, T. 1983. The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis 20, 3, 626--637.

[33]

STEWART, S., 2003. Confessions of a roto artist: Three rules for better mattes. http://www.pinnaclesys.com/SupportFiles/Rotoscoping.pdf.

[34]

SZELISKI, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions On Pattern Analysis and Machine Intelligence 12, 6, 513--528.

Digital Library

[35]

TORRESANI, L., AND BREGLER, C. 2002. Space-time tracking. In European Conference on Computer Vision, 801--802.

Digital Library

[36]

WITKIN, A., AND KASS, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), vol. 22, 159--168.

Digital Library

Cited By

Zheng YCun XXia MPun C(2024)Sketch Video SynthesisComputer Graphics Forum10.1111/cgf.1504443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15044
Du QDuan YXie ZTao XShi LJin Z(2024)Optical Flow-Based Spatiotemporal Sketch for Video Representation: A Novel FrameworkIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.334913034:8(6963-6977)Online publication date: Aug-2024
https://doi.org/10.1109/TCSVT.2023.3349130
Chen JLi QZhang WWang BZhang D(2024)A Degradation-Robust Keyframe Selection Method Based on Image Quality Evaluation for Visual LocalizationIEEE Internet of Things Journal10.1109/JIOT.2024.336579411:10(18421-18434)Online publication date: 15-May-2024
https://doi.org/10.1109/JIOT.2024.3365794
Show More Cited By

Index Terms

Keyframe-based tracking for rotoscoping and animation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Keyframe-based tracking for rotoscoping and animation
SIGGRAPH '04: ACM SIGGRAPH 2004 Papers

We describe a new approach to rotoscoping --- the process of tracking contours in a video sequence --- that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these ...
Stylized keyframe animation of fluid simulations
NPAR '14: Proceedings of the Workshop on Non-Photorealistic Animation and Rendering

We present a method that combines hand-drawn artwork with fluid simulations to produce animated fluids in the visual style of the artwork. Given a fluid simulation and a set of keyframes rendered by the artist in any medium, our system produces a set of ...
Interactive rotoscoping: extracting and tracking object sketch
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

In this paper, we discuss an integrated system for video rotoscoping extracting and tracking object sketch (structural shape) across video sequence. This system consists of two key components: object sketch computing and graph-based object tracking. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 23, Issue 3

August 2004

684 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1015706

Editor:
John C. Hart

Issue’s Table of Contents

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2004

Published in TOG Volume 23, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

121
Total Citations
View Citations
2,728
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)3

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zheng YCun XXia MPun C(2024)Sketch Video SynthesisComputer Graphics Forum10.1111/cgf.1504443:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15044
Du QDuan YXie ZTao XShi LJin Z(2024)Optical Flow-Based Spatiotemporal Sketch for Video Representation: A Novel FrameworkIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.334913034:8(6963-6977)Online publication date: Aug-2024
https://doi.org/10.1109/TCSVT.2023.3349130
Chen JLi QZhang WWang BZhang D(2024)A Degradation-Robust Keyframe Selection Method Based on Image Quality Evaluation for Visual LocalizationIEEE Internet of Things Journal10.1109/JIOT.2024.336579411:10(18421-18434)Online publication date: 15-May-2024
https://doi.org/10.1109/JIOT.2024.3365794
Ingle PKim Y(2023)Video Synopsis Algorithms and Framework: A Survey and Comparative EvaluationSystems10.3390/systems1102010811:2(108)Online publication date: 17-Feb-2023
https://doi.org/10.3390/systems11020108
Fan JZhang KZhao YLiu Q(2023)Unsupervised Video Object Segmentation via Weak User Interaction and Temporal ModulationChinese Journal of Electronics10.23919/cje.2022.00.13932:3(507-518)Online publication date: May-2023
https://doi.org/10.23919/cje.2022.00.139
Hachmann HRosenhahn BZimmermann RShirmohammadi SGriwodz CHefeeda MWang M(2023)Color-aware Deep Temporal Backdrop Duplex Matting SystemProceedings of the 14th Conference on ACM Multimedia Systems10.1145/3587819.3590973(205-216)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3587819.3590973
Pan YZhang RCheng STan SDing YMitchell KYang X(2023)Emotional Voice PuppetryIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324710129:5(2527-2535)Online publication date: 22-Feb-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247101
Zhou TPorikli FCrandall DVan Gool LWang W(2023)A Survey on Deep Learning Technique for Video SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.322557345:6(7099-7122)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3225573
Chan CYuan CSun CChen H(2023)Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00712(7709-7719)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00712
Du QDuan YXie ZTao XZeng RJin ZLi F(2023)Optical Flow-Based Video Sketch Graph Extraction2023 IEEE Globecom Workshops (GC Wkshps)10.1109/GCWkshps58843.2023.10464524(1279-1284)Online publication date: 4-Dec-2023
https://doi.org/10.1109/GCWkshps58843.2023.10464524
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents