article

Video matting of complex scenes

Authors:

Yung-Yu Chuang,

Aseem Agarwala,

David H. Salesin,

Richard SzeliskiAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 21, Issue 3

Pages 243 - 248

https://doi.org/10.1145/566654.566572

Published: 01 July 2002 Publication History

Abstract

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into "foreground," "background" and "unknown". The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.

References

[1]

BARRON, J. L., FLEET, D. J., AND BEAUCHEMIN, S. S. 1994. Performance of optical flow techniques. International Journal of Computer Vision 12, 1, 43-77.

[2]

BERMAN, A., DADOURIAN, A., AND VLAHOS, P., 2000. Method for removing from an image the background surrounding a selected object. U.S. Patent 6,134,346.

[3]

BLACK, M. J., AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75-104.

[4]

BLAKE, A., AND ISARD, M. 1998. Active Contours. Springer Verlag, London.

[5]

CHUANG, Y.-Y., CURLESS, B., SALESIN, D., AND SZELISKI, R. 2001. A Bayesian approach to digital matting. In Proceedings of Computer Vision and Pattern Recognition (CVPR 2001), vol. II, 264 - 271.

[6]

GLEICHER, M. 1995. Image snapping. In Proceedings of ACM SIGGRAPH 95, 183-190.

[7]

HILLMAN, P., HANNAH, J., AND RENSHAW, D. 2001. Alpha channel estimation in high resolution images and image sequences. In Proceedings of Computer Vision and Pattern Recognition (CVPR 2001), vol. I, 1063-1068.

[8]

KELLY, D. 2000. Digital Composition. The Coriolis Group.

[9]

LEE, M.-C., ET AL. 1997. A layered video object coding system using sprite and affine motion model. lEEE Transactions on Circuits and Systems for Video Technology 7, 1, 130-145.

[10]

MITSUNAGA, T., YOKOYAMA, T., AND TOTSUKA, T. 1995. Autokey: Human assisted key extraction. In Proceedings of ACM SIGGRAPH 95, 265-272.

[11]

MORTENSEN, E. N., AND BARRETT, W. A. 1995. Intelligent scissors for image composition. In Proceedings of ACM SIGGRAPH 95, 191-198.

[12]

PORTER, T., AND DUFF, T. 1984. Compositing digital images. In Computer Graphics (Proceedings of ACM SIGGRAPH 84), vol. 18, 253-259.

[13]

RUZON, M. A., AND TOMASI, C. 2000. Alpha estimation in natural images. In Proceedings of Computer Vision and Pattern Recognition (CVPR 2000), 18-25.

[14]

SMITH, A. R., AND BLINN, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH 96, 259-268.

[15]

SUN, S., HAYNOR, D., AND KIM, Y. 2000. Motion estimation based on optical flow with adaptive gradients. In Proceedings of International Conference on Image Processing (ICIP 2000), vol. I, 852-855.

[16]

SZELISKI, R., AND SHUM, H.-Y. 1997. Creating full view panoramic mosaics and environment maps. In Proceedings of ACM SIGGRAPH 97, 251-258.

[17]

WANG, J. Y. A., AND ADELSON, E. H. 1994. Representing moving images with layers. IEEE Transactions on Image Processing 3, 5, 625-638.

Cited By

Liu FGleicher MJin HAgarwala A(2023)Content-Preserving Warps for 3D Video StabilizationSeminal Graphics Papers: Pushing the Boundaries, Volume 210.1145/3596711.3596778(631-639)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1145/3596711.3596778
Lin GGao CHuang JKim CWang YZwicker MSaraf A(2023)OmnimatteRF: Robust Omnimatte with 3D Background Modeling2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.02145(23414-23423)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.02145
Sun YTang CTai Y(2023)Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01356(14112-14121)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01356
Show More Cited By

Index Terms

Video matting of complex scenes
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Interest point and salient region detections
  2. Computer graphics
    1. Image manipulation

Recommendations

Poisson matting
SIGGRAPH '04: ACM SIGGRAPH 2004 Papers

In this paper, we formulate the problem of natural image matting as one of solving Poisson equations with the matte gradient field. Our approach, which we call Poisson matting, has the following advantages. First, the matte is directly reconstructed ...
Video matting of complex scenes
SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background ...
Shadow matting and compositing

In this paper, we describe a method for extracting shadows from one natural scene and inserting them into another. We develop physically-based shadow matting and compositing equations and use these to pull a shadow matte from a source scene in which the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 21, Issue 3

July 2002

548 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/566654

Issue’s Table of Contents

Copyright © 2002 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2002

Published in TOG Volume 21, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

164
Total Citations
View Citations
2,789
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu FGleicher MJin HAgarwala A(2023)Content-Preserving Warps for 3D Video StabilizationSeminal Graphics Papers: Pushing the Boundaries, Volume 210.1145/3596711.3596778(631-639)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1145/3596711.3596778
Lin GGao CHuang JKim CWang YZwicker MSaraf A(2023)OmnimatteRF: Robust Omnimatte with 3D Background Modeling2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.02145(23414-23423)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.02145
Sun YTang CTai Y(2023)Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01356(14112-14121)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01356
Zhang YWang CCui MRen PXie XHua XBao HHuang QXu WShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Attention-guided Temporally Coherent Video Object MattingProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475623(5128-5137)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475623
Chen HXie WAfouras TNagrani AVedaldi AZisserman A(2021)Localizing Visual Sounds the Hard Way2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.01659(16862-16871)Online publication date: Jun-2021
https://doi.org/10.1109/CVPR46437.2021.01659
Sun YWang GGu QTang CTai Y(2021)Deep Video Matting via Spatio-Temporal Alignment and Aggregation2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.00690(6971-6980)Online publication date: Jun-2021
https://doi.org/10.1109/CVPR46437.2021.00690
Jia J(2021)Matte ExtractionComputer Vision10.1007/978-3-030-63416-2_12(795-799)Online publication date: 13-Oct-2021
https://doi.org/10.1007/978-3-030-63416-2_12
Pérez-Rúa JMiksik OCrivelli TBouthemy PTorr PPérez P(2020)ROAM: A Rich Object Appearance Model with Application to RotoscopingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2019.290496342:8(1996-2010)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1109/TPAMI.2019.2904963
Zou DChen XCao GWang X(2020)Unsupervised Video Matting via Sparse and Low-Rank RepresentationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2019.289533142:6(1501-1514)Online publication date: 1-Jun-2020
https://doi.org/10.1109/TPAMI.2019.2895331
Li HFang LZhang T(2020)Guest Editorial Introduction to the Special Section on Intelligent Visual Content Analysis and UnderstandingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2020.303141630:12(4405-4408)Online publication date: 1-Dec-2020
https://dl.acm.org/doi/10.1109/TCSVT.2020.3031416
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents