research-article

Discontinuity-aware video object cutout

Authors:

Xiangxu MengAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 31, Issue 6

Article No.: 175, Pages 1 - 10

https://doi.org/10.1145/2366145.2366194

Published: 01 November 2012 Publication History

Abstract

Existing video object cutout systems can only deal with limited cases. They usually require detailed user interactions to segment real-life videos, which often suffer from both inseparable statistics (similar appearance between foreground and background) and temporal discontinuities (e.g. large movements, newly-exposed regions following disocclusion or topology change).

In this paper, we present an efficient video cutout system to meet this challenge. A novel directional classifier is proposed to handle temporal discontinuities robustly, and then multiple classifiers are incorporated to cover a variety of cases. The outputs of these classifiers are integrated via another classifier, which is learnt from real examples. The foreground matte is solved by a coherent matting procedure, and remaining errors can be removed easily by additive spatio-temporal local editing. Experiments demonstrate that our system performs more robustly and more intelligently than existing systems in dealing with various input types, thus saving a lot of user labor and time.

References

[1]

Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. Graphics 23 (August), 584--591.

Digital Library

[2]

Apostoloff, N., and Fitzgibbon, A. 2004. Bayesian Video Matting Using Learnt Image Priors. In CVPR, 407--414.

[3]

Bai, X., and Sapiro, G. 2007. A geodesic framework for fast interactive image and video segmentation and matting. In ICCV, 1--8.

[4]

Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graphics 28 (July), 70:1--70:11.

Digital Library

[5]

Bai, X., Wang, J., and Sapiro, G. 2010. Dynamic color flow: a motion-adaptive color model for object segmentation in video. In ECCV, 617--630.

Digital Library

[6]

Bilmes, J. 1998. A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Tech. rep., International Computer Science Institute, Berkeley.

[7]

Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. Graph. 21 (July), 243--248.

Digital Library

[8]

Gong, M., Wang, L., Yang, R., and Yang, Y.-H. 2010. Real-Time Video matting using Multichannel Poisson Equations. In Graphics Interface, 89--96.

Digital Library

[9]

Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26 (July).

Digital Library

[10]

Levin, A., Lischinski, D., and Weiss, Y. 2008. A closedform solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 228--242.

Digital Library

[11]

Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Trans. Graphics 24 (July), 595--600.

Digital Library

[12]

Mortensen, E. N., and Barrett, W. A. 1999. Toboggan-Based Intelligent Scissors with a Four Parameter Edge Model. In CVPR.

[13]

Price, B., Cohen, S., and Morse, B. 2009. Livecut: Learning-based interactive video segmentation by evaluation of multiple propagated cues. In ICCV, 779--786.

[14]

Rhemann, C., Rother, C., Rav-Acha, A., and Sharp, T. 2008. High resolution matting via interactive trimap segmentation. In CVPR, 1--8.

[15]

Ross, D. A., Lim, J., Lin, R.-S., and Yang, M.-H. 2008. Incremental learning for robust visual tracking. International Journal of Computer Vision 77 (May), 125--141.

Digital Library

[16]

Sarim, M., Hilton, A., and Guillemaut, J.-Y. 2009. Non-Parametric patch based video matting. In BMVC, British Machine Vision Association, 98.1--98.11.

[17]

Tang, Z., Miao, Z., Wan, Y., and Zhang, D. 2011. Video matting via opacity propagation. The Visual Computer (19 April), 1--15.

Digital Library

[18]

Tong, R.-F., Zhangy, Y., and Ding, M. 2011. Video brush: A novel interface for efficient video cutout. Computer Graphics Forum 30, 2049--2057.

[19]

Vilalta, R., and Drissi, Y. 2002. A perspective view and survey of meta-learning. Artificial Intelligence Review 18, 77--95.

Digital Library

[20]

Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. ACM Trans. Graphics 24 (July), 585--594.

Digital Library

[21]

Yang, L., Sander, P. V., Lawrence, J., and Hoppe, H. 2011. Antialiasing recovery. ACM Trans. Graph. 30 (May), 22:1--22:9.

Digital Library

[22]

Yin, P., Criminisi, A., Winn, J., and Essa, I. 2007. Tree-based classifiers for bilayer video segmentation. In CVPR, 1--8.

[23]

Zhong, F., Qin, X., and Peng, Q. 2010. Transductive segmentation of live video with non-stationary background. In CVPR, 2189--2196.

Cited By

Zhou TPorikli FCrandall DVan Gool LWang W(2023)A Survey on Deep Learning Technique for Video SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.322557345:6(7099-7122)Online publication date: 1-Jun-2023
https://doi.org/10.1109/TPAMI.2022.3225573
Oh SLee JXu NKim S(2020)Space-time Memory Networks for Video Object Segmentation with User GuidanceIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2020.3008917(1-1)Online publication date: 2020
https://doi.org/10.1109/TPAMI.2020.3008917
Vecchio GPalazzo SGiordano DRundo FSpampinato C(2020)MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement LearningIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2019.296328231:12(5103-5115)Online publication date: Dec-2020
https://doi.org/10.1109/TNNLS.2019.2963282
Show More Cited By

Index Terms

Discontinuity-aware video object cutout
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Interest point and salient region detections

Recommendations

Interactive video cutout

We present an interactive system for efficiently extracting foreground objects from a video. We extend previous min-cut based image segmentation techniques to the domain of video with four new contributions. We provide a novel painting-based user ...
Video object cut and paste
SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

In this paper, we present a system for cutting a moving object out from a video clip. The cutout object sequence can be pasted onto another video or a background image. To achieve this, we first apply a new 3D graph cut based segmentation approach on ...
Video object cut and paste

In this paper, we present a system for cutting a moving object out from a video clip. The cutout object sequence can be pasted onto another video or a background image. To achieve this, we first apply a new 3D graph cut based segmentation approach on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 31, Issue 6

November 2012

794 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2366145

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2012

Published in TOG Volume 31, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology of the People's Republic of China
China Postdoctoral Science Foundation
Natural Science Fund for Distinguished Young Scholars of Shandong Province
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
536
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou TPorikli FCrandall DVan Gool LWang W(2023)A Survey on Deep Learning Technique for Video SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.322557345:6(7099-7122)Online publication date: 1-Jun-2023
https://doi.org/10.1109/TPAMI.2022.3225573
Oh SLee JXu NKim S(2020)Space-time Memory Networks for Video Object Segmentation with User GuidanceIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2020.3008917(1-1)Online publication date: 2020
https://doi.org/10.1109/TPAMI.2020.3008917
Vecchio GPalazzo SGiordano DRundo FSpampinato C(2020)MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement LearningIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2019.296328231:12(5103-5115)Online publication date: Dec-2020
https://doi.org/10.1109/TNNLS.2019.2963282
Gui YTian YZeng DXie ZCai Y(2020)Reliable and Dynamic Appearance Modeling and Label Consistency Enforcing for Fast and Coherent Video Object Segmentation With the Bilateral GridIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2019.296126730:12(4781-4795)Online publication date: Dec-2020
https://doi.org/10.1109/TCSVT.2019.2961267
Zhang LLu Y(2020)Video Object Segmentation by Latent Outcome RegressionIEEE Access10.1109/ACCESS.2020.2971964(1-1)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2971964
Oh SLee JXu NKim S(2019)Fast User-Guided Video Object Segmentation by Interaction-And-Propagation Networks2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2019.00539(5242-5251)Online publication date: Jun-2019
https://doi.org/10.1109/CVPR.2019.00539
Li KTao WLiu L(2019)Online Semantic Object Segmentation for Vision Robot Collected VideoIEEE Access10.1109/ACCESS.2019.29334797(107602-107615)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2933479
Lin KJiang NCheong LLu JXu X(2019)Robust Video Background Identification by Dominant Rigid Motion EstimationComputer Vision – ACCV 201810.1007/978-3-030-20890-5_11(163-178)Online publication date: 2-Jun-2019
https://doi.org/10.1007/978-3-030-20890-5_11
Oh SLee JSunkavalli KKim S(2018)Fast Video Object Segmentation by Reference-Guided Mask Propagation2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition10.1109/CVPR.2018.00770(7376-7385)Online publication date: Jun-2018
https://doi.org/10.1109/CVPR.2018.00770
Chen YHao CWu WWu E(2018)Efficient frame-sequential label propagation for video object segmentationMultimedia Tools and Applications10.1007/s11042-017-4520-577:5(6117-6133)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s11042-017-4520-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents