research-article

Deep Iterative Frame Interpolation for Full-frame Video Stabilization

Authors:

In So KweonAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 39, Issue 1

Article No.: 4, Pages 1 - 9

https://doi.org/10.1145/3363550

Published: 16 January 2020 Publication History

Abstract

Video stabilization is a fundamental and important technique for higher quality videos. Prior works have extensively explored video stabilization, but most of them involve cropping of the frame boundaries and introduce moderate levels of distortion. We present a novel deep approach to video stabilization that can generate video frames without cropping and low distortion. The proposed framework utilizes frame interpolation techniques to generate in between frames, leading to reduced inter-frame jitter. Once applied in an iterative fashion, the stabilization effect becomes stronger. A major advantage is that our framework is end-to-end trainable in an unsupervised manner. In addition, our method is able to run in near real-time (15 fps). To the best of our knowledge, this is the first work to propose an unsupervised deep approach to full-frame video stabilization. We show the advantages of our method through quantitative and qualitative evaluations comparing to the state-of-the-art methods.

References

[1]

Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, and Ravi Ramamoorthi. 2014. User-assisted video stabilization. In Proceedings of the Computer Graphics Forum, Vol. 33. 61--70.

Digital Library

[2]

Steven Bell, Alejandro Troccoli, and Kari Pulli. 2014. A non-linear filter for gyroscope-based video stabilization. In Proceedings of the European Conference on Computer Vision (ECCV’14). 294--308.

[3]

Chris Buehler, Michael Bosse, and Leonard McMillan. 2001. Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR’01), Vol. 2. 609--614.

[4]

Bing-Yu Chen, Ken-Yi Lee, Wei-Ting Huang, and Jong-Shan Lin. 2008. Capturing intention-based full-frame video stabilization. In Computer Graphics Forum, Vol. 27. 1805--1814.

[5]

Michael L. Gleicher and Feng Liu. 2007. Re-cinematography: Improving the camera dynamics of casual video. In Proceedings of the ACM International Conference on Multimedia. 27--36.

[6]

Amit Goldstein and Raanan Fattal. 2012. Video stabilization using epipolar geometry. ACM Trans. Graph. 31, 5 (2012), 126.

Digital Library

[7]

Ross Goroshin, Michael F. Mathieu, and Yann LeCun. 2015. Learning to linearize under uncertainty. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS’15). 1234--1242.

[8]

Matthias Grundmann, Vivek Kwatra, Daniel Castro, and Irfan Essa. 2012. Calibration-free rolling shutter removal. In Proceedings of the IEEE International Conference on Intelligent Computer Communication and Processing (ICCP’12). 1--8.

[9]

Matthias Grundmann, Vivek Kwatra, and Irfan Essa. 2011. Auto-directed video stabilization with robust l1 optimal camera paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 225--232.

Digital Library

[10]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 770--778.

[11]

Hua Huang, Xiao-Xiang Wei, and Lei Zhang. 2018. Encoding shaky videos by integrating efficient video stabilization. IEEE Trans. Circ. Syst. Vid. Technol. 29, 5 (2018).

[12]

Alexandre Karpenko, David Jacobs, Jongmin Baek, and Marc Levoy. 2011. Digital Video Stabilization and Rolling Shutter Correction Using Gyroscopes. Stanford Tech Report CTSR 2011-03.

[13]

Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3 (2009), 44.

Digital Library

[14]

Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala. 2011. Subspace video stabilization. ACM Trans. Graph. 30, 1 (2011), 4.

Digital Library

[15]

Feng Liu, Yuzhen Niu, and Hailin Jin. 2013a. Joint subspace stabilization for stereoscopic video. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). 73--80.

[16]

Shuaicheng Liu, Mingyu Li, Shuyuan Zhu, and Bing Zeng. 2017. CodingFlow: Enable video coding for video stabilization. IEEE Trans. Image Proc. 26, 7 (2017), 3291--3302.

Digital Library

[17]

Shuaicheng Liu, Ping Tan, Lu Yuan, Jian Sun, and Bing Zeng. 2016. Meshflow: Minimum latency online video stabilization. In Proceedings of the European Conference on Computer Vision (ECCV’16). 800--815.

[18]

Shuaicheng Liu, Yinting Wang, Lu Yuan, Jiajun Bu, Ping Tan, and Jian Sun. 2012. Video stabilization with a depth camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 89--95.

[19]

Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun. 2013b. Bundled camera paths for video stabilization. ACM Trans. Graph. 32, 4 (2013), 78.

Digital Library

[20]

Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun. 2014. Steadyflow: Spatially smooth optical flow for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 4209--4216.

Digital Library

[21]

Gucan Long, Laurent Kneip, Jose M. Alvarez, Hongdong Li, Xiaohu Zhang, and Qifeng Yu. 2016. Learning image matching by simply watching video. In Proceedings of the European Conference on Computer Vision (ECCV’16). 434--450.

[22]

Michael Mathieu, Camille Couprie, and Yann LeCun. 2016. Deep multi-scale video prediction beyond mean square error. In Proceedings of the International Conference on Learning Representations (ICLR’16).

[23]

Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou Tang, and Heung-Yeung Shum. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Anal. Mach. Intell. 28, 7 (2006), 1150--1163.

Digital Library

[24]

Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung. 2015. Phase-based frame interpolation for video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 1410--1418.

[25]

Simon Niklaus and Feng Liu. 2018. Context-aware synthesis for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).

[26]

Simon Niklaus, Long Mai, and Feng Liu. 2017a. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. 3.

[27]

Simon Niklaus, Long Mai, and Feng Liu. 2017b. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).

[28]

Hannes Ovrén and Per-Erik Forssén. 2015. Gyroscope-based video stabilisation with auto-calibration. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’15). 2090--2097.

[29]

F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. 2016. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).

[30]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention. Springer, 234--241.

[31]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from arXiv preprint arXiv:1409.1556 (2014).

[32]

Brandon M. Smith, Li Zhang, Hailin Jin, and Aseem Agarwala. 2009. Light field video stabilization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 341--348.

[33]

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. 2018. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).

[34]

Miao Wang, Guo-Ye Yang, Jin-Kun Lin, Ariel Shamir, Song-Hai Zhang, Shao-Ping Lu, and Shi-Min Hu. 2018. Deep online video stabilization with multi-grid warping transformation learning. IEEE Trans. Image Proc. 28, 5 (2018), 2283--2292.

[35]

Yu-Shuen Wang, Feng Liu, Pu-Sheng Hsu, and Tong-Yee Lee. 2013. Spatially and temporally optimized video stabilization. Proceedings of the IEEE Trans. on Vis. Comput. Graph. 19, 8 (2013), 1354--1361.

Digital Library

[36]

Sen-Zhe Xu, Jun Hu, Miao Wang, Tai-Jiang Mu, and Shi-Min Hu. 2018. Deep video stabilization using adversarial networks. In Proceedings of the Computer Graphics Forum, Vol. 37. 267--276.

[37]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Free-form image inpainting with gated convolution. Retrieved from arXiv preprint arXiv:1806.03589 (2018).

[38]

Zihan Zhou, Hailin Jin, and Yi Ma. 2013. Plane-based content preserving warps for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). 2299--2306.

Digital Library

Cited By

Abramov NEmelyanova YFralenko VKhachumov VKhachumov MShustova MTalalaev A(2024)Intelligent Methods for Forest Fire Detection Using Unmanned Aerial VehiclesFire10.3390/fire70300897:3(89)Online publication date: 15-Mar-2024
https://doi.org/10.3390/fire7030089
Jang JBan YLee K(2024)Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video StabilizationApplied Sciences10.3390/app1410429014:10(4290)Online publication date: 18-May-2024
https://doi.org/10.3390/app14104290
Zhao MWang CLi JJiang Z(2024)Video anomaly detection based on frame memory bank and decoupled asymmetric convolutionsJournal of Electronic Imaging10.1117/1.JEI.33.5.05300633:05Online publication date: 1-Sep-2024
https://doi.org/10.1117/1.JEI.33.5.053006
Show More Cited By

Index Terms

Deep Iterative Frame Interpolation for Full-frame Video Stabilization
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Full-Frame Video Stabilization with Motion Inpainting

Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual ...
Video frame interpolation via optical flow estimation with image inpainting
Abstract
As we all know, video frame rate determines the quality of the video. The higher the frame rate, the smoother the movements in the picture, the clearer the information expressed, and the better the viewing experience for people. Video ...
Fast frame-rate up-conversion of depth video via video coding
MM '11: Proceedings of the 19th ACM international conference on Multimedia

Recent development of depth sensors has facilitated the progress of 2D-plus-depth methods for 3D video representation, for which frame-rate up-conversion (FRUC) of depth video is a critical step. However, due to the computational cost of state-of-the-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 39, Issue 1

February 2020

112 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3366374

Editor:
Marc Alexa
TU Berlin, Germany

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 January 2020

Accepted: 01 September 2019

Revised: 01 July 2019

Received: 01 April 2019

Published in TOG Volume 39, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Korea government (MSIT)
Institute for Information 8 communications Technology Promotion (IITP)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
1,228
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)7

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abramov NEmelyanova YFralenko VKhachumov VKhachumov MShustova MTalalaev A(2024)Intelligent Methods for Forest Fire Detection Using Unmanned Aerial VehiclesFire10.3390/fire70300897:3(89)Online publication date: 15-Mar-2024
https://doi.org/10.3390/fire7030089
Jang JBan YLee K(2024)Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video StabilizationApplied Sciences10.3390/app1410429014:10(4290)Online publication date: 18-May-2024
https://doi.org/10.3390/app14104290
Zhao MWang CLi JJiang Z(2024)Video anomaly detection based on frame memory bank and decoupled asymmetric convolutionsJournal of Electronic Imaging10.1117/1.JEI.33.5.05300633:05Online publication date: 1-Sep-2024
https://doi.org/10.1117/1.JEI.33.5.053006
Kerim ARamos WMarcolino LNascimento EJiang R(2024)Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00678(6916-6925)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00678
Zhou STan WYan B(2024)A Motion Distillation Framework for Video Frame InterpolationIEEE Transactions on Multimedia10.1109/TMM.2023.331497126(3728-3740)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3314971
Liu TWan GBai HKong XTang BWang F(2024)Real-Time Video Stabilization Algorithm Based on SuperPointIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.334284973(1-13)Online publication date: 2024
https://doi.org/10.1109/TIM.2023.3342849
John IYari ZBogucki ASwiatek MChrost HWlodarski MChrapkiewicz RLi J(2024)Unsupervised Deep Learning-Driven Stabilization of Smartphone-Based Quantitative Pupillometry for Mobile Emergency Medicine2024 IEEE International Symposium on Biomedical Imaging (ISBI)10.1109/ISBI56570.2024.10635305(1-5)Online publication date: 27-May-2024
https://doi.org/10.1109/ISBI56570.2024.10635305
Ali MIm EKim DKim T(2024)Harnessing Meta-Learning for Improving Full-Frame Video Stabilization2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01198(12605-12614)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01198
Peng ZYe XZhao WLiu TSun HLi BCao Z(2024)3D Multi-frame Fusion for Video Stabilization2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00717(7507-7516)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00717
Di Salvo EBeghdadi ACattai TCuomo FColonnese S(2024)Boosting UAVs Live Uplink Streaming by Video StabilizationIEEE Access10.1109/ACCESS.2024.345221012(121291-121304)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3452210
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents