research-article

Video Stabilization Based on Feature Trajectory Augmentation and Selection and Robust Mesh Grid Warping

Authors:

Chang-Su KimAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 24, Issue 12

Pages 5260 - 5273

https://doi.org/10.1109/TIP.2015.2479918

Published: 01 December 2015 Publication History

Abstract

We propose a video stabilization algorithm, which extracts a guaranteed number of reliable feature trajectories for robust mesh grid warping. We first estimate feature trajectories through a video sequence and transform the feature positions into rolling-free smoothed positions. When the number of the estimated trajectories is insufficient, we generate virtual trajectories by augmenting incomplete trajectories using a low-rank matrix completion scheme. Next, we detect feature points on a large moving object and exclude them so as to stabilize camera movements, rather than object movements. With the selected feature points, we set a mesh grid on each frame and warp each grid cell by moving the original feature positions to the smoothed ones. For robust warping, we formulate a cost function based on the reliability weights of each feature point and each grid cell. The cost function consists of a data term, a structure-preserving term, and a regularization term. By minimizing the cost function, we determine the robust mesh grid warping and achieve the stabilization. Experimental results demonstrate that the proposed algorithm reconstructs videos more stably than the conventional algorithms.

References

[1]

Y. J. Koh, J.-Y. Sim, and C.-S. Kim, “Robust video stabilization based on mesh grid warping of rolling-free features,” in Proc. IEEE ICIP, Oct. 2014, pp. 5781–5785.

[2]

J. Nakamura, Ed., Image Sensors and Signal Processing for Digital Still Cameras. Boca Raton, FL, USA: CRC Press, 2005.

Digital Library

[3]

C. Morimoto and R. Chellappa, “Fast electronic digital image stabilization,” in Proc. IEEE ICPR, vol. 3. Aug. 1996, pp. 284–288.

[4]

Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “Full-frame video stabilization with motion inpainting,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 7, pp. 1150–1163, Jul. 2006.

Digital Library

[5]

B.-Y. Chen, K.-Y. Lee, W.-T. Huan, and J.-S. Lin, “Capturing intention-based full-frame video stabilization,” Comput. Graph. Forum, vol. 27, no. 7, pp. 1805–1814, Oct. 2008.

[6]

M. L. Gleicher and F. Liu, “Re-cinematography: Improving the camerawork of casual video,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 5, no. 1, Oct. 2008, Art. ID.

[7]

M. Grundmann, V. Kwatra, and I. Essa, “Auto-directed video stabilization with robust L1 optimal camera paths,” in Proc. IEEE CVPR, Jun. 2011, pp. 225–232.

[8]

M. Grundmann, V. Kwatra, D. Castro, and I. Essa, “Calibration-free rolling shutter removal,” in Proc. IEEE ICCP, Apr. 2012, pp. 1–8.

[9]

S. Liu, L. Yuan, P. Tan, and J. Sun, “Bundled camera paths for video stabilization,” ACM Trans. Graph., vol. 32, no. 4, Jul. 2013, Art. ID.

[10]

K.-Y. Lee, Y.-Y. Chuang, B.-Y. Chen, and M. Ouhyoung, “Video stabilization using robust feature trajectories,” in Proc. IEEE 12th ICCV, Sep. 2009, pp. 1397–1404.

[11]

F. Liu, M. Gleicher, H. Jin, and A. Agarwala, “Content-preserving warps for 3D video stabilization,” ACM Trans. Graph., vol. 28, no. 3, Aug. 2009, Art. ID.

Digital Library

[12]

S. Liu, Y. Wang, L. Yuan, J. Bu, P. Tan, and J. Sun, “Video stabilization with a depth camera,” in Proc. IEEE CVPR, Jun. 2012, pp. 89–95.

[13]

F. Liu, M. Gleicher, J. Wang, H. Jin, and A. Agarwala, “Subspace video stabilization,” ACM Trans. Graph., vol. 30, no. 1, Jan. 2011, Art. ID.

[14]

F. Liu, Y. Niu, and H. Jin, “Joint subspace stabilization for stereoscopic video,” in Proc. IEEE ICCV, Dec. 2013, pp. 73–80.

[15]

A. Goldstein and R. Fattal, “Video stabilization using epipolar geometry,” ACM Trans. Graph., vol. 31, no. 5, Aug. 2012, Art. ID.

[16]

Y.-S. Wang, F. Liu, P.-S. Hsu, and T.-Y. Lee, “Spatially and temporally optimized video stabilization,” IEEE Trans. Vis. Comput. Graphics, vol. 19, no. 8, pp. 1354–1361, Aug. 2013.

Digital Library

[17]

Z. Zhou, H. Jin, and Y. Ma, “Plane-based content preserving warps for video stabilization,” in Proc. IEEE CVPR, Jun. 2013, pp. 2299–2306.

[18]

T. H. Lee, Y.-G. Lee, and B. C. Song, “Fast 3D video stabilization using ROI-based warping,” J. Vis. Commun. Image Represent., vol. 25, no. 5, pp. 943–950, Jul. 2014.

[19]

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge, U.K.: Cambridge Univ. Press, 2000.

Digital Library

[20]

T. Igarashi, T. Moscovich, and J. F. Hughes, “As-rigid-as-possible shape manipulation,” ACM Trans. Graph., vol. 24, no. 3, pp. 1134–1141, Jul. 2005.

Digital Library

[21]

S. Liu, L. Yuan, P. Tan, and J. Sun, “SteadyFlow: Spatially smooth optical flow for video stabilization,” in Proc. IEEE CVPR, Jun. 2014, pp. 4209–4216.

[22]

C.-K. Liang, L.-W. Chang, and H. H. Chen, “Analysis and compensation of rolling shutter effect,” IEEE Trans. Image Process., vol. 17, no. 8, pp. 1323–1330, Aug. 2008.

Digital Library

[23]

S. Baker, E. Bennett, S. B. Kang, and R. Szeliski, “Removing rolling shutter wobble,” in Proc. IEEE CVPR, Jun. 2010, pp. 2392–2399.

[24]

E. Ringaby and P.-E. Forssén, “Efficient video rectification and stabilisation for cell-phones,” Int. J. Comput. Vis., vol. 96, no. 3, pp. 335–352, Feb. 2012.

Digital Library

[25]

A. Karpenko, D. Jacobs, J. Baek, and M. Levoy, “Digital video stabilization and rolling shutter correction using gyroscopes,” Dept. Comput. Sci., Stanford Univ., Stanford, CA, USA, Tech Rep. CTSR 2011-03, Sep. 2011.

[26]

Y. Sun and G. Liu, “Rolling shutter distortion removal based on curve interpolation,” IEEE Trans. Consum. Electron., vol. 58, no. 3, pp. 1045–1050, Aug. 2012.

[27]

J. Shi and C. Tomasi, “Good features to track,” in Proc. IEEE CVPR, Jun. 1994, pp. 593–600.

[28]

N. Srebro and T. Jaakkola, “Weighted low-rank approximation,” in Proc. ICML, 2003, pp. 720–727.

[29]

Video Stabilization. [Online]. Available: http://mcl.korea.ac.kr/research/stabilization, accessed Nov. 1, 2014.

[30]

M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981.

Digital Library

[31]

S. Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inf. Theory, vol. 28, no. 2, pp. 129–137, Mar. 1982.

Digital Library

[32]

S.-H. Lee, J.-H. Kim, K. P. Choi, J.-Y. Sim, and C.-S. Kim, “Video saliency detection based on spatiotemporal feature learning,” in Proc. IEEE ICIP, Oct. 2014, pp. 1120–1124.

[33]

Blender 3D. [Online]. Available: http://www.blender.org, accessed Apr. 25, 2015.

Cited By

Kou TLiu XSun WJia JMin XZhai GLiu NEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)StableVQA: A Deep No-Reference Quality Assessment Model for Video StabilityProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611860(1066-1076)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611860
Wang JLing QLi P(2023)Robust Video Stabilization based on Motion DecompositionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358049819:5(1-24)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3580498
Roberto MMaia HPedrini H(2023)Survey on Digital Video Stabilization: Concepts, Methods, and ChallengesACM Computing Surveys10.1145/349452555:3(1-37)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3494525
Show More Cited By

Index Terms

Video Stabilization Based on Feature Trajectory Augmentation and Selection and Robust Mesh Grid Warping
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Animation
      1. Motion capture
      2. Motion processing

Index terms have been assigned to the content through auto-classification.

Recommendations

A robust low-rank matrix completion based on truncated nuclear norm and Lp-norm
Abstract
The low-rank matrix completion problem has aroused notable attention in various fields, such as engineering and applied sciences. The classical methods approximate the rank minimization problem by minimizing the nuclear norm, therefore obtaining ...
Robust Low-Rank Matrix Completion by Riemannian Optimization
^† Special Section on Two Themes: CSE Software and Big Data in CSE

Low-rank matrix completion is the problem where one tries to recover a low-rank matrix from noisy observations of a subset of its entries. In this paper, we propose RMC, a new method to deal with the problem of robust low-rank matrix completion, i.e., ...
Topological interference management via low-rank tensor completion for time-varying topology networks
Highlights
- A novel low-rank tensor completion model for the TIM problem in the time-varying network is proposed.
Abstract
In this paper, we introduce a tensor-based topological interference management framework, in which a novel low-rank tensor completion (LRTC) model is proposed to manage interferences in time-varying topology networks. We use particular ...

Comments

Information & Contributors

Information

Published In

Copyright © 2015.

Publisher

IEEE Press

Publication History

Published: 01 December 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kou TLiu XSun WJia JMin XZhai GLiu NEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)StableVQA: A Deep No-Reference Quality Assessment Model for Video StabilityProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611860(1066-1076)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611860
Wang JLing QLi P(2023)Robust Video Stabilization based on Motion DecompositionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358049819:5(1-24)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3580498
Roberto MMaia HPedrini H(2023)Survey on Digital Video Stabilization: Concepts, Methods, and ChallengesACM Computing Surveys10.1145/349452555:3(1-37)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3494525
Ye JPan EXu W(2023)Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing RobotsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.330558834:4(2591-2603)Online publication date: 15-Aug-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3305588
Li XMo HWang FLi Y(2023)Real-Time and Robust Video Stabilization Based on Block-Wised Gradient FeaturesIEEE Transactions on Consumer Electronics10.1109/TCE.2023.330595369:4(1141-1151)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TCE.2023.3305953
Wang YHuang QJiang CLiu JShang MMiao Z(2023)Video stabilizationNeurocomputing10.1016/j.neucom.2022.10.008516:C(205-230)Online publication date: 7-Jan-2023
https://dl.acm.org/doi/10.1016/j.neucom.2022.10.008
Ren H(2023)Sports video athlete detection based on deep learningNeural Computing and Applications10.1007/s00521-022-07077-935:6(4201-4210)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1007/s00521-022-07077-9

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents