Real-Time and Robust Video Stabilization Based on Block-Wised Gradient Features
Authors: 
Xinke Li, Haofan Mo, Fan Wang, Yongming LiAuthors Info & Claims
Xinke Li, Haofan Mo, Fan Wang, Yongming LiAuthors Info & ClaimsPages 1141 - 1151
Abstract
Portable camera devices are commonly used in modern society. However, the videos captured by such consumer electronics cameras usually exhibit heavy shake which affect the consumer experience. Also, consumer cameras often have limited computational resources. Thus, how to efficiently eliminate camera shake has become a hot issue in the field of consumer electronics. This paper proposed a fast video stabilization algorithm based on block-wised gradient features. After extracting the gradient features, the block-based feature descriptors are generated to perform feature point matching. Then, we calculate the affine transformation matrix of two consecutive frames of the video sequence to register the first frame to generate the final stable video. Experiments show that our fast method has excellent real-time performance and stabilization performance. To deal with complex shaky scenes, we further proposed an improved robust method. In our improved algorithm, the camera’s trajectory from the transformation matrix between two frames is extracted and the total camera path is calculated to acquire the final stabilized video. The results of experiments demonstrate that our improved method can perform real-time stabilizing for shaky videos with various complicated scenes.
References
[1]
H. Kim and G. Kim, “Deep neural network-based indoor emergency awareness using contextual information from sound, human activity, and indoor position on mobile device,” IEEE Trans. Consum. Electron., vol. 66, no. 4, pp. 271–278, Nov. 2020. 10.1109/TCE.2020.3015197.
[2]
R. Chatterjee, S. Mazumdar, R. Sherratt, R. Halder, T. Maitra, and D. Giri, “Real-time speech emotion analysis for smart home assistants,” IEEE Trans. Consum. Electron., vol. 67, no. 1, pp. 68–76, Feb. 2021. 10.1109/TCE.2021.3056421.
[3]
S. Liu and Y. Zhang, “Detail-preserving underexposed image enhancement via optimal weighted multi-exposure fusion,” IEEE Trans. Consum. Electron., vol. 65, no. 3, pp. 303–311, Aug. 2019. 10.1109/TCE.2019.2893644.
[4]
Q. Xie, X. Chen, L. Zhang, A. Jiang, and F. Cui, “A robust and efficient video anti-shaking algorithm for low-end smartphone platforms,” IEEE Trans. Consum. Electron., vol. 65, no. 1, pp. 1–10, Aug. 2019. 10.1109/TCE.2018.2883456.
[5]
F. Han, L. Xie, Y. Yin, H. Zhang, G. Chen, and S. Lu, “Video stabilization for camera shoot in mobile devices via inertial-visual state tracking,” IEEE Trans. Mobile Comput., vol. 20, no. 4, pp. 1714–1729, Apr. 2021. 10.1109/TMC.2019.2961313.
[6]
J. Yu and R. Ramamoorthi, “Selfie video stabilization,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 2, pp. 701–711, Feb. 2021. 10.1109/TPAMI.2019.2931897.
[7]
Z. Jianget al., “VADS: Visual attention detection with a smartphone,” in Proc. IEEE Int. Conf. Comput. Commun., San Francisco, CA, USA, 2016, pp. 1–9. 10.1109/INFOCOM.2016.7524398.
[8]
G. Hanning, N. Forslöw, P. Forssén, E. Ringaby, D. Törnqvist, and J. Callmer, “Stabilizing cell phone video using inertial measurement sensors,” in Proc. IEEE Int. Conf. Comput. Vis., Barcelona, Spain, 2011, pp. 1–8. 10.1109/ICCVW.2011.6130215.
[9]
J. Kopf, M. F. Cohen, and R. Szeliski, “First-person hyper-lapse videos,” ACM Trans. Graph., vol. 33, no. 4, pp. 1–10, Jul. 2014. 10.1145/2601097.2601195.
[10]
E. Ringaby and P. Forssén, “Efficient video rectification and stabilisation for cell-phones,” Int. J. Comput. Vis., vol. 96, no. 3, pp. 335–352, Feb. 2012. 10.1007/s11263-011-0465-8.
[11]
C.-C. Chen, “An optical image stabilization using novel ultrasonic linear motor and fuzzy sliding-mode controller for portable digital camcorders,” IEEE Trans. Consum. Electron., vol. 63, no. 4, pp. 343–349, Nov. 2017. 10.1109/TCE.2017.015010.
[12]
J. Dong and H. Liu, “Video stabilization for strict real-time applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 4, pp. 716–724, Apr. 2017. 10.1109/TCSVT.2016.2589860.
[13]
M. Wanget al., “Deep online video stabilization with multi-grid warping transformation learning,” IEEE Trans. Image Process., vol. 28, no. 5, pp. 2283–2292, May 2019. 10.1109/TIP.2018.2884280.
[14]
Z. Deng, D. Yang, X. Zhang, Y. Dong, C. Liu, and Q. Shen, “Real-time image stabilization method based on optical flow and binary point feature matching,” Electronics, vol. 9, no. 9, pp. 198–206, Jan. 2020. 10.3390/electronics9010198.
[15]
J. Yang, D. Schonfeld, C. Chen, and M. Mohamed, “Online video stabilization based on particle filters,” in Proc. Int. Conf. Image Process., Atlanta, GA, USA, 2006, pp. 1545–1548.
[16]
Z. Wang, L. Zhang, and H. Huang, “High-quality real-time video stabilization using trajectory smoothing and mesh-based warping,” IEEE Access, vol. 6, pp. 25157–25166, 2018. 10.1109/ACCESS.2018.2828653.
[17]
W. Guilluy, L. Oudre, and A. Beghdadi, “Video stabilization: Overview, challenges and perspectives,” Signal Process. Image Commun., vol. 90, no. 3, Jan. 2011, Art. no. 10.1016/j.image.2020.116015.
[18]
N. A. Tsoligkas, S. Xalkiadis, D. Xu, and I. French, “A guide to digital image stabilization procedure—An overview,” in Proc. 18th Int. Conf. Syst. Signals Image Process., 2011, pp. 1–4.
[19]
Y.-S. Wang, F. Liu, P.-S. Hsu, and T.-Y. Lee, “Spatially and temporally optimized video stabilization,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 8, pp. 1354–1361, Aug. 2013. 10.1109/TVCG.2013.11.
[20]
Y. J. Koh, C. Lee, and C.-S. Kim, “Video stabilization based on feature trajectory augmentation and selection and robust mesh grid warping,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 5260–5273, Dec. 2015. 10.1109/TIP.2015.2479918.
[21]
K.-Y. Lee, Y.-Y. Chuang, B.-Y. Chen, and M. Ouhyoung, “Video stabilization using robust feature trajectories,” in Proc. IEEE 12th Int. Conf. Comput. Vis., Kyoto, Japan, 2009, pp. 1397–1404. 10.1109/ICCV.2009.5459297.
[22]
A. Goldstein and R. Fattal, “Video stabilization using epipolar geometry,” ACM Trans. Graph., vol. 31, no. 5, pp. 1–10, Aug. 2012. 10.1145/2231816.2231824.
[23]
T. Ma, Y. Nie, Q. Zhang, Z. Zhang, H. Sun, and G. Li, “Effective video stabilization via joint trajectory smoothing and frame warping,” IEEE Trans. Vis. Comput. Graphics, vol. 26, no. 11, pp. 3163–3176, Nov. 2020. 10.1109/TVCG.2019.2923196.
[24]
T. H. Lee, Y. Lee, and B. C. Song, “Fast 3D video stabilization using ROI-based warping,” J. Vis. Commun. Image Represent., vol. 25, no. 5, pp. 943–950, Jul. 2014. 10.1016/j.jvcir.2014.02.011.
[25]
S. Liu, Y. Wang, L. Yuan, J. Bu, P. Tan, and J. Sun, “Video stabilization with a depth camera,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Providence, RI, USA, 2012, pp. 89–95.
[26]
F. Liu, M. Gleicher, H. Jin, and A. Agarwala, “Content-preserving warps for 3D video stabilization,” ACM Trans. Graph., vol. 28, no. 3, pp. 1–9, Jul. 2009. 10.1145/1531326.1531350.
[27]
H. Guo, S. Liu, S. Zhu, H. Shen, and B. Zeng, “View-consistent MeshFlow for stereoscopic video stabilization,” IEEE Trans. Comput. Imag., vol. 4, no. 4, pp. 573–584, Dec. 2018. 10.1109/TCI.2018.2866227.
[28]
G. Ros, J. Guerrero, A. D. Sappa, D. Ponsa, and A. M. Lopez, “VSLAM pose initialization via Lie groups and Lie algebras optimization,” in Proc. IEEE Int. Conf. Robot. Autom., Karlsruhe, Germany, 2013, pp. 5740–5747. 10.1109/ICRA.2013.6631402.
[29]
S.-Z. Xu, J. Hu, M. Wang, T.-J. Mu, and S.-M. Hu, “Deep video stabilization using adversarial networks,” Comput. Graph. Forum, vol. 37, no. 7, pp. 267–276, 2018. 10.1111/cgf.13566.
[30]
J. Choi and I. S. Kweon, “DIFRINT: Deep iterative frame interpolation for full-frame video stabilization,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., Seoul, South Korea, 2019, pp. 3732–3736. 10.1109/ICCVW.2019.00463.
[31]
J. Yu and R. Ramamoorthi, “Robust video stabilization by optimization in CNN weight space,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Long Beach, CA, USA, 2019, pp. 3795–3803. 10.1109/CVPR.2019.00392.
[32]
Y. Xu, J. Zhang, S. Maybank, and D. Tao, “DUT: Learning video stabilization by simply watching unstable videos,” Nov. 2020, arxiv.abs/2011.14574.
[33]
C. Huang, H. Yin, Y. Tai, and C. Tang, “StableNet: Semi-online, multi-scale deep video stabilization,” Jul. 2019, arxiv.abs/1907.10283.
[34]
M. Zhao and Q. Ling, “PWStableNet: Learning pixel-wise warping maps for video stabilization,” IEEE Trans. Image Process., vol. 29, pp. 3582–3595, 2020. 10.1109/TIP.2019.2963380.
[35]
B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. DARPA Image Understand. Workshop, 1981, pp. 674–679.
[36]
Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “Full-frame video stabilization with motion in painting,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 7 pp. 1150–1163, Jul. 2006. 10.1109/TPAMI.2006.141.
[37]
Y. Yang, L. Wang, M. Jiang, and B. Zhang, “Video stabilization based on Savitzky–Golay and L1 norm optimization,” Signal Process., vol. 36, no. 11, pp. 1829–1837, Nov. 2020.
[38]
S. Liu, P. Tan, L. Yuan, J. Sun, and B. Zeng, “Meshflow: Minimum latency online video stabilization,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 800–815. 10.1007/978-3-319-46466-4_48.
[39]
B. Pinto and P. R. Anurenjan, “Video stabilization using speeded up robust features,” in Proc. Int. Conf. Commun. Signal Process., Kerala, India, 2011, pp. 527–531. 10.1109/ICCSP.2011.5739378.
[40]
C. Song, H. Zhao, W. Jing, and H. Zhu, “Robust video stabilization based on particle filtering with weighted feature points,” IEEE Trans. Consum. Electron., vol. 58, no. 2, pp. 570–577, May 2012. 10.1109/TCE.2012.6227462.
[41]
J. Yang, D. Schonfeld, and M. Mohamed, “Robust video stabilization based on particle filter tracking of projected camera motion,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 7, pp. 945–954, Jul. 2009. 10.1109/TCSVT.2009.2020252.
[42]
E. Rosten, R. Porter, and T. Drummond, “Faster and better: A machine learning approach to corner detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 105–119, Jan. 2010. 10.1109/TPAMI.2008.275.
[43]
B. Li, Y. Chen, J. Ren, and L. Cheng, “A fast video stabilization method based on feature matching and histogram clustering,” in Proc. Int. Conf. ITITS, Xian, China, 2015, pp. 315–325. 10.1007/978-3-319-38771-0_31.
[44]
J. Xu, H.-W. Chang, S. Yang, and M. Wang, “Fast feature-based video stabilization without accumulative global motion estimation,” IEEE Trans. Consumer Electron., vol. 58, no. 3, pp. 993–999, Aug. 2012. 10.1109/TCE.2012.6311347.
[45]
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Jan. 2004. 10.1023/B:VISI.0000029664.99615.94.
[46]
P. Torr and A. Zisserman, “MLESAC: A new robust estimator with application to estimating image geometry,” Comput. Vis. Image Understanding, vol. 78, no. 1, pp. 138–156, Apr. 2000. 10.1006/cviu.1999.0832.
[47]
M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981. 10.1016/b978-0-08-051581-6.50070-2.
[48]
Q. Ling, S. Deng, F. Li, Q. Huang, and X. Li, “A feedback-based robust video stabilization method for traffic videos,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 3, pp. 561–572, Mar. 2018. 10.1109/TCSVT.2016.2618934.
[49]
M. Grundmann, V. Kwatra, and I. Essa, “Auto-directed video stabilization with robust L1 optimal camera paths,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Colorado Springs, CO, USA, 2011, pp. 225–232. 10.1109/CVPR.2011.5995525.
[50]
M. Zhao and Q. Ling, “Adaptively meshed video stabilization,” IEEE Trans. Circuits Syst. Video Techn., vol. 31, no. 9, pp. 3504–3517, Sep. 2021. 10.1109/TCSVT.2020.3040753.
Index Terms
- Real-Time and Robust Video Stabilization Based on Block-Wised Gradient Features
Index terms have been assigned to the content through auto-classification.
Recommendations
Video stabilization using epipolar geometry
We present a new video stabilization technique that uses projective scene reconstruction to treat jittered video sequences. Unlike methods that recover the full three-dimensional geometry of the scene, this model accounts for simple geometric relations ...
Video stabilization with a depth camera
CVPR '12: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)Previous video stabilization methods often employ homographies to model transitions between consecutive frames, or require robust long feature tracks. However, the homography model is invalid for scenes with significant depth variations, and feature ...
A fast video stabilization system based on speeded-up robust features
ISVC'11: Proceedings of the 7th international conference on Advances in visual computing - Volume Part IIA fast and efficient video stabilization method based on speeded-up robust features (SURF) is presented in this paper. The SURF features are extracted and tracked in each frame and then refined through Random Sample Consensus (RANSAC) to estimate the ...
Comments
Information & Contributors
Information
Published In
1558-4127 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 17 August 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Sep 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in