Temporal Coherent Video Super-resolution via Pre-frame-constrained Sparse Reconstruction
Pages 223 - 232
Abstract
In this paper, we extend the sparse representation based image super-resolution method to process videos, mainly aiming at obtaining temporally consistent consecutive high-resolution (HR) video frames. In our formulation, the previous estimated HR frame is used to guide the sparse reconstruction of current low-resolution (LR) frame, which is able to obtain more consistent representations. We show that such guidance is robust and effective by incorporating with a non-rigid dense correspondence based motion compensation schema. We also propose a dictionary updating strategy which regularly updates the dictionaries that are critical for the sparse representation procedure using the newly reconstructed HR frames. To further preserve sharp edges and remove reconstruction errors, once a HR image is recovered, we refine it with a L0-norm based optimization that constrains the final HR output with relatively sparse gradients. Experimental results on natural videos demonstrated the effectiveness of our proposed method.
References
[1]
Aharon Michal, Elad Michael, and Bruckstein Alfred. 2006. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on signal processing 54, 11 (2006), 4311--4322.
[2]
Fernanda Brandi, Ricardo de Queiroz, and Debargha Mukherjee. 2008. Super resolution of video using key frames. In Circuits and Systems, 2008. ISCAS 2008. IEEE International Symposium on. IEEE, 1608--1611.
[3]
Qiqin Dai, Seunghwan Yoo, Armin Kappeler, and Aggelos K Katsaggelos. 2015. Dictionary-based multiple frame video super-resolution. In Image Processing (ICIP), 2015 IEEE International Conference on. IEEE, 83--87.
[4]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2016), 295--307.
[5]
Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision. Springer, 391--407.
[6]
Wei Fan and Dit-Yan Yeung. 2007. Image hallucination using neighbor embedding over visual primitive manifolds. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 1--7.
[7]
Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. 2004. Fast and robust multiframe super resolution. IEEE transactions on image processing 13, 10 (2004), 1327--1344.
[8]
William T Freeman, Thouis R Jones, and Egon C Pasztor. 2002. Example-based super-resolution. IEEE Computer graphics and Applications 22, 2 (2002), 56--65.
[9]
Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM transactions on graphics (TOG) 30, 4 (2011), 70.
[10]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single Image Super-Resolution From Transformed Self-Exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197--5206.
[11]
Kui Jia, Xiaogang Wang, and Xiaoou Tang. 2013. Image transformation based on learning dictionaries across image spaces. IEEE transactions on pattern analysis and machine intelligence 35, 2 (2013), 367--380.
[12]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694--711.
[13]
Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K Katsaggelos. 2016. Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging 2, 2 (2016), 109--122.
[14]
Seung P Kim and W-Y Su. 1993. Recursive high-resolution reconstruction of blurred multiframe images. IEEE Transactions on Image Processing 2, 4 (1993), 534--539.
[15]
Sang-Min Kwak, Joo-Hee Moon, and Jong-Ki Han. 2007. Modified cubic convolution scaler for edge-directed nonuniform data. Optical Engineering 46, 10 (2007), 107001--107001.
[16]
Xiaoguang Li, Kin Man Lam, Guoping Qiu, Lansun Shen, and Suyu Wang. 2009. Example-based image super-resolution with class-specific predictors. Journal of Visual Communication and Image Representation 20, 5 (2009), 312--322.
[17]
Xin Li and Michael T Orchard. 2001. New edge-directed interpolation. IEEE transactions on image processing 10, 10 (2001), 1521--1527.
[18]
Ce Liu and Deqing Sun. 2011. A Bayesian approach to adaptive video super resolution. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 209--216.
[19]
Ce Liu and Deqing Sun. 2014. On Bayesian adaptive video super resolution. IEEE transactions on pattern analysis and machine intelligence 36, 2 (2014), 346--360.
[20]
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2009. Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning. ACM, 689--696.
[21]
Kamal Nasrollahi and Thomas B Moeslund. 2014. Super-resolution: a comprehensive survey. Machine vision and applications 25, 6 (2014), 1423--1468.
[22]
Tomer Peleg and Michael Elad. 2014. A statistical prediction model based on sparse representations for single image super-resolution. IEEE transactions on image processing 23, 6 (2014), 2569--2582.
[23]
Matan Protter, Michael Elad, Hiroyuki Takeda, and Peyman Milanfar. 2009. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions on image processing 18, 1 (2009), 36--51.
[24]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874--1883.
[25]
Byung Cheol Song, Shin-Cheol Jeong, and Yanglim Choi. 2011. Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training. IEEE Transactions on Circuits and Systems for Video Technology 21, 3 (2011), 274--285.
[26]
Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2008. Image super-resolution using gradient profile prior. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 1--8.
[27]
Jian Sun, Nan-Ning Zheng, Hai Tao, and Heung-Yeung Shum. 2003. Image hallucination with primal sketch priors. In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, Vol. 2. IEEE, II--729.
[28]
Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007. Kernel regression for image processing and reconstruction. IEEE Transactions on image processing 16, 2 (2007), 349--366.
[29]
Hiroyuki Takeda, Peyman Milanfar, Matan Protter, and Michael Elad. 2009. Super-resolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing 18, 9 (2009), 1958--1975.
[30]
Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. 2017. Detail-Revealing Deep Video Super-Resolution. In The IEEE International Conference on Computer Vision (ICCV).
[31]
Qing Wang and Rabab Kreidieh Ward. 2007. A new orientation-adaptive interpolation method. IEEE Transactions on Image Processing 16, 4 (2007), 889--900.
[32]
Shenlong Wang, Lei Zhang, Yan Liang, and Quan Pan. 2012. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2216--2223.
[33]
Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L 0 gradient minimization. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 174.
[34]
Jianchao Yang, Zhaowen Wang, Zhe Lin, Scott Cohen, and Thomas Huang. 2012. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing 21, 8 (2012), 3467--3478.
[35]
Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. 2008. Image super-resolution as sparse representation of raw image patches. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 1--8.
[36]
Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. 2010. Image super-resolution via sparse representation. IEEE transactions on image processing 19, 11 (2010), 2861--2873.
[37]
Linwei Yue, Huanfeng Shen, Jie Li, Qiangqiang Yuan, Hongyan Zhang, and Liangpei Zhang. 2016. Image super-resolution: The techniques, applications, and future. Signal Processing 128 (2016), 389--408.
[38]
Haichao Zhang, Jianchao Yang, Yanning Zhang, and Thomas Huang. 2010. Non-local kernel regression for image and video restoration. Computer Vision--ECCV 2010 (2010), 566--579.
[39]
M Zhao, M Bosma, and G Haan. 2007. Making the best of legacy video on modern displays. Journal of the Society for Information Display 15, 1 (2007), 49--60.
Index Terms
- Temporal Coherent Video Super-resolution via Pre-frame-constrained Sparse Reconstruction
Recommendations
Video super-resolution via pre-frame constrained and deep-feature enhanced sparse reconstruction
Highlights- A novel sparse reconstruction formulation is introduced for video super-resolution, which uses the previous estimated high-resolution frame as regularization ...
AbstractThis paper presents a new video super-resolution (SR) method that can generate high-quality and temporally coherent high-resolution (HR) videos. Starting from the traditional sparse reconstruction framework that works well for image SR,...
Image super-resolution via sparse representation
This paper presents a new approach to single-image superresolution, based upon sparse signal representation. Research on image statistics suggests that image patches can be well-represented as a sparse linear combination of elements from an ...
Super Resolution Reconstruction via Multiple Frames Joint Learning
CMSP '11: Proceedings of the 2011 International Conference on Multimedia and Signal Processing - Volume 01This paper presents a novel multi-frame joint learning approach for image super resolution via sparse representation. Based on the assumption that several low-resolution patches degraded from a same high-resolution patch under subpixel translation can ...
Comments
Information & Contributors
Information
Published In
June 2018
284 pages
ISBN:9781450364010
DOI:10.1145/3208159
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 June 2018
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- University Grants Committee (Hong Kong)
- The Science and Technology Project of Guangzhou City
- The National Natural Science Foundation of China (NSFC)
Conference
CGI 2018
Acceptance Rates
CGI 2018 Paper Acceptance Rate 35 of 159 submissions, 22%;
Overall Acceptance Rate 35 of 159 submissions, 22%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 106Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in