Double-Layer Search and Adaptive Pooling Fusion for Reference-Based Image Super-Resolution
Authors: 
Kehua Guo, Liang Chen, Xiangyuan Zhu, Xiaoyan Kui, Jian Zhang, Heyuan ShiAuthors Info & Claims
Kehua Guo, Liang Chen, Xiangyuan Zhu, Xiaoyan Kui, Jian Zhang, Heyuan ShiAuthors Info & ClaimsArticle No.: 15, Pages 1 - 23
Abstract
Reference-based image super-resolution (RefSR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) images by introducing HR reference images. The key step of RefSR is to transfer reference features to LR features. However, existing methods still lack an efficient transfer mechanism, resulting in blurry details in the generated image. In this article, we propose a double-layer search module and an adaptive pooling fusion module group for reference-based image super-resolution, called DLASR. Based on the re-search strategy, the double-layer search module can produce an accurate index map and score map. These two maps are used to filter out accurate reference features, which greatly increases the efficiency of feature transfer in the later stage. Through two continuous feature-enhancement steps, the adaptive pooling fusion module group can transfer more valuable reference features to the corresponding LR features. In addition, a structure reconstruction module is proposed to recover the geometric information of the images, which further improves the visual quality of the generated image. We conduct comparative experiments on a variety of datasets, and the results prove that DLASR achieves significant improvements over other state-of-the-art methods, in terms of quantitative accuracy and qualitative visual effect. The code is available at https://github.com/clttyou/DLASR.
References
[1]
Hussein A. Aly and Eric Dubois. 2005. Image up-sampling using total-variation regularization with a new observation model. IEEE Trans. Image Process. 14, 10 (2005), 1647–1659.
[2]
Vivek Boominathan, Kaushik Mitra, and Ashok Veeraraghavan. 2014. Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In Proceedings of the 2014 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–10.
[3]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 184–199.
[4]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295–307.
[5]
Nacer Farajzadeh and Negin S. Rezaei. 2014. Vehicle logo recognition using image matching and textural features. In Proceedings of the Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields. 82–86.
[6]
Raanan Fattal. 2007. Image upsampling via imposed edge statistics. In ACM SIGGRAPH 2007 Papers. 95–102.
[7]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. arXiv preprint arXiv:1505.07376 (2015).
[8]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014).
[9]
Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. 2020. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5407–5416.
[10]
Robert M. Haralick, Karthikeyan Shanmugam, and Its’ Hak Dinstein. 1973. Textural features for image classification. IEEE Trans. Syst Man.Cybe6 (1973), 610–621.
[11]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.
[12]
Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. 2019. ATTGAN: Facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28, 11 (2019), 5464–5478.
[13]
Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In Proceedings of the 2010 20th International Conference on Pattern Recognition. IEEE, 2366–2369.
[14]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700–4708.
[15]
Jia Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.
[16]
Xin Jin, Jianfeng Xu, Kazuyuki Tasaka, and Zhibo Chen. 2021. Multi-task learning-based all-in-one collaboration framework for degraded image super-resolution. ACM Trans. Multimedia Comput. Commun. 17, 1 (2021), 1–21.
[17]
Diederik P. Kingma and Jimmy Ba. 2014. ADAM: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[18]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.
[19]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423.
[20]
Mengyan Li, Zhaoyu Zhang, Guochen Xie, and Jun Yu. 2020. A deep learning approach for face hallucination guided by facial boundary responses. ACM Trans. Multimedia Comput. Commun. 16, 1 (2020), 1–23.
[21]
Xin Li and Michael T. Orchard. 2001. New edge-directed interpolation. IEEE Trans. Image Process. 10, 10 (2001), 1521–1527.
[22]
Xianguo Li, Yemei Sun, Yanli Yang, and Changyun Miao. 2019. Symmetrical residual connections for single image super-resolution. ACM Trans. Multimedia Comput. Commun. 15, 1 (2019), 1–10.
[23]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136–144.
[24]
Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).
[25]
Qun Liu, Saikat Basu, Sangram Ganguly, Supratik Mukhopadhyay, Robert DiBiano, Manohar Karki, and Ramakrishna Nemani. 2020. Deepsat v2: Feature augmented convolutional neural nets for satellite image classification. Remote Sensing Letters 11, 2 (2020), 156–165.
[26]
Xiangbin Liu, Jiesheng He, Liping Song, Shuai Liu, and Gautam Srivastava. 2021. Medical image classification based on an adaptive size deep learning model. ACM Trans. Multimedia Comput. Commun. 17, 3s (2021), 1–18.
[27]
David G. Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision, Vol. 2. IEEE, 1150–1157.
[28]
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2020. SRFLOW: Learning the super-resolution space with normalizing flow. In Proceedings of the European Conference on Computer Vision. Springer, 715–732.
[29]
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. 2021. Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition 110 (2021), 107332.
[30]
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76, 20 (2017), 21811–21838.
[31]
Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. 2003. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine 20, 3 (2003), 21–36.
[32]
Mehdi S. M. Sajjadi, Bernhard Scholkopf, and Michael Hirsch. 2017. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 4491–4500.
[33]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874–1883.
[34]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[35]
Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2008. Image super-resolution using gradient profile prior. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.
[36]
Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2010. Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans. Image Process. 20, 6 (2010), 1529–1542.
[37]
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops.
[38]
Qing Yan, Yi Xu, Xiaokang Yang, and Truong Q. Nguyen. 2015. Single image superresolution based on gradient profile sharpness. IEEE Trans. Image Process. 24, 10 (2015), 3187–3202.
[39]
Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5791–5800.
[40]
Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep edge guided recurrent residual learning for image super-resolution. IEEE Trans. Image Process. 26, 12 (2017), 5895–5907.
[41]
Huanjing Yue, Xiaoyan Sun, Jingyu Yang, and Feng Wu. 2013. Landmark image super-resolution by retrieving web images. IEEE Trans. Image Process. 22, 12 (2013), 4865–4878.
[42]
Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711–730.
[43]
Dongyang Zhang, Jie Shao, and Heng Tao Shen. 2020. Kernel attention network for single image super-resolution. ACM Trans. Multimedia Comput. Commun. 16, 3 (2020), 1–15.
[44]
Lei Zhang and Xiaolin Wu. 2006. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15, 8 (2006), 2226–2238.
[45]
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. 2019. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3096–3105.
[46]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV). 286–301.
[47]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472–2481.
[48]
Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. 2019. Image super-resolution by neural texture transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7982–7991.
[49]
Mandan Zhao, Gaochang Wu, Yipeng Li, Xiangyang Hao, Lu Fang, and Yebin Liu. 2018. Cross-scale reference-based light field super-resolution. IEEE Trans. Comput. Imaging 4, 3 (2018), 406–418.
[50]
Haitian Zheng, Mengqi Ji, Lei Han, Ziwei Xu, Haoqian Wang, Yebin Liu, and Lu Fang. 2017. Learning cross-scale correspondence and patch-based synthesis for reference-based super-resolution. In Proceedings of the BMVC.
[51]
Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang. 2018. Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In Proceedings of the European Conference on Computer Vision (ECCV). 88–104.
[52]
Zhihang Zhong, Ye Gao, Yinqiang Zheng, and Bo Zheng. 2020. Efficient spatio-temporal recurrent neural network for video deblurring. In Proceedings of the European Conference on Computer Vision. Springer, 191–207.
[53]
Xiangyuan Zhu, Kehua Guo, Hui Fang, Liang Chen, Sheng Ren, and Bin Hu. 2021. Cross view capture for stereo image super-resolution. IEEE Trans. Multimedia (2021).
[54]
Xiangyuan Zhu, Kehua Guo, Sheng Ren, Bin Hu, Min Hu, and Hui Fang. 2021. Lightweight image super-resolution with expectation-maximization attention mechanism. IEEE Trans. Circuits Syst. Video Techn. (2021).
[55]
Yu Zhu, Yanning Zhang, Boyan Bonev, and Alan L. Yuille. 2015. Modeling deformable gradient compositions for single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5417–5425.
Index Terms
- Double-Layer Search and Adaptive Pooling Fusion for Reference-Based Image Super-Resolution
Recommendations
A multi-view references image super-resolution framework for generating the large-FOV and high-resolution image
AbstractThis paper investigates the generation of large field-of-view (FOV) and high-resolution (HR) panorama images for smartphones. Existing techniques like image stitching struggle to produce satisfactory results due to geometry misalignment and ...
Highlights- A novel multi-view references image super-resolution framework (MVRefSR) is proposed to generate the large field-of-view and high-resolution panoramas from a new perspective.
- Two modules, a fusion weight estimation module and a ...
Image super-resolution: use of self-learning and gabor prior
ACCV'12: Proceedings of the 11th Asian conference on Computer Vision - Volume Part IIIRecent approaches on single image super-resolution (SR) have attempted to exploit self-similarity to avoid the use of multiple images. In this paper, we propose an SR method based on self-learning and Gabor prior. Given a low resolution (LR) test image ...
Hyperspectral image super-resolution through clustering-based sparse representation
AbstractPromoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we ...
Comments
Information & Contributors
Information
Published In
January 2024
639 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 August 2023
Online AM: 21 June 2023
Accepted: 13 June 2023
Revised: 07 May 2023
Received: 07 October 2022
Published in TOMM Volume 20, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Natural Science Foundation of China
- Hunan Provincial Science and Technology Plan Project
- National Science Foundation of Hunan Province, China
- National Social Science Fund of China
- Fundamental Research Funds for the Central Universities of the Central South University
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 417Total Downloads
- Downloads (Last 12 months)275
- Downloads (Last 6 weeks)20
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in