research-article

Structure-aware Meta-fusion for Image Super-resolution

Authors:

Yizhou YuAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 18, Issue 2

Article No.: 60, Pages 1 - 25

https://doi.org/10.1145/3477553

Published: 16 February 2022 Publication History

Abstract

There are two main categories of image super-resolution algorithms: distortion oriented and perception oriented. Recent evidence shows that reconstruction accuracy and perceptual quality are typically in disagreement with each other. In this article, we present a new image super-resolution framework that is capable of striking a balance between distortion and perception. The core of our framework is a deep fusion network capable of generating a final high-resolution image by fusing a pair of deterministic and stochastic images using spatially varying weights. To make a single fusion model produce images with varying degrees of stochasticity, we further incorporate meta-learning into our fusion network. Once equipped with the kernel produced by a kernel prediction module, our meta fusion network is able to produce final images at any desired level of stochasticity. Experimental results indicate that our meta fusion network outperforms existing state-of-the-art SISR algorithms on widely used datasets, including PIRM-val, DIV2K-val, Set5, Set14, Urban100, Manga109, and B100. In addition, it is capable of producing high-resolution images that achieve low distortion and high perceptual quality simultaneously.

References

[1]

Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126–135.

[2]

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. 2016. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems. 3981–3989.

Digital Library

[3]

Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem. 2018. Finding tiny faces in the wild with generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 21–30.

[4]

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings British Machine Vision Conference. 135.1–135.10.

[5]

Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. 2018. The 2018 PIRM challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.

[6]

Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6228–6237.

[7]

Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 1. IEEE, I–I.

[8]

Jun-Ho Choi, Jun-Hyuk Kim, Manri Cheon, and Jong-Seok Lee. 2020. Deep learning-based image super-resolution considering quantitative and perceptual quality. Neurocomputing 398 (2020), 347–359.

[9]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295–307.

Digital Library

[10]

Karen Egiazarian, Jaakko Astola, Nikolay Ponomarenko, Vladimir Lukin, Federica Battisti, and Marco Carli. 2006. New full-reference quality metrics based on HVS. In Proceedings of the 2nd International Workshop on Video Processing and Quality Metrics, Vol. 4.

[11]

William T. Freeman, Thouis R. Jones, and Egon C. Pasztor. 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 56–65.

Digital Library

[12]

Weifeng Ge, Bingchen Gong, and Yizhou Yu. 2018. Image super-resolution via deterministic-stochastic synthesis and local statistical rectification. ACM Trans. Graph. 37, 6 (2018), 1–14.

Digital Library

[13]

Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. 2019. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1604–1613.

[14]

Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. 2019. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3897–3906.

[15]

Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. 2019. Meta-SR: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1575–1584.

[16]

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.

[17]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.

[18]

Sheng Li, Fengxiang He, Bo Du, Lefei Zhang, Yonghao Xu, and Dacheng Tao. 2019. Fast spatio-temporal residual network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10522–10531.

[19]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136–144.

[20]

Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2359–2368.

[21]

Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2020. SRFlow: Learning the super-resolution space with normalizing flow. In Proceedings of the European Conference on Computer Vision (ECCV’20).

Digital Library

[22]

Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbeláez, and Luc Van Gool. 2016. Convolutional oriented boundaries. In Proceedings of the European Conference on Computer Vision. Springer, 580–596.

[23]

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV’01), Vol. 2. IEEE, 416–423.

[24]

Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76, 20 (2017), 21811–21838.

Digital Library

[25]

Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2012. Making a “completely blind” image quality analyzer. IEEE Sign. Process. Lett. 20, 3 (2012), 209–212.

[26]

Pablo Navarrete Michelini, Dan Zhu, and Hanwen Liu. 2018. Multi–scale recursive and perception–distortion controllable image super–resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.

[27]

Viorica Pătrăucean, Pierre Gurdjos, and Rafael Grompone Von Gioi. 2012. A parameterless line segment and elliptical arc detector with enhanced ellipse fitting. In Proceedings of the European Conference on Computer Vision. Springer, 572–585.

Digital Library

[28]

Nikolay Ponomarenko, Flavia Silvestri, Karen Egiazarian, Marco Carli, Jaakko Astola, and Vladimir Lukin. 2007. On between-coefficient contrast masking of DCT basis functions. In Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics, Vol. 4.

[29]

Sachin Ravi and Hugo Larochelle. 2016. Optimization as a model for few-shot learning. In International Conference on Learning Representations.

[30]

Jae Woong Soh, Gu Yong Park, Junho Jo, and Nam Ik Cho. 2019. Natural and realistic single image super-resolution with explicit natural manifold discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8122–8131.

[31]

Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.

[32]

Yu-Wing Tai, Shuaicheng Liu, Michael S. Brown, and Stephen Lin. 2010. Super resolution using edge prior and single image detail synthesis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2400–2407.

[33]

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 114–125.

[34]

Radu Timofte, Vincent De Smet, and Luc Van Gool. 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1920–1927.

Digital Library

[35]

Radu Timofte, Vincent De Smet, and Luc Van Gool. 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the Asian Conference on Computer Vision. Springer, 111–126.

[36]

Subeesh Vasu, Nimisha Thekke Madam, and A. N. Rajagopalan. 2018. Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.

[37]

Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. 2021. Exploring sparsity in image super-resolution for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4917–4926.

[38]

Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. 2021. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10581–10590.

[39]

Longguang Wang, Yingqian Wang, Zaiping Lin, Jungang Yang, Wei An, and Yulan Guo. 2021. Learning a single network for scale-arbitrary super-resolution. In Proceedings of the IEEE Conference on International Conference on Computer Vision. 10581–10590.

[40]

Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 606–615.

[41]

Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. 2019. Deep network interpolation for continuous imagery effect transition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1692–1701.

[42]

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV’18).

[43]

Yu-Xiong Wang and Martial Hebert. 2016. Learning to learn: Model regression networks for easy small sample learning. In Proceedings of the European Conference on Computer Vision. Springer, 616–634.

[44]

Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 1395–1403.

Digital Library

[45]

Xuan Xu and Xin Li. 2019. Scan: Spatial color attention networks for real single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 0–0.

[46]

Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. 2008. Image super-resolution as sparse representation of raw image patches. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.

[47]

Deniz Yıldırım and Oğuz Güngör. 2012. A novel image fusion method using IKONOS satellite images. J. Geodesy Geoinf. 1, 1 (2012), 75–83.

[48]

Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711–730.

Digital Library

[49]

Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3217–3226.

[50]

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286–301.

Digital Library

[51]

Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. 2019. Image super-resolution by neural texture transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7982–7991.

Cited By

Lin JWang Y(2024)TSFormer: Tracking Structure Transformer for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3696452Online publication date: 20-Sep-2024
https://doi.org/10.1145/3696452
Yang YQiu SZeng LPan Z(2024)Detail-preserving Joint Image UpsamplingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366524620:8(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3665246
Zhang YDing DMa ZLi Z(2024)A Reconfigurable Framework for Neural Network Based Video In-Loop FilteringACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364046720:6(1-20)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3640467
Show More Cited By

Index Terms

Structure-aware Meta-fusion for Image Super-resolution
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing
  2. Machine learning
    1. Machine learning approaches

Recommendations

Multispectral Image Super-Resolution Using Structure-Guided RGB Image Fusion
Pattern Recognition and Computer Vision
Abstract
Due to hardware limitation, multispectral imaging device usually cannot achieve high spatial resolution. To address the issue, this paper proposes a multispectral image super-resolution algorithm by fusing the low-resolution multispectral image ...
Singular value decomposition based fusion for super-resolution image reconstruction

In this paper, we address a super-resolution problem of generating a high-resolution image from low-resolution images. The proposed super-resolution method consists of three steps: image registration, singular value decomposition (SVD)-based image ...
Deep Residual Attention Network for Spectral Image Super-Resolution
Computer Vision – ECCV 2018 Workshops
Abstract
Spectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 2

May 2022

494 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3505207

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 February 2022

Accepted: 01 July 2021

Revised: 01 July 2021

Received: 01 December 2020

Published in TOMM Volume 18, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Hong Kong PhD Fellowship and Hong Kong Research Grants Council through Research Impact Fund

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
768
Total Downloads

Downloads (Last 12 months)140
Downloads (Last 6 weeks)8

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin JWang Y(2024)TSFormer: Tracking Structure Transformer for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3696452Online publication date: 20-Sep-2024
https://doi.org/10.1145/3696452
Yang YQiu SZeng LPan Z(2024)Detail-preserving Joint Image UpsamplingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366524620:8(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3665246
Zhang YDing DMa ZLi Z(2024)A Reconfigurable Framework for Neural Network Based Video In-Loop FilteringACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364046720:6(1-20)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3640467
Hsu WLin H(2024)Context-detail-aware United Network for Single Image DerainingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363940720:5(1-18)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3639407
Fang JYu YWang ZDing XHu R(2023)An Image Arbitrary-Scale Super-Resolution Network Using Frequency-domain InformationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361637620:3(1-23)Online publication date: 10-Nov-2023
https://dl.acm.org/doi/10.1145/3616376
Zhang YDing GDing DMa ZLi Z(2023)On Content-Aware Post-Processing: Adapting Statistically Learned Models to Dynamic ContentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361292520:1(1-23)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3612925
Jin YWu JWang WYan YJiang JZheng J(2023)Cascading Blend Network for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360895220:1(1-21)Online publication date: 25-Aug-2023
https://dl.acm.org/doi/10.1145/3608952
Huang CPeng XLiu DLu Y(2023)Text Image Super-Resolution Guided by Text Structure and Embedding PriorsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/359592419:6(1-18)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3595924
Wang MXu JZhang LLi JZhang KWang SMa S(2023)Compressed Screen Content Image Super ResolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358996319:6(1-20)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3589963
Gong WZhang YWang WCheng PGonzàlez J(2023)Meta-MMFNet: Meta-learning-based Multi-model Fusion Network for Micro-expression RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353957620:2(1-20)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3539576

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents