Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Image Super-Resolution via Lightweight Attention-Directed Feature Aggregation Network

Published: 06 February 2023 Publication History

Abstract

The advent of convolutional neural networks (CNNs) has brought substantial progress in image super-resolution (SR) reconstruction. However, most SR methods pursue deep architectures to boost performance, and the resulting large model sizes are impractical for real-world applications. Furthermore, they insufficiently explore the internal structural information of image features, disadvantaging the restoration of fine texture details. To solve these challenges, we propose a lightweight architecture based on a CNN named attention-directed feature aggregation network (AFAN), consisting of chained stacking multi-aware attention modules (MAAMs) and a simple channel attention module (SCAM), for image SR. Specifically, in each MAAM, we construct a space-aware attention block (SAAB) and a dimension-aware attention block (DAAB) that individually yield unique three-dimensional modulation coefficients to adaptively recalibrate structural information from an asymmetric convolution residual block (ACRB). The synergistic strategy captures multiple content features that are both space-aware and dimension-aware to preserve more fine-grained details. In addition, to further enhance the accuracy and robustness of the network, SCAM is embedded in the last MAAM to highlight channels with high activated values at low computational load. Comprehensive experiments verify that our proposed network attains high qualitative accuracy while employing fewer parameters and moderate computational requirements, exceeding most state-of-the-art lightweight approaches.

References

[1]
Namhyuk Ahn, Byungkon Kang, and Kyung Ah Sohn. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV’18). 252–268.
[2]
Supratik Banerjee, Cagri Ozcinar, Aakanksha Rana, Aljosa Smolic, and Michael Manzke. 2020. Sub-Pixel Back-Projection Network For Lightweight Single Image Super-Resolution. arXiv:2008.01116 (2020). https://arxiv.org/abs/2008.01116.
[3]
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-line Alberi Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the 23rd British Machine Vision Conference (BMVC’12). British Machine Vision Association, Surrey, 1–10.
[4]
D. Chao, C. L. Chen, and X. Tang. 2016. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision (ECCV’16). 391–407.
[5]
Xuelei Chen, Shiqing Wei, Chao Yi, Lingwei Quan, and Cunyue Lu. 2020. Progressive attentional learning for underwater image super-resolution. In International Conference on Intelligent Robotics and Applications. 233–243.
[6]
J. Choi and M. Kim. 2017. A deep convolutional neural network with selection units for super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 1150–1156.
[7]
Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. 2021. Fast, accurate and lightweight super-resolution with neural architecture search. In International Conference on Pattern Recognition (ICPR’21). 59–64.
[8]
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. 2020. Multi-objective reinforced evolution in mobile neural architecture search. In Computer Vision - ECCV 2020 Workshops, Adrien Bartoli and Andrea Fusiello (Eds.). 99–113.
[9]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision. 184–199.
[10]
Guangwei Gao, Wenjie Li, Juncheng Li, Fei Wu, Huimin Lu, and Yi Yu. 2021. Feature distillation interaction weighting network for lightweight image super-resolution. CoRRw abs/2112.08655 (2021).
[11]
B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau. 2003. Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing 12, 5 (2003), 597–606.
[12]
Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R. Martin, Ming-Ming Cheng, and Shi-Min Hu. 2022. Attention mechanisms in computer vision: A survey. Computational Visual Media 8, 3 (2022), 331–368.
[13]
Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng. 2019. ODE-inspired network design for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). 1732–1741.
[14]
Jie Hu, Li Shen, Gang Sun, and Samuel Albanie. 2017. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 7132–7141.
[15]
Yanting Hu, Jie Li, Yuanfei Huang, and Xinbo Gao. 2020. Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 30, 11 (2020), 3911–3927.
[16]
Jiabin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5179–5206.
[17]
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia. 2024–2032.
[18]
Zheng Hui, Xiumei Wang, and Xinbo Gao. 2018. Fast and accurate single image super-resolution via information distillation network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 723–731.
[19]
Md Jahidul Islam, Sadman Sakib Enan, Peigen Luo, and Junaed Sattar. 2020. Underwater image super-resolution using deep residual multipliers. In IEEE International Conference on Robotics and Automation (ICRA’20). 900–906.
[20]
Zhuqing Jiang, Honghui Zhu, Yue Lu, Guodong Ju, and Aidong Men. 2020. Lightweight super-resolution using deep neural learning. IEEE Transactions on Broadcasting 66, 4 (2020), 814–823.
[21]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 1646–1654.
[22]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 1637–1645.
[23]
Jun Hyuk Kim, Jun Ho Choi, Manri Cheon, and Jong Seok Lee. 2018. RAM: Residual Attention Module for Single Image Super-Resolution. arXiv: 1811.12043 (2018). https://arxiv.org/abs/1811.12043.
[24]
Weisheng Lai, Jiabin Huang, Narendra Ahuja, and Ming Hsuan Yang. 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 624–632.
[25]
Rushi Lan, Long Sun, Zhenbing Liu, Huimin Lu, Cheng Pang, and Xiaonan Luo. 2021. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51, 3 (2021), 1443–1453.
[26]
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 4681–4690.
[27]
Biao Li, Bo Wang, Jiabin Liu, Zhiquan Qi, and Yong Shi. 2020. s-LWSR: Super lightweight super-resolution network. IEEE Transactions on Image Processing 29 (2020), 8368–8380.
[28]
Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. 2018. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 517–532.
[29]
Zhuangzi Li. 2019. Image Super-Resolution Using Attention Based DenseNet with Residual Deconvolution. arXiv:1907.05282 (2019). http://arxiv.org/abs/1907.05282.
[30]
Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. 2019. Feedback network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19).
[31]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 1132–1140.
[32]
Huan Liu, Feilong Cao, Chenglin Wen, and Qinghua Zhang. 2020. Lightweight multi-scale residual networks with attention for image super-resolution. Knowledge-Based Systems 203 (2020), 106103.
[33]
J. Liu, W. Zhang, Y. Tang, J. Tang, and G. Wu. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’20). 2356–2365.
[34]
Yuqing Liu, Qi Jia, Xin Fan, Shanshe Wang, Siwei Ma, and Wen Gao. 2021. Cross-SRN: Structure-preserving super-resolution network with cross convolution. IEEE Transactions on Circuits and Systems for Video Technology (2021), 1–1.
[35]
Shao-Yuan Lo, Hsueh-Ming Hang, Sheng-Wei Chan, and Jing-Jhih Lin. 2019. Efficient dense modules of asymmetric convolution for real-time semantic segmentation. In Proceedings of the ACM Multimedia Asia. ACM, 1–6.
[36]
Tao Lu, Yu Wang, Jiaming Wang, Wei Liu, and Yanduo Zhang. 2021. Single image super-resolution via multi-scale information polymerization network. IEEE Signal Processing Letters 28 (2021), 1305–1309.
[37]
D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2002. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference on Computer Vision (CVPR’02). 416–423.
[38]
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools & Applications 76, 20 (2017), 21811–21838.
[39]
Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou. 2021. Rotate to attend: Convolutional triplet attention module. In IEEE Winter Conference on Applications of Computer Vision (WACV’21). 3138–3147.
[40]
Deqiang Ouyang, Yonghui Zhang, and Jie Shao. 2019. Video-based person re-identification via spatio-temporal attentional and two-stream fusion convolutional networks. Pattern Recognition Letters 117 (2019), 153–160.
[41]
Karam Park, Jae Woong Soh, and Nam Ik Cho. 2021. Dynamic residual self-attention network for lightweight single image super-resolution. IEEE Transactions on Multimedia (2021), 1–1.
[42]
Pejman Rasti, Tõnis Uiboupin, Sergio Escalera, and Gholamreza Anbarjafari. 2016. Convolutional neural network super resolution for face recognition in surveillance monitoring. In International Conference on Articulated Motion and Deformable Objects. 175–184.
[43]
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 1874–1883.
[44]
Wenzhe Shi, Jose Caballero, Christian Ledig, Xiahai Zhuang, and Daniel Rueckert. 2013. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 9–16.
[45]
Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan, and Xiaonan Luo. 2021. Lightweight image super-resolution via weighted multi-scale residual network. IEEE/CAA Journal of Automatica Sinica 8, 7 (2021), 1271–1280.
[46]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 2818–2826.
[47]
Ying Tai, Jian Yang, and Xiaoming Liu. 2017. Image super-resolution via deep recursive residual network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 3147–3155.
[48]
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). 4539–4547.
[49]
Chunwei Tian, Yong Xu, Wangmeng Zuo, Chia-Wen Lin, and David Zhang. 2022. Asymmetric CNN for Image Superresolution. IEEE Transactions on Systems, Man, and Cybernetics: Systems 52, 6 (2022), 3718–3730.
[50]
Chunwei Tian, Ruibin Zhuge, Zhihao Wu, Yong Xu, Wangmeng Zuo, Chen Chen, and Chia-Wen Lin. 2020. Lightweight image super-resolution with enhanced CNN. Knowledge-Based Systems 205 (2020), 106235.
[51]
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming Hsuan Yang, and Qi Guo. 2017. NTIRE 2017 challenge on single image super-resolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 114–125.
[52]
Jin Wan, Hui Yin, Zhihao Liu, Aixin Chong, and Yanting Liu. 2021. Lightweight image super-resolution by multi-scale aggregation. IEEE Transactions on Broadcasting 67, 2 (2021), 372–382.
[53]
Chaofeng Wang, Zheng Li, and Jun Shi. 2019. Lightweight Image Super-Resolution with Adaptive Weighted Learning Network. arXiv:1904.02358 (2019). https://arxiv.org/abs/1904.02358
[54]
Xuehui Wang, Qing Wang, Yuzhi Zhao, Junchi Yan, Lei Fan, and Long Chen. 2020. Lightweight Single-Image Super-Resolution Network with Attentive Auxiliary Feature Learning. In Proceedings of the Asian Conference on Computer Vision (ACCV). 268–285.
[55]
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2019. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV’19) Workshops. 63–79.
[56]
Deyun Wei and Zhaowu Wang. 2022. Channel rearrangement multi-branch network for image super-resolution. Digital Signal Processing 120 (2022), 103254.
[57]
Sanghyun Woo, Jongchan Park, Joon Young Lee, and In So Kweon. 2018. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV’18). 3–19.
[58]
Huapeng Wu, Jie Gui, Jun Zhang, James T. Kwok, and Zhihui Wei. 2021. Pyramidal Dense Attention Networks for Lightweight Image Super-Resolution. arXiv:2106.06996 (2021). https://arxiv.org/abs/2106.06996.
[59]
Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In International Conference on Curves and Surfaces. 711–730.
[60]
Dongyang Zhang, Jie Shao, Xinyao Li, and Heng Tao Shen. 2021. Remote sensing image super-resolution via mixed high-order attention network. IEEE Transactions on Geoscience and Remote Sensing 59, 6 (2021), 5183–5196.
[61]
Huanrong Zhang, Jie Xiao, and Zhi Jin. 2021. Multi-scale image super-resolution via a single extendable deep network. IEEE Journal of Selected Topics in Signal Processing 15, 2 (2021), 253–263.
[62]
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 3929–3938.
[63]
Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2017. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 3262–3271.
[64]
Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1671–1681.
[65]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286–301.
[66]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 2472–2481.
[67]
Yan Zhang, Shangxue Yang, Yemei Sun, Shudong Liu, and Xianguo Li. 2021. Attention-guided multi-path cross-CNN for underwater image super-resolution. Signal, Image and Video Processing 16, 1 (2021), 155–163.
[68]
Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. 2020. Efficient Image Super-Resolution Using Pixel Attention. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 56–72.
[69]
Feiyang Zhu and Qijun Zhao. 2019. Efficient single image super-resolution via hybrid residual feature learning with compact back-projection network. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19) Workshops.

Cited By

View all
  • (2024)A Multi-Attention Feature Distillation Neural Network for Lightweight Single Image Super-ResolutionInternational Journal of Intelligent Systems10.1155/2024/32552332024Online publication date: 15-Feb-2024
  • (2024)Spatiotemporal Inconsistency Learning and Interactive Fusion for Deepfake Video DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3664654Online publication date: 13-May-2024
  • (2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
  • Show More Cited By

Index Terms

  1. Image Super-Resolution via Lightweight Attention-Directed Feature Aggregation Network

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2
    March 2023
    540 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/3572860
    • Editor:
    • Abdulmotaleb El Saddik
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 February 2023
    Online AM: 30 June 2022
    Accepted: 23 June 2022
    Revised: 08 June 2022
    Received: 21 January 2022
    Published in TOMM Volume 19, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Super-resolution
    2. lightweight
    3. attention mechanism
    4. asymmetric convolution
    5. spatial information

    Qualifiers

    • Research-article
    • Refereed

    Funding Sources

    • National Natural Science Foundation of China
    • Science and Technology Project of Jiangxi Provincial Education Department
    • Nanchang Key Laboratory Construction Project

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)206
    • Downloads (Last 6 weeks)37
    Reflects downloads up to 08 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)A Multi-Attention Feature Distillation Neural Network for Lightweight Single Image Super-ResolutionInternational Journal of Intelligent Systems10.1155/2024/32552332024Online publication date: 15-Feb-2024
    • (2024)Spatiotemporal Inconsistency Learning and Interactive Fusion for Deepfake Video DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3664654Online publication date: 13-May-2024
    • (2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
    • (2024)Pseudo Content Hallucination for Unpaired Image CaptioningProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658080(320-329)Online publication date: 30-May-2024
    • (2024)MF2ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364081720:6(1-21)Online publication date: 8-Mar-2024
    • (2024)Dynamic Weighted Adversarial Learning for Semi-Supervised Classification under Intersectional Class MismatchACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363531020:4(1-24)Online publication date: 11-Jan-2024
    • (2024)Deep Modular Co-Attention Shifting Network for Multimodal Sentiment AnalysisACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363470620:4(1-23)Online publication date: 11-Jan-2024
    • (2024)Efficient Video Transformers via Spatial-temporal Token Merging for Action RecognitionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363378120:4(1-21)Online publication date: 11-Jan-2024
    • (2024)Learning a Novel Ensemble Tracker for Robust Visual TrackingIEEE Transactions on Multimedia10.1109/TMM.2023.330793926(3194-3206)Online publication date: 1-Jan-2024
    • (2024)Efficient Dual-Branch Information Interaction Network for Lightweight Image Super-ResolutionIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.345006373(1-11)Online publication date: 2024
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media