research-article

Joint Distortion Restoration and Quality Feature Learning for No-reference Image Quality Assessment

Authors:

Zhongyuan Wang,

Zixiang XiongAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 20, Issue 7

Article No.: 195, Pages 1 - 20

https://doi.org/10.1145/3649899

Published: 27 March 2024 Publication History

Abstract

No-reference image quality assessment (NR-IQA) methods, inspired by the free energy principle, improve the accuracy of image quality prediction by simulating the human brain’s repair process for distorted images. However, existing methods use separate optimization schemes for distortion restoration and quality prediction, which undermines the accurate mapping of feature representations to quality scores. To address this issue, we propose a joint restoration and quality feature learning NR-IQA (RQFL-IQA) method to jointly tackle distortion image restoration and quality prediction within a unified framework. To accurately establish the quality reconstruction relationship between distorted and restored images, a hybrid loss function based on pixel-wise and structure-wise representations is used to improve the restoration capability of the image restoration network. The proposed RQFL-IQA exploits rich labels, including restored images and quality scores, to enable the model to learn more discriminative features and establish a more accurate mapping from feature representation to quality scores. In addition, to avoid the impact of poor restoration on quality prediction, we propose a module with a cleaning function to reweight the fusion of restored and primitive features to achieve more perceptual consistency in feature fusion. Experimental results on public IQA datasets show that the proposed RQFL-IQA is superior over existing methods.

References

[1]

Sewoong Ahn, Yeji Choi, and Kwangjin Yoon. 2021. Deep learning-based distortion sensitivity prediction for full-reference image quality assessment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’21). 344–353.

[2]

Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. 2017. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 27, 1 (2017), 206–219.

[3]

Junming Chen, Haiqiang Wang, Ge Li, and Shan Liu. 2021. Nested error map generation network for no-reference image quality assessment. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’21). 2080–2084.

[4]

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. In European Conference on Computer Vision. Springer, 17–33.

Digital Library

[5]

Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. 2020. Perceptual quality assessment of smartphone photography. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). 3674–3683.

[6]

Deepti Ghadiyaram and Alan C. Bovik. 2015. Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans. Image Process. 25, 1 (2015), 372–387.

Digital Library

[7]

S. Alireza Golestaneh, Saba Dadsetan, and Kris M. Kitani. 2022. No-reference image quality assessment via transformers, relative ranking, and self-consistency. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV’22). 3989–3999.

[8]

Ke Gu, Dacheng Tao, Jun-Fei Qiao, and Weisi Lin. 2018. Learning a no-reference quality assessment model of enhanced images with big data. IEEE Trans. Neural Netw. Learn. Syst. 29, 4 (2018), 1301–1313.

[9]

Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2015. Using free energy principle for blind image quality assessment. IEEE Trans. Multim. 17, 1 (2015), 50–63.

[10]

Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. 2020. KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Trans. Image Process. 29 (2020), 4041–4056.

Digital Library

[11]

Baojin Huang, Zhongyuan Wang, Kui Jiang, Qin Zou, Xin Tian, Tao Lu, and Zhen Han. 2023. Joint segmentation and identification feature learning for occlusion face recognition. IEEE Trans. Neural Netw. Learn. Syst. 34, 12 (2023), 10875–10888.

[12]

Baojin Huang, Zhongyuan Wang, Guangcheng Wang, Zhen Han, and Kui Jiang. 2023. Local eyebrow feature attention network for masked face recognition. ACM Trans. Multim. Comput., Commun. Applic. 19, 3 (2023), 1–19.

Digital Library

[13]

Baojin Huang, Zhongyuan Wang, Guangcheng Wang, Kui Jiang, Zhen Han, Tao Lu, and Chao Liang. 2023. PLFace: Progressive learning for face recognition with mask bias. Pattern Recog. 135 (2023), 109142.

Digital Library

[14]

Ziqing Huang and Shiguang Liu. 2021. Perceptual hashing with visual content understanding for reduced-reference screen content image quality assessment. IEEE Trans. Circ. Syst. Vid. Technol. 31, 7 (2021), 2808–2823.

[15]

Le Kang, Peng Ye, Yi Li, and David Doermann. 2015. Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks. In IEEE International Conference on Image Processing (ICIP’15). 2791–2795.

Digital Library

[16]

Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. 2021. Musiq: Multi-scale image quality transformer. In IEEE/CVF International Conference on Computer Vision. 5148–5157.

[17]

Jongyoo Kim and Sanghoon Lee. 2017. Deep learning of human visual sensitivity in image quality assessment framework. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 1969–1977.

[18]

Woojae Kim, Anh-Duc Nguyen, Sanghoon Lee, and Alan Conrad Bovik. 2020. Dynamic receptive field generation for full-reference image quality assessment. IEEE Trans. Image Process. 29 (2020), 4219–4231.

Digital Library

[19]

Eric Cooper Larson and Damon Michael Chandler. 2010. Most apparent distortion: Full-reference image quality assessment and the role of strategy. J. Electron. Imag. 19, 1 (2010), 011006.

[20]

Fan Li, Yangfan Zhang, and Pamela C. Cosman. 2021. MMMNet: An end-to-end multi-task deep convolution neural network with multi-scale and multi-hierarchy fusion for blind image quality assessment. IEEE Trans. Circ. Syst. Vid. Technol. 31, 12 (2021), 4798–4811.

[21]

Qiaohong Li, Weisi Lin, Jingtao Xu, and Yuming Fang. 2016. Blind image quality assessment using statistical structural and luminance features. IEEE Trans. Multim. 18, 12 (2016), 2457–2469.

Digital Library

[22]

Hanhe Lin, Vlad Hosu, and Dietmar Saupe. 2019. KADID-10k: A large-scale artificially distorted IQA database. In 11th International Conference on Quality of Multimedia Experience (QoMEX’19). 1–3.

[23]

Hanhe Lin, Vlad Hosu, and Dietmar Saupe. 2020. DeepFL-IQA: Weak supervision for deep IQA feature learning. arXiv preprint arXiv:2001.08113 (2020).

[24]

Kwan-Yee Lin and Guanxiang Wang. 2018. Hallucinated-IQA: No-reference image quality assessment via adversarial learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 732–741.

[25]

Weisi Lin and C.-C. Jay Kuo. 2011. Perceptual visual quality metrics: A survey. J. Vis. Commun. Image Represent. 22, 4 (2011), 297–312.

Digital Library

[26]

Xialei Liu, Joost Van De Weijer, and Andrew D. Bagdanov. 2017. RankIQA: Learning from rankings for no-reference image quality assessment. In IEEE International Conference on Computer Vision (ICCV’17). 1040–1049.

[27]

Qing Ma, Junjun Jiang, Xianming Liu, and Jiayi Ma. 2022. Deep unfolding network for spatiospectral image super-resolution. IEEE Trans. Computat. Imag. 8 (2022), 28–40.

[28]

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21, 12 (2012), 4695–4708.

Digital Library

[29]

Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2012. Making a “completely blind” image quality analyzer. IEEE Sig. Process. Lett. 20, 3 (2012), 209–212.

[30]

Anush Krishna Moorthy and Alan Conrad Bovik. 2011. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Trans. Image Process. 20, 12 (2011), 3350–3364.

Digital Library

[31]

Zhaoqing Pan, Feng Yuan, Jianjun Lei, Yuming Fang, Xiao Shao, and Sam Kwong. 2022. VCRNet: Visual compensation restoration network for no-reference image quality assessment. IEEE Trans. Image Process. 31 (2022), 1613–1627.

Digital Library

[32]

Zhaoqing Pan, Hao Zhang, Jianjun Lei, Yuming Fang, Xiao Shao, Nam Ling, and Sam Kwong. 2022. DACNN: Blind image quality assessment via a distortion-aware convolutional neural network. IEEE Trans. Circ. Syst. Vid. Technol. 32, 11 (2022), 7518–7531.

Digital Library

[33]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 721, 8026–8037.

[34]

Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C.-C. Jay Kuo. 2015. Image database TID2013: Peculiarities, results and perspectives. Sig. Process.: Image Commun. 30 (2015), 57–77.

[35]

Hongyu Ren, Diqi Chen, and Yizhou Wang. 2018. RAN4IQA: Restorative adversarial nets for no-reference image quality assessment. In 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence. 7308–7314.

[36]

Soomin Seo, Sehwan Ki, and Munchurl Kim. 2021. A novel just-noticeable-difference-based saliency-channel attention residual network for full-reference image quality predictions. IEEE Trans. Circ. Syst. Vid. Technol. 31, 7 (2021), 2602–2616.

[37]

H. R. Sheikh. 2005. LIVE image quality assessment database release 2. Retrieved from http://live.ece.utexas.edu/research/quality

[38]

Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. 2020. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). 3664–3673.

[39]

Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning. PMLR, 6105–6114.

[40]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).

[41]

Zhaolin Wan, Ke Gu, and Debin Zhao. 2020. Reduced reference stereoscopic image quality assessment using sparse representation and natural scene statistics. IEEE Trans. Multim. 22, 8 (2020), 2024–2037.

[42]

Guangcheng Wang, Quan Shi, Han Wang, Ke Gu, Mengting Wei, Lai Kuan Wong, and Mingxing Wang. 2023. Vision-based PM\(_{2.5}\) concentration estimation with natural scene statistical analysis. IEEE Trans. Artif. Intell. (2023), 1–11. DOI:

[43]

Guangcheng Wang, Zhongyuan Wang, Ke Gu, Kui Jiang, and Zheng He. 2022. Reference-free DIBR-synthesized video quality metric in spatial and temporal domains. IEEE Trans. Circ. Syst. Vid. Technol. 32, 3 (2022), 1119–1132.

[44]

Guangcheng Wang, Zhongyuan Wang, Ke Gu, Leida Li, Zhifang Xia, and Lifang Wu. 2020. Blind quality metric of DIBR-synthesized images in the discrete wavelet transform domain. IEEE Trans. Image Process. 29 (2020), 1802–1814.

Digital Library

[45]

Guangcheng Wang, Zhongyuan Wang, Baojin Huang, Kui Jiang, Zheng He, Hancheng Zhu, Jinsheng Xiao, and Xin Tian. 2022. Two-stage unsupervised facial image quality measurement. Inf. Sci. 611 (2022), 432–445.

Digital Library

[46]

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612.

Digital Library

[47]

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022. Uformer: A general u-shaped transformer for image restoration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’22). IEEE, 17662–17672.

[48]

Long Xu, Jia Li, Weisi Lin, Yongbing Zhang, Lin Ma, Yuming Fang, and Yihua Yan. 2017. Multi-task rank learning for image quality assessment. IEEE Trans. Circ. Syst. Vid. Technol. 27, 9 (2017), 1833–1843.

Digital Library

[49]

Sheng Yang, Qiuping Jiang, Weisi Lin, and Yongtao Wang. 2019. SGDNet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment. In 27th ACM International Conference on Multimedia. 1383–1391.

Digital Library

[50]

Sheng Yang, Guosheng Lin, Qiuping Jiang, and Weisi Lin. 2020. A dilated inception network for visual saliency prediction. IEEE Trans. Multim. 22, 8 (2020), 2163–2176.

[51]

Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. 2022. MANIQA: Multi-dimension attention network for no-reference image quality assessment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1191–1200.

[52]

Yang Yang, Yingqiu Ding, Ming Cheng, and Weiming Zhang. 2023. No-reference quality assessment for contrast-distorted images based on gray and color-gray-difference space. ACM Trans. Multim. Comput., Commun. Applic 19, 2 (2023), 1–20.

Digital Library

[53]

Peng Ye, Jayant Kumar, Le Kang, and David Doermann. 2012. Unsupervised feature learning framework for no-reference image quality assessment. In IEEE Conference on Computer Vision and Pattern Recognition. 1098–1105.

Digital Library

[54]

Yuntong Ye, Yi Chang, Hanyu Zhou, and Luxin Yan. 2021. Closing the loop: Joint rain generation and removal via disentangled image translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). IEEE, 2053–2062.

[55]

Mengzhu Yu, Zhenjun Tang, Xianquan Zhang, Bineng Zhong, and Xinpeng Zhang. 2022. Perceptual hashing with complementary color wavelet transform and compressed sensing for reduced-reference image quality assessment. IEEE Trans. Circ. Syst. Vid. Technol. 32, 11 (2022), 7559–7574.

Digital Library

[56]

Chaofan Zhang, Ziqing Huang, Shiguang Liu, and Jian Xiao. 2022. Dual-channel multi-task CNN for no-reference screen content image quality assessment. IEEE Trans. Circ. Syst. Vid. Technol. 32, 8 (2022), 5011–5025.

Digital Library

[57]

Lin Zhang, Zhongyi Gu, Xiaoxu Liu, Hongyu Li, and Jianwei Lu. 2014. Training quality-aware filters for no-reference image quality assessment. IEEE MultiM. 21, 4 (2014), 67–75.

Cited By

Hu WYang YHu H(2024)Pseudo Label Association and Prototype-Based Invariant Learning for Semi-Supervised NIR-VIS Face RecognitionIEEE Transactions on Image Processing10.1109/TIP.2024.336453033(1448-1463)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3364530
Yang YHu WHu H(2024)Unsupervised NIR-VIS Face Recognition via Homogeneous-to-Heterogeneous Learning and Residual-Invariant EnhancementIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.334617619(2112-2126)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2023.3346176

Index Terms

Joint Distortion Restoration and Quality Feature Learning for No-reference Image Quality Assessment

Recommendations

Auxiliary Information Guided Self-attention for Image Quality Assessment
Image quality assessment (IQA) is an important problem in computer vision with many applications. We propose a transformer-based multi-task learning framework for the IQA task. Two subtasks: constructing an auxiliary information error map and completing ...
No-reference quality assessment of deblocked images

JPEG is the most commonly used image compression standard. In practice, JPEG images are easily subject to blocking artifacts at low bit rates. To reduce the blocking artifacts, many deblocking algorithms have been proposed. However, they also introduce ...
An image quality assessment method based on edge extraction and singular value for blurriness
Abstract
The automatic assessment of perceived image quality is crucial in the field of image processing. To achieve this idea, we propose an image quality assessment (IQA) method for blurriness. The features of gradient and singular value were extracted ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 20, Issue 7

July 2024

973 pages

EISSN:1551-6865

DOI:10.1145/3613662

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 March 2024

Online AM: 28 February 2024

Accepted: 25 February 2024

Revised: 23 November 2023

Received: 26 July 2023

Published in TOMM Volume 20, Issue 7

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Key R&D Program in Hubei Province

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
297
Total Downloads

Downloads (Last 12 months)297
Downloads (Last 6 weeks)28

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hu WYang YHu H(2024)Pseudo Label Association and Prototype-Based Invariant Learning for Semi-Supervised NIR-VIS Face RecognitionIEEE Transactions on Image Processing10.1109/TIP.2024.336453033(1448-1463)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3364530
Yang YHu WHu H(2024)Unsupervised NIR-VIS Face Recognition via Homogeneous-to-Heterogeneous Learning and Residual-Invariant EnhancementIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.334617619(2112-2126)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2023.3346176

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents