research-article

Locally Adaptive Structure and Texture Similarity for Image Quality Assessment

Authors:

Kede MaAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 2483 - 2491

https://doi.org/10.1145/3474085.3475419

Published: 17 October 2021 Publication History

Abstract

The latest advances in full-reference image quality assessment (IQA) involve unifying structure and texture similarity based on deep representations. The resulting Deep Image Structure and Texture Similarity (DISTS) metric, however, makes rather global quality measurements, ignoring the fact that natural photographic images are locally structured and textured across space and scale. In this paper, we describe a locally adaptive structure and texture similarity index for full-reference IQA, which we term A-DISTS. Specifically, we rely on a single statistical feature, namely the dispersion index, to localize texture regions at different scales. The estimated probability (of one patch being texture) is in turn used to adaptively pool local structure and texture measurements. The resulting A-DISTS is adapted to local image content, and is free of expensive human perceptual scores for supervised training. We demonstrate the advantages of A-DISTS in terms of correlation with human data on ten IQA databases and optimization of single image super-resolution methods.

References

[1]

Edward H. Adelson. 2001. On seeing stuff: The perception of materials by humans and machines. In SPIE Human Vision and Electronic Imaging. 1--12.

[2]

Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. 2018. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing, Vol. 27, 1 (2018), 206--219.

[3]

Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, Vol. 39, 3/4 (1952), 324--345.

[4]

Peibei Cao, Zhangyang Wang, and Kede Ma. 2021. Debiased subjective assessment of real-world image enhancement. In IEEE Conference on Computer Vision and Pattern Recognition.

[5]

D. R. Cox and P. A. W Lewis. 1966. The statistical analysis of series of events. The Mathematical Gazette, Vol. 51, 377 (1966), 266--267.

[6]

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. 2020. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).

[7]

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. 2021. Comparison of full-reference image quality models for optimization of image processing systems. International Journal of Computer Vision, Vol. 129 (2021), 1258--1281.

Digital Library

[8]

Zhengfang Duanmu, Wentao Liu, Zhongling Wang, and Zhou Wang. 2021. Quantifying visual image quality: A Bayesian view. Annual Review of Vision Science (2021).

[9]

Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In Conference on Neural Information Processing Systems. 262--270.

Digital Library

[10]

S Alireza Golestaneh, Mahesh M Subedar, and Lina J Karam. 2015. The effect of texture granularity on texture synthesis quality. In Applications of Digital Image Processing XXXVIII, Vol. 9599. 356 -- 361.

[11]

Olivier J Hénaff and Eero P Simoncelli. 2016. Geodesics of learned representations. In International Conference on Learning Representations. 1--10.

[12]

Justin Johnson, Alexandre Alahi, and Fei-Fei Li. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. 694--711.

[13]

Bela Julesz. 1962. Visual pattern discrimination. IRE Transactions on Information Theory, Vol. 8, 2 (1962), 84--92.

[14]

Wei-Sheng Lai, Jia-Bin Huang, Zhe Hu, Narendra Ahuja, and Ming-Hsuan Yang. 2016. A comparative study for single image blind deblurring. In IEEE Conference on Computer Vision and Pattern Recognition. 1701--1709.

[15]

Valero Laparra, Johannes Ballé, Alexander Berardino, and Eero P Simoncelli. 2016. Perceptual image quality assessment using a normalized Laplacian pyramid. Electronic Imaging, Vol. 2016, 16 (2016), 1--6.

[16]

Eric C. Larson and Damon M. Chandler. 2010. Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, Vol. 19, 1 (2010), 1--21.

[17]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshop. 136--144.

[18]

Hanhe Lin, Vlad Hosu, and Dietmar Saupe. 2019. KADID-10k: A large-scale artificially distorted IQA database. In IEEE International Conference on Quality of Multimedia Experience. 1--3.

[19]

Yiming Liu, Jue Wang, Sunghyun Cho, Adam Finkelstein, and Szymon Rusinkiewicz. 2013. A no-reference metric for evaluating the quality of motion deblurring. ACM Transactions on Graphics, Vol. 32, 6 (2013), 175:1--175:12.

Digital Library

[20]

Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. 2017b. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, Vol. 158 (2017), 1--16.

Digital Library

[21]

Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. 2017a. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, Vol. 26, 2 (2017), 1004--1016.

Digital Library

[22]

Xiongkuo Min, Guangtao Zhai, Ke Gu, Yucheng Zhu, Jiantao Zhou, Guodong Guo, Xiaokang Yang, Xinping Guan, and Wenjun Zhang. 2019. Quality evaluation of image dehazing methods using synthetic hazy images. IEEE Transactions on Multimedia, Vol. 21, 9 (2019), 2319--2333.

[23]

Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C.-C. Jay Kuo. 2015. Image database TID2013: Peculiarities, results and perspectives. Signal Processing Image Communication, Vol. 30 (2015), 57--77.

Digital Library

[24]

Javier Portilla and Eero P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, Vol. 40, 1 (2000), 49--70.

Digital Library

[25]

Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. 2018. PieAPP: Perceptual image-error assessment through pairwise preference. In IEEE Conference on Computer Vision and Pattern Recognition. 1808--1817.

[26]

Hamid R. Sheikh and Alan C. Bovik. 2006. Image information and visual quality. IEEE Transactions on Image Processing, Vol. 15, 2 (2006), 430--444.

Digital Library

[27]

Hamid R. Sheikh, Zhou Wang, Alan C. Bovik, and Lawrence Cormack. 2006. Image and video quality assessment research at textLIVE. [Online]. Available: http://live.ece.utexas.edu/research/quality/.

[28]

Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations. 1--14.

[29]

Shishun Tian, Lu Zhang, Luce Morin, and Olivier Déforges. 2018. A benchmark of DIBR synthesized view quality assessment metrics on a new database for immersive media applications. IEEE Transactions on Multimedia, Vol. 21, 5 (2018), 1235--1247.

Digital Library

[30]

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. NTIRE 2017 challenge on single image super-resolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition. 114--125.

[31]

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In European Conference on Computer Vision Workshops. 1--16.

[32]

Zhou Wang. 2016. Objective image quality assessment: Facing the real-world challenges. Electronic Imaging, Vol. 2016, 13 (2016), 1--6.

[33]

Zhou Wang and Alan C. Bovik. 2009. Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE Signal Processing Magazine, Vol. 26, 1 (2009), 98--117.

[34]

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612.

Digital Library

[35]

Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik. 2003. Multiscale structural similarity for image quality assessment. In IEEE Asilomar Conference on Signals, System and Computers. 1398--1402.

[36]

Andrew B Watson. 2000. Visual detection of spatial contrast patterns: Evaluation of five simple models. Optics Express, Vol. 6, 1 (2000), 12--33.

[37]

Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C. Bovik. 2014. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing, Vol. 23, 2 (2014), 684--695.

Digital Library

[38]

Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 3217--3226.

[39]

Lin Zhang, Ying Shen, and Hongyu Li. 2014. VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image Processing, Vol. 23, 10 (2014), 4270--4281.

[40]

Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, Vol. 20, 8 (2011), 2378--2386.

Digital Library

[41]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition. 586--595.

[42]

Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. 2019. RankSRGAN: Generative adversarial networks with ranker for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 3096--3105.

Cited By

Pau DPisani ACandelieri A(2024)Towards Full Forward On-Tiny-Device Learning: A Guided Search for a Randomly Initialized Neural NetworkAlgorithms10.3390/a1701002217:1(22)Online publication date: 5-Jan-2024
https://doi.org/10.3390/a17010022
Sugahara RDu W(2024)NNST-based Image Outpainting via SinGANProceedings of the 2024 10th International Conference on Computing and Artificial Intelligence10.1145/3669754.3669759(26-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3669754.3669759
Ding KZhong RWang ZYu YFang Y(2024)Adaptive Structure and Texture Similarity Metric for Image Quality Assessment and OptimizationIEEE Transactions on Multimedia10.1109/TMM.2023.333320826(5398-5409)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3333208
Show More Cited By

Index Terms

Locally Adaptive Structure and Texture Similarity for Image Quality Assessment
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. General and reference
  1. Cross-computing tools and techniques
    1. Metrics

Recommendations

A reduced-reference quality assessment metric for super-resolution reconstructed images with information gain and texture similarity
Abstract
Super-resolution (SR) image reconstruction has been extensively studied in recent years due to its broad uses in machine vision, medical imaging, remote sensing and monitoring systems. However, evaluating the performance of SR ...
Highlights
- A novel IQA metric for super-resolution(SR) reconstructed images is proposed.
- ...
Content-partitioned structural similarity index for image quality assessment

The assessment of image quality is important in numerous image processing applications. Two prominent examples, the Structural Similarity Image (SSIM) index and Multi-scale Structural Similarity (MS-SSIM) operate under the assumption that human visual ...
Image Quality Assessment Based on Structure and Edge Similarity
ICICTA '11: Proceedings of the 2011 Fourth International Conference on Intelligent Computation Technology and Automation - Volume 02

The quality assessment of images is of fundamental importance for most image applications. Structural similarity measure (SSIM) has been widely used which is very easy to calculate but it has been proved not so good in some cases such as Gaussian noise ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Hong Kong RGC Early Career Scheme

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
437
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)14

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pau DPisani ACandelieri A(2024)Towards Full Forward On-Tiny-Device Learning: A Guided Search for a Randomly Initialized Neural NetworkAlgorithms10.3390/a1701002217:1(22)Online publication date: 5-Jan-2024
https://doi.org/10.3390/a17010022
Sugahara RDu W(2024)NNST-based Image Outpainting via SinGANProceedings of the 2024 10th International Conference on Computing and Artificial Intelligence10.1145/3669754.3669759(26-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3669754.3669759
Ding KZhong RWang ZYu YFang Y(2024)Adaptive Structure and Texture Similarity Metric for Image Quality Assessment and OptimizationIEEE Transactions on Multimedia10.1109/TMM.2023.333320826(5398-5409)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3333208
Liao XWei XZhou MLi ZKwong S(2024)Image Quality Assessment: Measuring Perceptual Degradation via Distribution Measures in Deep Feature SpacesIEEE Transactions on Image Processing10.1109/TIP.2024.340917633(4044-4059)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3409176
Zhou MWang HWei XFeng YLuo JPu HZhao JWang LChu ZWang XFang BShang Z(2024)HDIQA: A Hyper Debiasing Framework for Full Reference Image Quality AssessmentIEEE Transactions on Broadcasting10.1109/TBC.2024.335357370:2(545-554)Online publication date: Jun-2024
https://doi.org/10.1109/TBC.2024.3353573
Lang SLiu XZhou MLuo JPu HZhuang XWang JWei XZhang TFeng YShang Z(2024)A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and ConformerIEEE Transactions on Broadcasting10.1109/TBC.2023.330834970:1(316-324)Online publication date: Mar-2024
https://doi.org/10.1109/TBC.2023.3308349
Liao XWei XZhou MKwong S(2024)Full-Reference Image Quality Assessment: Addressing Content Misalignment Issue by Comparing Order Statistics of Deep FeaturesIEEE Transactions on Broadcasting10.1109/TBC.2023.329483570:1(305-315)Online publication date: Mar-2024
https://doi.org/10.1109/TBC.2023.3294835
Li CChen JZhu AZhao D(2024)Chinese Character Font Generation Based on Diffusion Model2024 5th International Conference on Computer Engineering and Application (ICCEA)10.1109/ICCEA62105.2024.10603853(699-705)Online publication date: 12-Apr-2024
https://doi.org/10.1109/ICCEA62105.2024.10603853
Miyata T(2024)ZEN-IQA: Zero-Shot Explainable and No-Reference Image Quality Assessment With Vision Language ModelIEEE Access10.1109/ACCESS.2024.340272912(70973-70983)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3402729
Tian WSanchez-Azofeifa AKan ZZhao QZhang GWu YJiang K(2024)NR-IQA for UAV hyperspectral image based on distortion constructing, feature screening, and machine learningInternational Journal of Applied Earth Observation and Geoinformation10.1016/j.jag.2024.104130133(104130)Online publication date: Sep-2024
https://doi.org/10.1016/j.jag.2024.104130
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents