Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Blind visual quality assessment for image super-resolution by convolutional neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image super-resolution aims to increase the resolution of images with good visual experience. Over the past decades, there have been many image super-resolution algorithms proposed for various multimedia processing applications. However, how to evaluate the visual quality of high-resolution images generated by image super-resolution methods is still challenging. In this paper, a Convolutional Neural Network is designed to predict the visual quality of image super-resolution. The proposed network consists of two convolutional layers, two pooling layers including average, min and max pooling, three fully connected layers and one regression layer. The contribution of the proposed method is twofold. The first one is that we propose a the deep convolutional neural network to extract the high-level intrinsic features more effectively than the hand-crafted features for super-resolution images, which can be used to estimate the image quality accurately. The other is that we divide the super-resolution image into small patches, to consider the local information for the visual quality assessment of super-resolution image as well as increase the number of training data for the deep neural network. Experimental results show that the proposed metric can obtain better performance than other existing ones in visual quality assessment of image super-resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Aly HA, Dubois E (2005) Image up-sampling using total-variation regularization with a new observation model. IEEE Trans Image Process 14(10):1647–1659

    Article  Google Scholar 

  2. Burger HC, Schuler C, Harmeling S (2013) Learning How to Combine Internal and External Denoising Methods. Pattern Recognition 24(11):121–130

  3. Chakrabarti A, Rajagopalan AN, Chellappa R (2007) Super-resolution of face images using kernel PCA-based prior. IEEE Trans Multimed 9(4):888–892

    Article  Google Scholar 

  4. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 275–282

  5. Chen Z, Jiang T, Tian Y (2014) Quality assessment for comparing image enhancement algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3003–3010

  6. Dong W, Zhang L, Shi G (2013) Nonlocally centralized sparse representation for image restoration. IEEE Trans Image Process 22(4):1620–30

    Article  MathSciNet  MATH  Google Scholar 

  7. Dong C, Chen CL, He K (2014) Learning a Deep Convolutional Network for Image Super-Resolution. In: European Conference on Computer Vision, pp 184–199

    Google Scholar 

  8. Fang Y, Zeng K, Wang Z, Lin W, Fang Z, Lin CW (2014) Objective quality assessment for image retargeting based on structural similarity. IEEE J Emerging Sel Top Circ Syst 4(1):95–105

    Article  Google Scholar 

  9. Fang Y, Ma K, Wang Z, Lin W, Fang Z, Zhai G (2015) No-reference quality assessment of contrast-distorted images based on natural scene statistics. IEEE Signal Process Lett 22(7):838–842

    Google Scholar 

  10. Fang Y, Yan J, Liu J, Wang S, Li Q, Guo Z (2017) Objective quality assessment of screen content images by uncertainty weighting. IEEE Trans Image Process 26(4):2016–2027

    Article  MathSciNet  Google Scholar 

  11. Freedman G, Fattal R (2010) Image and video upscaling from local self-examples. Acm Trans Graph 30(2):474–484

    Google Scholar 

  12. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In: IEEE International Conference on Computer Vision, pp 349–356

  13. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. J Mach Learn Res 9:249–256

    Google Scholar 

  14. Gu K, Wang S, Zhai G, Ma S, Yang X, Lin W, Zhang W, Gao W (2016) Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure. IEEE Transactions on Multimedia 18(3):432–443

    Article  Google Scholar 

  15. Guo Y, Ding G, Han J et al. (2017) Zero-shot learning with transferred samples. IEEE Trans Image Process 26(7):3277–3290

    Article  MathSciNet  Google Scholar 

  16. Guo Y, Ding G, Liu L et al (2017) Learning to hash with optimized anchor embedding for scalable retrieval. IEEE Trans Image Process PP(99):1–1

    Article  MathSciNet  Google Scholar 

  17. He L, Tao D, Li X, Gao X (2012) Sparse representation for blind image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1146–1153

  18. He L, Qi H, Zaretzki R (2013) Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. IEEE Conf Comput Vis Pattern Recogn 9(4):345–352

    Google Scholar 

  19. Hou W, Gao X, Tao D (2015) Blind image quality assessment via deep learning. IEEE Trans Neural Netw Learn Syst 26(6):1275–1286

    Article  MathSciNet  Google Scholar 

  20. Huang W (2016) A novel disease severity prediction scheme via big pair-wise ranking and learning techniques using image-based personal clinical data. Signal Process 124:233–245

    Article  Google Scholar 

  21. Huang J, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 5197–5206

  22. Huang W, Ding H, Chen G (2018) A novel deep multi-channel residual networks-based metric learning method for moving human localization in video surveillance. Signal Process 142:104–113

    Article  Google Scholar 

  23. Ichigaya A, Nishida Y, Nakasu E (2008) Non reference method for estimating PSNR of MPEG-2 coded video by using DCT coefficients and picture energy. IEEE Trans Circ Syst Video Technol 18(6):817– 826

    Article  Google Scholar 

  24. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift, Computer Science

  25. Jiang J, Hu R, Wang Z (2014) Noise robust face hallucination via locality-constrained representation. IEEE Trans Multimed 16(5):1268–1281

    Article  Google Scholar 

  26. Kang L, Ye P, Li Y, Doermann D (2014) Convolution neural network for no-reference image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1733–1740

  27. Katkovnik V, Foiand A, Egiazarian K, Astola J (2010) From local kernel to nonlocal multiple-model image denoising. Int J Comput Vis 86(1):1–32

    Article  MathSciNet  Google Scholar 

  28. Kim KI, Kwon Y (2010) Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 32(6):1127–1133

    Article  Google Scholar 

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Conference on Neural Information Processing Systems

  30. Legge GE, Foley JM (1980) Contrast masking in human vision. Journal of the Optical Society of America 70(12):1458–1471

    Article  Google Scholar 

  31. Li X (2002) Blind image quality assessment. In: IEEE International Conference on Image Processing

  32. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Transactions on Image Processing 10(10):1521–7

    Article  Google Scholar 

  33. Li C, Bovik A, Wu X (2011) Blind image quality assessment using a general regression neural network. IEEE Trans Neural Netw 22(5):793–799

    Article  Google Scholar 

  34. Li M, Liu J, Ren J, Guo Z (2015) Adaptive general scale interpolation based on weighted autoregressive models. IEEE Trans Circ Syst Video Technol 25(2):200–211

    Article  Google Scholar 

  35. Li Y, Liu J, Yang W, Guo Z (2015) Neighborhood regression for edge-preserving image super-resolution. In: IEEE International Conference on Acoustics, Speech and Signal Processing

  36. Li L, Lin W, Wang X, Yang G, Bahrami K, Kot AC (2016) No-reference image blur assessment based on discrete orthogonal moments. IEEE Trans Cybern 46 (1):39–50

    Article  Google Scholar 

  37. Li Q, Lin W, Xu J, Fang Y (2016) Blind image quality assessment using statistical structural and luminance features. Transactions on Multimedia 18(12):2457–2469

    Article  Google Scholar 

  38. Lin W, Jay Kuo C-C (2011) Perceptual visual quality metrics: a survey. J Vis Commun Image Represent 22(4):297–312

    Article  Google Scholar 

  39. Lin Z, Ding G, Han J et al (2016) Cross-view retrieval via probability-based semantics-preserving hashing. IEEE Trans Cybern PP(99):1–14

    Google Scholar 

  40. Lin Z, Ding G, Han J, Shao L (2017) End-to-end feature-aware label space encoding for multilabel classification with many classes. IEEE Trans Neural Netw Learn Syst 99:1–16

    Google Scholar 

  41. Liu J, Yang W, Zhang X, Guo Z (2017) Retrieval compensated group structured sparsity for image super-resolution. IEEE Trans Multimed 19(2):302–316

    Article  Google Scholar 

  42. Ma C, Yang C, Yang X, Yang M (2016) Learning a No-reference Quality Metric for Single-Image Super-Resolution. The Korea-Japan joint workshop on Frontiers of Computer Vision

  43. Ma K, Duanmu Z, Wu Q et al (2017) Waterloo exploration database: new challenges for image quality assessment models. IEEE Trans Image Process 26 (2):1004–1016

    Article  MathSciNet  Google Scholar 

  44. Mairal J, Bach F, Ponce J (2009) Non-local sparse models for image restoration. IEEE Int Conf Comput Vis 30(2):2272–2279

    Google Scholar 

  45. Marquina A, Osher SJ (2008) Image super-resolution by TV-regularization and bregman iteration. J Sci Comput 37(3):367–382

    Article  MathSciNet  MATH  Google Scholar 

  46. Mittal A, Moorthy AK, Bovik A (2012) No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 24(12):4695–4708

    Article  MathSciNet  MATH  Google Scholar 

  47. Mittal A, Soundararajan R, Bovik A (2013) Making a ’completely blind’ image quality analyzer. IEEE Signal Process Lett 20(3):209–212

    Article  Google Scholar 

  48. Moorthy AK, Bovik A (2011) Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Trans Image Process 20(12):3350–3364

    Article  MathSciNet  MATH  Google Scholar 

  49. Mosseri I, Zontak M, Irani M (2013) Combining the power of Internal and External denoising. IEEE International Conference on Computational Photography 8772(18):1–9

  50. Nasrollahi K, Moeslund TB (2014) Super-resolution: a comprehensive survey. Mach Vis Appl 25(6):1423–1468

    Article  Google Scholar 

  51. Pei S, Chen L (2015) Image quality assessment using human visual DOG model fused with random forest. IEEE Trans Image Process 24(11):3282–3892

    Article  MathSciNet  Google Scholar 

  52. Saad MA, Bovik A, Charrier C (2012) Blind image quality assessment: A natural scene statistics approach in the dct domain. IEEE Trans Image Process 21(8):3339–3352

    Article  MathSciNet  MATH  Google Scholar 

  53. Sun L, Hays J (2012) Super-resolution from internet-scale scene matching. In: IEEE International Conference on Computational Photography, pp 1–12

  54. Sun J, Zhu J, Tappen MF (2010) Context-constrained hallucination for image super-resolution. IEEE Conf Comput Vis Pattern Recogn 26(2):231–238

    Google Scholar 

  55. Sun J, Sun J, Xu Z, Shum HY (2011) Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Transactions on Image Processing

  56. Tang H, Joshi N, Kapoor A (2011) Learning a blind measure of perceptual image quality. In: IEEE Conference on Computer Vision and Pattern Recognition

  57. Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International Conference on Computer Vision, pp 1920–1927

  58. Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, pp 111-126

    Google Scholar 

  59. Timofte R, Rothe R, Gool LV (2015) Seven ways to improve example-based single image super resolution. Computer Science

  60. Timofte R, Smet VD, Gool LV (2016) Semantic super-resolution: When and where is it useful?. Comput Vis Image Underst 142:1–12

    Article  Google Scholar 

  61. Tsai RY, Huang TS (1984) Multipleframe image restoration and registration. In: Advances in Computer Vision and Image Processing

  62. Wang Z, Li Q (2011) Information content weighting for perceptual image quality assessment. IEEE Trans Image Process 20(5):1185–1198

  63. Wang Z, Simoncelli E, Bovik A (2003) Multi-scale structual similarity for image quality assessment. In: IEEE Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers

  64. Wang Z, Bovik A, Sheikh HR, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  65. Wang S, Zhang L, Liang Y (2012) Semi-Coupled Dictionary Learning with Applications to Image Super-Resolution and Photo-Sketch Synthesis. IEEE Conf Comput Vis Pattern Recogn 157(10):2216–2223

    Google Scholar 

  66. Wang Z, Liu D, Yang J (2015) Deep networks for image super-resolution with sparse prior. In: IEEE International Conference on Computer Vision, pp 370–378

  67. Wang Z, Yang Y, Wang Z (2015) Learning super-resolution jointly from external and internal examples. IEEE Trans Image Process 24(11):4359–71

    Article  MathSciNet  Google Scholar 

  68. Wu Q, Li H, Meng F et al (2016) Blind image quality assessment based on multichannel feature fusion and label transfer. IEEE Trans Circ Syst Video Technol 26 (3):425–440

    Article  Google Scholar 

  69. Wu Q, Li H, Meng F et al (2016) No reference image quality assessment metric via multi-domain structural information and piecewise regression. J Vis Commun Image Represent 32(C):205–216

    Google Scholar 

  70. Wu Q, Li H, Wang Z et al (2017) Blind image quality assessment based on rank-order regularized regression. IEEE Trans Multimed PP(99):1–1

    Google Scholar 

  71. Xiong Z, Xu D, Sun X (2013) Example-based super-resolution with soft information and decision. IEEE Trans Multimed 15(6):1458–1465

    Article  Google Scholar 

  72. Xue W, Zhang L, Mou X (2013) Learning without Human Scores for Blind Image Quality Assessment. In: IEEE Conference on Computer Vision and Pattern Recognition

  73. Xue W, Mou X, Zhang L, Bovik A, Feng X (2014) Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features. IEEE Trans Image Process 23(11):4850–4862

    Article  MathSciNet  MATH  Google Scholar 

  74. Yang J, Wright J, Huang TS (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  75. Yang C, Huang J, Yang M (2011) Exploiting self-similarities for single frame super-resolution. In: Asian Conference on Computer Vision, pp 497–510

    Chapter  Google Scholar 

  76. Yang J, Wang Z, Lin Z (2012) Coupled dictionary training for image super-resolution. IEEE Trans Image Process 21(8):3467–78

    Article  MathSciNet  MATH  Google Scholar 

  77. Yang M, Wang Y (2013) A self-learning approach to single image super-resolution. IEEE Trans Multimed 15(3):498–508

    Article  Google Scholar 

  78. Yang CMC-Y, Yang M (2014) Single-image super-resolution: a benchmark. In: European Conference on Computer Vision

  79. Yang S, Liu J, Fang Y, Guo Z (2016) Joint-feature guided depth map super-resolution with face priors. IEEE Transactions on Cybernetics

  80. Yang W, Liu J, Li M, Guo Z (2016) Isophote-constrained autoregressive model with adaptive window extension for image interpolation, IEEE Transactions on Circuit System for Video Technology

  81. Yang W, Deng S, Hu Y, Xing J, Liu J (2017) Real-Time Deep Video SpaTial Resolution UpConversion SysTem (STRUCT++ Demo). In: ACM Multimedia

  82. Yang W, Feng J, Yang J, Zhao F, Liu J, Guo Z, Yan S (2017) Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution. IEEE Transaction on Image Processing

  83. Ye P, Kumar J, Kang L, Doermann DS (2012) Unsupervised feature learning framework for no-reference image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition

  84. Yue H, Sun X, Yang J (2013) Landmark image super-resolution by retrieving web images. IEEE Trans Image Process 22(12):4865–4878

    Article  MathSciNet  MATH  Google Scholar 

  85. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

  86. Zhang L, Zhang L, Bovik A (2015) A feature-enriched completely blind image quality evaluator. IEEE Trans Image Process 24(8):2579–2591

    Article  MathSciNet  Google Scholar 

  87. Zhang Y, Liu J, Yang W, Guo Z (2015) Image super-resolution based on structure-modulated sparse representation. IEEE Trans Image Process 24(9):2797–2810

    Article  MathSciNet  Google Scholar 

  88. Zhang P, Zhuo T, Huang W, Chen K, Kankanhalli M (2017) Online object tracking based on CNN with spatial-temporal saliency guided sampling. Neurocomputing 257:115–127

    Article  Google Scholar 

  89. Zhang Q, Liu Y, Blum S et al (2017) Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review. Information Fusion

  90. Zhu Y, Zhang Y, Yuille AL (2014) Single image super-resolution using deformable patches. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2917–2924

  91. Zhu Z, Guo F, Yu H (2014) Fast single image super-resolution via self-example learning and sparse representation. IEEE Trans Multimed 16(8):2178–2190

    Article  Google Scholar 

  92. Zontak M, Irani M (2011) Internal statistics of a single natural image. IEEE Conf Comput Vis Pattern Recogn 2(7):977–984

    Google Scholar 

  93. Zuo W, Zhang L, Song C, Zhang D (2013) Texture enhanced image denoising via gradient histogram preservation. In: IEEE Conference on Computer Vision and Pattern Recognition

Download references

Acknowledgments

This work was partially funded by the Natural Science Foundation of China under Grant 61571212, and by Natural Science Foundation of Jiangxi Province in China under Grant 20071BBE50068, 20171BCB23048, 20161ACB21014 and Grant GJJ160420.

Author information

Authors and Affiliations

  1. School of Information Technology, Jiangxi University of Finance and Economics, Nanchang, China

    Yuming Fang & Chi Zhang

  2. Institute of Computer Science and Technology, Peking University, Beijing, China

    Wenhan Yang, Jiaying Liu & Zongming Guo

Authors
  1. Yuming Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenhan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiaying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zongming Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zongming Guo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, Y., Zhang, C., Yang, W. et al. Blind visual quality assessment for image super-resolution by convolutional neural network. Multimed Tools Appl 77, 29829–29846 (2018). https://doi.org/10.1007/s11042-018-5805-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5805-z

Keywords

Search

Navigation