A deep learning approach to no-reference image quality assessment for Monte Carlo rendered images
Authors: 
J. Whittle, 
M. W. JonesAuthors Info & Claims
J. Whittle, M. W. JonesAuthors Info & ClaimsPages 23 - 31
Abstract
In Full-Reference Image Quality Assessment (FR-IQA) images are compared with ground truth images that are known to be of high visual quality. These metrics are utilized in order to rank algorithms under test on their image quality performance. Throughout the progress of Monte Carlo rendering processes we often wish to determine whether images being rendered are of sufficient visual quality, without the availability of a ground truth image. In such cases FR-IQA metrics are not applicable and we instead must utilise No-Reference Image Quality Assessment (NR-IQA) measures to make predictions about the perceived quality of unconverged images. In this work we propose a deep learning approach to NR-IQA, trained specifically on noise from Monte Carlo rendering processes, which significantly outperforms existing NR-IQA methods and can produce quality predictions consistent with FR-IQA measures that have access to ground truth images.
References
[1]
{AAB15} Abadi M., Agarwal A., Barham P.: TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. URL: http://tensorflow.org/. 8
[2]
{AB09} Autrusseau F., Babel M.: Subjective quality assessment of lar coded art images, 2009. http://www.irccyn.ec-nantes.fr/autrusse/Databases/. 2
[3]
{Bar17} Barron J. T.: A more general robust loss function. CoRR abs/1701.03077 (2017). URL: http://arxiv.org/abs/1701.03077. 4
[4]
{BK16} Bae S. H., Kim M.: A novel image quality assessment with globally and locally consilient visual quality perception. IEEE Transactions on Image Processing 25, 5 (May 2016), 2392--2406. 1, 2
[5]
{BM98} Bolin M. R., Meyer G. W.: A perceptually based adaptive sampling algorithm. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (1998), ACM, pp. 299--309. 2
[6]
{BMM*16} Bosse S., Maniry D., Múller K., Wiegand T., Samek W.: Deep neural networks for no-reference and full-reference image quality assessment. CoRR abs/1612.01697 (2016). URL: http://arxiv.org/abs/1612.01697. 3
[7]
{BMWS16} Bosses., Maniry D., Wiegand T., Samek W.: A deep neural network for image quality assessment. In 2016 IEEE International Conference on Image Processing (ICIP) (Sept 2016), pp. 3773--3777. 3
[8]
{C*15} Chollet F., et al.: Keras. https://github.com/fchollet/keras, 2015. 8
[9]
{CTE05} Cline D., Talbot J., Egbert P.: Energy redistribution path tracing. ACM Trans. Graph. 24, 3 (jul 2005), 1186--1195. 4
[10]
{Dal93} Daly S.: Digital images and human vision. MIT Press, Cambridge, MA, USA, 1993, ch. The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity, pp. 179--206. URL: http://dl.acm.org/citation.cfm?id=197765.197783. 2
[11]
{FHD*17} Fu X., Huang J., Ding X., Liao Y., Paisley J.: Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing 26, 6 (June 2017), 2944--2956. 3
[12]
{Fra99} Franzen R.: Kodak lossless true color image suite. source: http://r0k.us/graphics/kodak 4 (1999). 2
[13]
{HČA*12} Herzog R., Čadik M., Aydčin T. O., Kim K. I., Myszkowski K., Seidel H.-P.: Norm: No-reference image quality metric for realistic image synthesis. In Computer Graphics Forum (2012), vol. 31, Wiley Online Library, pp. 545--554. 2
[14]
{HSKS11} Horita Y., Shibata K., Kawayoke Y., Sazzad Z. P.: Mict image quality evaluation database. {Online}, http://mict.eng.u-toyama.ac.jp/mictdb.html (2011). 2
[15]
{HZRS15} He K., Zhang X., Ren S., Sun J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR abs/1502.01852 (2015). URL: http://arxiv.org/abs/1502.01852. 4
[16]
{IS15} Ioffe S., Szegedy C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (2015), pp. 448--456. 4
[17]
{Jak10} Jakob W.: Mitsuba renderer, 2010. http://www.mitsuba-renderer.org. 4
[18]
{JM12} Jakob W., Marschner S.: Manifold exploration: a markov chain monte carlo technique for rendering scenes with difficult specular transport. ACM Transactions on Graphics (TOG) 31, 4 (2012), 58. 4
[19]
{Kaj86} Kajiya J. T.: The rendering equation. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques (New York, NY, USA, 1986), SIGGRAPH '86, ACM, pp. 143--150. 2, 4
[20]
{KB14} Kingma D. P., Ba J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014). URL: http://arxiv.org/abs/1412.6980. 4, 5
[21]
{KB15} Kamble V., Bhurchandi K.: No-reference image quality assessment algorithms: A survey. Optik - International Journal for Light and Electron Optics 126, 11--12 (2015), 1090 -- 1097. URL: http://www.sciencedirect.com/science/article/pii/S003040261500145X, 2
[22]
{KBS15} Kalantari N. K., Bako S., Sen P.: A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH 2015) 34, 4 (2015). 3
[23]
{Kel97} Keller A.: Instant radiosity. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques (1997), ACM Press/Addison-Wesley Publishing Co., pp. 49--56. 2
[24]
{KGBE16} Kundu D., Ghadiyaram D., Bovik A. C., Evans B. L.: No-reference image quality assessment for high dynamic range images. In 2016 50th Asilomar Conference on Signals, Systems and Computers (Nov 2016), pp. 1847--1852. 2, 6
[25]
{KS00} Kang S. B., Shum H.-Y.: A review of image-based rendering techniques. Institute of Electrical and Electronics Engineers, Inc. URL: https://www.microsoft.com/en-us/research/publication/a-review-of-image-based-rendering-techniques/. 2
[26]
{KSH12} Krizhevsky A., Sutskever I., Hinton G. E.: Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, Pereira F., Burges C. J. C., Bottou L., Weinberger K. Q., (Eds.). Curran Associates, Inc., 2012, pp. 1097--1105. 3
[27]
{KSKAC02} Kelemen C., Szirmay-Kalos L., Antal G., Csonka F.: A simple and robust mutation strategy for the metropolis light transport algorithm. Computer Graphics Forum 21, 3 (2002), 531--540. 4
[28]
{LBW11} Li C., Bovik A. C., Wu X.: Blind image quality assessment using a general regression neural network. IEEE Transactions on Neural Networks 22, 5 (2011), 793--799. 3
[29]
{LHZ*16} Liu L., Hua Y., Zhao Q., Huang H., Bovik A. C.: Blind image quality assessment by relative gradient statistics and adaboosting neural network. Signal Processing: Image Communication 40 (2016), 1 -- 15. URL: http://www.sciencedirect.com/science/article/pii/S0923596515001708, 3, 6
[30]
{LSD15} Long J., Shelhamer E., Darrell T.: Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3431--3440. 3
[31]
{Lub95} Lubin J.: A visual discrimination model for imaging system design and evaluation. In Vision Models for Target Detection and Recognition: In Memory of 'Arthur Menendez. World Scientific, 1995, pp. 245--283. 2
[32]
{LW93} Lafortune E. P., Willems Y. D.: Bi-directional path tracing. In Proceedings of CompuGraphics (1993), vol. 93, pp. 145--153. 4
[33]
{MB10} Moorthy A. K., Bovik A. C.: A two-step framework for constructing blind image quality indices. IEEE Signal processing letters 17, 5 (2010), 513--516. 6
[34]
{MKRH11} Mantiuk R., Kim K. J., Rempel A. G., Heidrich W.: Hdr-vdp-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on Graphics (TOG) 30, 4(2011), 40. 1
[35]
{MMB12} Mittal A., Moorthy A. K., Bovik A. C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21, 12 (Dec 2012), 4695--4708. 2, 6
[36]
{MRT00} Myszkowski K., Rokita P., Tawara T.: Perception-based fast rendering and antialiasing of walkthrough sequences. IEEE Transactions on Visualization and Computer Graphics 6, 4 (2000), 360--379. 2
[37]
{MSB13} Mittal A., Soundararajan R., Bovik A. C.: Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters 20, 3 (2013), 209--212. 6
[38]
{Mys98} Myszkowski K.: The visible differences predictor: Applications to global illumination problems. In Rendering Techniques' 98. Springer, 1998, pp. 223--236. 2
[39]
{PIL*13} Ponomarenko N., Ieremeiev O., Lukin V., Egiazarian K., Jin L., Astola J., Vozel B., Chehdi K., Carli M., Battisti F., Kuo C. C. J.: Color image database tid2013: Peculiarities and preliminary results. In European Workshop on Visual Information Processing (EUVIP) (June 2013), pp. 106--111. 2
[40]
{PLZ*09} Ponomarenko N., Lukin V., Zelensky A., Egiazarian K., Carli M., Battisti F.: Tid2008-a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics 10, 4 (2009), 30--45. 2
[41]
{RPG99} Ramasubramanian M., Pattanaik S. N., Greenberg D. P.: A perceptually based physical error metric for realistic image synthesis. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (1999), ACM Press/Addison-Wesley Publishing Co., pp. 73--82. 2
[42]
{RSSF02} Reinhard E., Stark M., Shirley P., Ferwerda J.: Photographic tone reproduction for digital images. ACM transactions on graphics (TOG) 21, 3 (2002), 267--276. 4
[43]
{SBC05} Sheikh H. R., Bovik A. C., Cormack L.: No-reference quality assessment using natural scene statistics: Jpeg2000. IEEE Transactions on Image Processing 14, 11 (Nov 2005), 1918--1927. 2, 6
[44]
{SBC10} Saad M. A., Bovik A. C., Charrier C.: A dct statistics-based blind image quality index. IEEE Signal Processing Letters 17, 6 (2010), 583--586. 6
[45]
{SLJ*14} Szegedy C., Liu W., Jia Y., Sermanet P., Reed S., Anguelov D., Erhan D., Vanhoucke V., Rabinovich A.: Going deeper with convolutions, corr abs/1409.4842. arXiv preprint arXiv:1409.4842 (2014). 3
[46]
{SWCB14} Sheikh H. R., Wang Z., Cormack L., Bovik A. C.: LIVE Image Quality Assessment Database Release 2, apr 2014. 2
[47]
{SZ14} Simonyan K., Zisserman A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). 3
[48]
{Vea97} Veach E.: Robust Monte Carlo methods for light transport simulation. PhD thesis, Stanford university, 1997. 4
[49]
{VG97} Veach E., Guibas L. J.: Metropolis light transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (New York, NY, USA, 1997), SIGGRAPH '97, ACM Press/Addison-Wesley Publishing Co., pp. 65--76. 4
[50]
{WBSS04} Wang Z., Bovik A. C., Sheikh H. R., Simoncelli E. P.: Image quality assessment: From error visibility to structural similarity. Trans. Img. Proc. 13, 4 (apr 2004), 600--612. 1, 3
[51]
{WJM17} Whittle J., Jones M. W., Mantiuk R.: Analysis of reported error in monte carlo rendered images. The Visual Computer 33, 6 (2017), 705--713. 1, 2, 3
[52]
{WSB03} Wang Z., Simoncelli E. P., Bovik A. C.: Multiscale structural similarity for image quality assessment. In Signals, Systems and Computers, 2004. Conference Record of the Thirty-Seventh Asilomar Conference on (2003), vol. 2, Ieee, pp. 1398--1402. 1, 2
[53]
{YPG01} Yee H., Pattanaik S., Greenberg D. P.: Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments. ACM Transactions on Graphics (TOG) 20, 1 (2001), 39--65. 2
[54]
{ZSP17} Zhang H., Sindagi V., Patel V. M.: Image de-raining using a conditional generative adversarial network. CoRR abs/1701.05957 (2017). URL: http://arxiv.org/abs/1701.05957. 3
[55]
{ZZC*17} Zhang K., Zuo W., Chen Y., Meng D., Zhang L.: Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing 26, 7 (July 2017), 3142--3155. 3
Index Terms
- A deep learning approach to no-reference image quality assessment for Monte Carlo rendered images
Recommendations
Analysis of reported error in Monte Carlo rendered images
Evaluating image quality in Monte Carlo rendered images is an important aspect of the rendering process as we often need to determine the relative quality between images computed using different algorithms and with varying amounts of computation. The ...
Kurtosis-based no-reference quality assessment of JPEG2000 images
No-reference (NR) image quality assessment (QA) presumes no prior knowledge of reference (distortion-free) images and seeks to quantitatively predict visual quality solely from the distorted images. We develop kurtosis-based NR quality measures for ...
Reduced-reference Perceptual Discrepancy Learning for Image Restoration Quality Assessment
Artificial IntelligenceAbstractImage restoration has been receiving extensive attention owing to its widespread applications, and numerous restoration algorithms have been proposed. However, how to accurately evaluate the performances of image restoration algorithms remains ...
Comments
Information & Contributors
Information
Published In
Publisher
Eurographics Association
Goslar, Germany
Publication History
Published: 13 September 2018
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in