research-article

Open access

Video Quality Assessment with Texture Information Fusion for Streaming Applications

Authors:

Vignesh V. Menon,

Prajit T. Rajendran,

Klaus Schoeffmann,

Christian TimmererAuthors Info & Claims

MHV '24: Proceedings of the 3rd Mile-High Video Conference

Pages 1 - 6

https://doi.org/10.1145/3638036.3640798

Published: 14 March 2024 Publication History

Abstract

The rise in video streaming applications has increased the demand for video quality assessment (VQA). In 2016, Netflix introduced Video Multi-Method Assessment Fusion (VMAF), a full reference VQA metric that strongly correlates with perceptual quality, but its computation is time-intensive. We propose a Discrete Cosine Transform (DCT)-energy-based VQA with texture information fusion (VQ-TIF) model for video streaming applications that determines the visual quality of the reconstructed video compared to the original video. VQ-TIF extracts Structural Similarity (SSIM) and spatiotemporal features of the frames from the original and reconstructed videos and fuses them using a long short-term memory (LSTM)-based model to estimate the visual quality. Experimental results show that VQ-TIF estimates the visual quality with a Pearson Correlation Coefficient (PCC) of 0.96 and a Mean Absolute Error (MAE) of 2.71, on average, compared to the ground truth VMAF scores. Additionally, VQ-TIF estimates the visual quality at a rate of 9.14 times faster than the state-of-the-art VMAF implementation, along with an 89.44 % reduction in energy consumption, assuming an Ultra HD (2160p) display resolution.

References

[1]

Weisi Lin and C.-C. Jay Kuo, "Perceptual visual quality metrics: A survey," Journal of Visual Communication and Image Representation, vol. 22, no. 4, pp. 297--312, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1047320311000204

Digital Library

[2]

Abdelhak Bentaleb, Bayan Taani, Ali C. Begen, Christian Timmerer, and Roger Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," IEEE Communications Surveys Tutorials, vol. 21, no. 1, pp. 562--585, 2019. [Online]. Available: https://doi.org/10.1109/COMST.2018.2862938

[3]

Vignesh V Menon, Reza Farahani, Prajit T Rajendran, Samira Afzal, Klaus Schoeffmann, and Christian Timmerer, "Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming," in 2023 International Conference on Visual Communications and Image Processing (VCIP), 2023.

[4]

Vignesh V Menon, Samira Afzal, Prajit T Rajendran, Klaus Schoeffmann, Radu Prodan, and Christian Timmerer, "Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming," 2023.

[5]

Shahi Dost, Faryal Saud, Maham Shabbir, Muhammad Gufran Khan, Muhammad Shahid, and Benny Lovstrom, "Reduced reference image and video quality assessments: review of methods," EURASIP Journal on Image and Video Processing, 2022. [Online]. Available: https://doi.org/10.1186/s13640-021-00578-y

Digital Library

[6]

Onur Keles, M. Akin Yilmaz, A. Murat Tekalp, Cansu Korkmaz, and Zafer Dogan, "On the Computation of PSNR for a Set of Images or Video," in 2021 Picture Coding Symposium (PCS), 2021, pp. 1--5.

[7]

Afshin T. Nasrabadi, Milad A. Shirsavar, Azarnush Ebrahimi, and Mohammed Ghanbari, "Investigating the PSNR calculation methods for video sequences with source and channel distortions," in 2014 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, Jun. 2014, pp. 1--4.

[8]

Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik, "Multiscale structural similarity for image quality assessment," in The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, 2003, pp. 1398--1402 Vol.2.

[9]

Pavan C Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, and Alan C. Bovik, "High Frame Rate Video Quality Assessment using VMAF and Entropic Differences," in 2021 Picture Coding Symposium (PCS), 2021, pp. 1--5. [Online]. Available: https://doi.org/10.1109/PCS50896.2021.9477462

[10]

Reza Rassool, "VMAF reproducibility: Validating a perceptual practical video quality metric," in 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), 2017, pp. 1--2. [Online]. Available: https://doi.org/10.1109/BMSB.2017.7986143

[11]

Vignesh V Menon, Prajit T Rajendran, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, and Christian Timmerer, "JND-aware Two-pass Pertitle Encoding Scheme for Adaptive Live Streaming," IEEE Transactions on Circuits and Systems for Video Technology, pp. 1--1, 2023. [Online]. Available: https://doi.org/10.1109/TCSVT.2023.3290725

Digital Library

[12]

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," in IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600--612.

Digital Library

[13]

Alain Horé and Djemel Ziou, "Image Quality Metrics: PSNR vs. SSIM," in 2010 20th International Conference on Pattern Recognition, 2010, pp. 2366--2369.

[14]

Alexandros Stergiou and Ronald Poppe, "AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling," in IEEE Transactions on Image Processing, vol. 32, 2023, pp. 251--266.

[15]

Alexandre Mercat, Marko Viitanen, and Jarno Vanne, UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development, 2020, p. 297--302. [Online]. Available: https://doi.org/10.1145/3339825.3394937

Digital Library

[16]

Manri Cheon and Jong-Seok Lee, "Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 7, 2018, pp. 1467--1480.

[17]

Li Song, Xun Tang, Wei Zhang, Xiaokang Yang, and Pingjian Xia, "The SJTU 4K Video Sequence Dataset," in 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Jul. 2013, pp. 34--35.

[18]

Shyamprasad Chikkerur, Vijay Sundaram, Martin Reisslein, and Lina J. Karam, "Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison," in IEEE Transactions on Broadcasting, vol. 57, no. 2, 2011, pp. 165--182.

[19]

Shiqi Wang, Xiang Zhang, Siwei Ma, and Wen Gao, "Reduced reference image quality assessment using entropy of primitives," in 2013 Picture Coding Symposium (PCS), 2013, pp. 193--196.

[20]

Xu Wang, Gangyi Jiang, and Mei Yu, "Reduced Reference Image Quality Assessment Based on Contourlet Domain and Natural Image Statistics," in 2009 Fifth International Conference on Image and Graphics, 2009, pp. 45--50.

Digital Library

[21]

Yves Meyer, Wavelets: Algorithms & Applications, 1993.

[22]

Muhammad Shahid, Andreas Rossholm, Benny Lövström, and Hans-Jürgen Zepernick, "No-reference image and video quality assessment: a classification and review of recent approaches," in EURASIP Journal on image and Video Processing, vol. 2014. Springer, 2014 pp. 1--32.

[23]

Vignesh V Menon, Reza Farahani, Prajit T Rajendran, Mohammed Ghanbari, Hermann Hellwagner, and Christian Timmerer, "Transcoding Quality Prediction for Adaptive Video Streaming," in Proceedings of the 2nd Mile-High Video Conference, 2023, p. 103--109. [Online]. Available: https://doi.org/10.1145/3588444.3591012

Digital Library

[24]

Manish Narwaria, Weisi Lin, Ian Vince McLoughlin, Sabu Emmanuel, and Liang-Tien Chia, "Fourier Transform-Based Scalable Image Quality Measure," IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3364--3377, 2012.

Digital Library

[25]

Margaret H. Pinson and Stephen Wolf, "An objective method for combining multiple subjective data sets," in Visual Communications and Image Processing 2003, Jun. 2003, pp. 583--592.

[26]

Zhou Wang and Alan C. Bovik, "Modern Image Quality Assessment," Synthesis Lectures on Image, Video, and Multimedia Processing, vol. 2, no. 1, pp. 1--156, 2006.

[27]

Zhou Wang and Qiang Li, "Information Content Weighting for Perceptual Image Quality Assessment," IEEE Transactions on Image Processing, vol. 20, no. 5, pp. 1185--1198, 2011.

Digital Library

[28]

Zhou Wang and Eero Simoncelli, "Reduce-reference image quality assessment using a wavelet-domain natural image statistic model," Proceedings of SPIE - The International Society for Optical Engineering, vol. 5666, 03 2005.

[29]

Rajiv Soundararajan and Alan C. Bovik, "Video Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing," IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 4, pp. 684--694, 2013.

Digital Library

[30]

Tsung-Jung Liu, Yu-Chieh Lin, Weisi Lin, and C.-C. Jay Kuo, "Visual quality assessment: recent developments, coding applications and future trends," in APSIPA Transactions on Signal and Information Processing, vol. 2. Cambridge University Press, 2013, p. e4.

[31]

Junyong You and Jari Korhonen, "Deep Neural Networks for No-Reference Video Quality Assessment," in 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 2349--2353.

[32]

ITU-T, "P.910: Subjective video quality assessment methods for multimedia applications," Nov. 2021. [Online]. Available: https://www.itu.int/rec/T-REC-P.910-202111-I/en

[33]

Vignesh V Menon, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, and Christian Timmerer, "Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming," in Proceedings of the First International Workshop on Green Multimedia Systems, 2023, p. 16--18. [Online]. Available: https://doi.org/10.1145/3593908.3593942

Digital Library

[34]

N B Harikrishnan, Vignesh V Menon, Manoj S Nair, and Gayathri Narayanan, "Comparative evaluation of image compression techniques," in 2017 International Conference on Algorithms, Methodology, Models and Applications in Emerging Technologies (ICAMMAET), 2017, pp. 1--4.

[35]

Nikhil Ketkar, Introduction to Keras, Oct. 2017, pp. 95--109.

[36]

Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization," in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, 2015.

[37]

Netflix, "VMAF 4K model." [Online]. Available: https://github.com/Netflix/vmaf/blob/master/model/vmaf_4k_v0.6.1.json

[38]

BCG-GAMMA and MILA, "CodeCarbon." [Online]. Available: https://codecarbon.io/

[39]

Alan Jović, Karla Brkić, and N. Bogunović, "A review of feature selection methods with applications," in 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 2015, pp. 1200--1205.

[40]

ITU-T, "Statistical analysis, evaluation and reporting guidelines of quality measurements (ITU-T P.1401)," 2020. [Online]. Available: https://www.itu.int/rec/T-REC-P.1401/en

Cited By

Azimi ZFarahani RMenon VTimmerer CProdan R(2024)Towards ML-Driven Video Encoding Parameter Selection for Quality and Energy Optimization2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598278(80-83)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598278
Menon VHelmrich CWieckowski ABross BMarpe D(2024)Convex-Hull Estimation using Xpsnr for Versatile Video Coding2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10647490(1829-1835)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10647490

Index Terms

Video Quality Assessment with Texture Information Fusion for Streaming Applications
1. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming

Recommendations

Video quality assessment combining structural distortion and human visual system
AICI'12: Proceedings of the 4th international conference on Artificial Intelligence and Computational Intelligence

This paper presents an improved structural similarity index (SSIM) for video quality assessment based on human visual system (HVS). To integrate visual characteristics to our SSIM, different weighted values are determined by those visual characteristics ...
Spatiotemporal Saliency Detection based Video Quality Assessment
ICIMCS'16: Proceedings of the International Conference on Internet Multimedia Computing and Service

The distortion occurring in salient region can be more annoying than that in the other region. In this paper a novel VQA algorithm is proposed based on spatiotemporal saliency detection. Spatiotemporal saliency is detected using Random Walk with Restart ...
Full-Reference SSIM Metric for Video Quality Assessment with Saliency-Based Features
New Trends in Image Analysis and Processing -- ICIAP 2015 Workshops
Abstract
This paper uses models of visual attention in order to estimate the human visual perception and thus improve metrics of Video Quality Assessment. This work reports on the use of the saliency based model in a full-reference structural similarity ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MHV '24: Proceedings of the 3rd Mile-High Video Conference

February 2024

150 pages

ISBN:9798400704932

DOI:10.1145/3638036

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MHV '24

Sponsor:

SIGMM

MHV '24: Mile-High Video Conference

February 11 - 14, 2024

CO, Denver, USA

Upcoming Conference

MHV '25

Sponsor:
sigmm

Mile-High Video Conference

February 18 - 20, 2025

Denver , CO , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
433
Total Downloads

Downloads (Last 12 months)433
Downloads (Last 6 weeks)60

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Azimi ZFarahani RMenon VTimmerer CProdan R(2024)Towards ML-Driven Video Encoding Parameter Selection for Quality and Energy Optimization2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598278(80-83)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598278
Menon VHelmrich CWieckowski ABross BMarpe D(2024)Convex-Hull Estimation using Xpsnr for Versatile Video Coding2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10647490(1829-1835)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10647490

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten