No-reference Video Quality Assessment Based on Spatio-temporal Perception Feature Fusion
Authors: 
Yaya Tan, Guangqian Kong, Xun Duan, Huiyun Long, Yun WuAuthors Info & Claims
Yaya Tan, Guangqian Kong, Xun Duan, Huiyun Long, Yun WuAuthors Info & ClaimsPages 1317 - 1335
Abstract
Quality assessment of real, user-generated content videos lacking reference videos is a challenging problem. For such scenarios, we propose an objective quality assessment method for no-reference video from the spatio-temporal perception characteristics of the video. First, a dual-branch network is constructed from distorted video frames and frame difference maps generated from a global perspective, considering the interaction between spatial and temporal information, incorporating a motion-guided attention module, and fusing spatio-temporal perceptual features from a multiscale perspective. Second, an InceptionTime network is introduced to further perform long-term sequence fusion to obtain the final perceptual quality score. Finally, the results were evaluated on the four user-generated content video databases of KoNViD-1k, CVD2014, LIVE_VQC and LIVE_Qualcomm, and the experimental results show that the network outperforms other partially recent no-reference VQA methods.
References
[1]
Chen S, Zheng B, and Li J A method of image quality assessment for compressive sampling video transmission J Electron 2012 29 6 598-603
[2]
Chen J, Liu S, Chen Z (2017) Gender classification in live videos. In: 2017 IEEE International Conference on Image Processing (ICIP), pp 1602–1606. IEEE
[3]
Mittal A, Moorthy AK, and Bovik AC No-reference image quality assessment in the spatial domain IEEE Trans Image Process 2012 21 12 4695-4708
[4]
Mittal A, Saad MA, and Bovik AC A completely blind video integrity oracle IEEE Trans Image Process 2015 25 1 289-300
[5]
Saad MA, Bovik AC, and Charrier C Blind prediction of natural video quality IEEE Trans Image Process 2014 23 3 1352-1365
[6]
Hochreiter S and Schmidhuber J Long short-term memory Neural Comput 1997 9 8 1735-1780
[7]
Li D, Jiang T, Jiang M (2019) Quality assessment of in-the-wild videos. Association for Computing Machinery, New York, NY, USA.
[8]
Otroshi Shahreza H, Amini A, Behroozi H (2019) No-reference video quality assessment using recurrent neural networks. In: 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), pp 1–5.
[9]
Simonyan K, Zisserman A (2015) Two-stream convolutional networks for action recognition. In: Proceedings of the Neural Information Processing Systems (NIPS)
[10]
Zhang Y, Gao X, He L, Lu W, and He R Blind video quality assessment with weakly supervised learning and resampling strategy IEEE Trans Circuits Syst Video Technol 2018 29 8 2244-2255
[11]
Pan D, Wang X, Shi P, and Yu S No-reference video quality assessment based on modeling temporal-memory effects Displays 2021 70
[12]
Wu W, Liu Z, Chen Z, Liu S (2020) No-reference video quality assessment based on similarity map estimation. In: 2020 IEEE International Conference on Image Processing (ICIP), pp 181–185. IEEE
[13]
Qian J, Wu D, Li L, Cheng D, and Wang X Image quality assessment based on multi-scale representation of structure Digital Signal Processing 2014 33 125-133
[14]
Wang Z and Li Q Video quality assessment using a statistical model of human visual speed perception JOSA A 2007 24 12 61-69
[15]
Zhang L, Zhang L, Mou X, and Zhang D Fsim: A feature similarity index for image quality assessment IEEE Trans Image Process 2011 20 8 2378-2386
[16]
Xue W, Zhang L, Mou X, and Bovik AC Gradient magnitude similarity deviation: A highly efficient perceptual image quality index IEEE Trans Image Process 2013 23 2 684-695
[17]
Kuang Z, Yu J, Li Z, Zhang B, and Fan J Integrating multi-level deep learning and concept ontology for large-scale visual recognition Pattern Recogn 2018 78 198-214
[18]
Kuang Z, Guo Z, Fang J, Yu J, Babaguchi N, and Fan J Unnoticeable synthetic face replacement for image privacy protection Neurocomputing 2021 457 322-333
[19]
Bosse S, Maniry D, Muller K-R, Wiegand T, and Samek W Deep neural networks for no-reference and full-reference image quality assessment IEEE Trans Image Process 2018 27 1 206-219
[20]
Li S, Han X, Zubair M, Ma S (2019) Stereo image quality assessment based on sparse binocular fusion convolution neural network. In: 2019 IEEE Visual Communications and Image Processing (VCIP), pp 1–4. IEEE
[21]
Olshausen BA and Field DJ Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 1996 381 6583 607-609
[22]
Yu J, Rui Y, and Tao D Click prediction for web image reranking using multimodal sparse coding IEEE Trans Image Process 2014 23 5 2019-2032
[23]
Su S, Yan Q, Zhu Y, Zhang C, Ge X, Sun J, Zhang Y (2020) Blindly assess image quality in the wild guided by a self-adaptive hyper network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3667–3676
[24]
Zhang Y, Yang Q (2021) A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering
[25]
Seshadrinathan K and Bovik AC Motion tuned spatio-temporal quality assessment of natural videos IEEE Trans Image Process 2009 19 2 335-350
[26]
Korhonen J Two-level approach for no-reference consumer video quality assessment IEEE Trans Image Process 2019 28 12 5923-5938
[27]
Tu Z, Wang Y, Birkbeck N, Adsumilli B, and Bovik A Ugc-vqa: Benchmarking blind video quality assessment for user generated content IEEE Trans Image Process 2021 30 4449-4464
[28]
Tu Z, Yu X, Wang Y, Birkbeck N, Adsumilli B, and Bovik AC Rapique: Rapid and accurate video quality prediction of user generated content IEEE Open J Signal Process 2021 2 425-440 arXiv:2101.10955
[29]
Li D, Jiang T, and Jiang M Unified quality assessment of in-the-wild videos with mixed datasets training Int J Comput Vision 2021 129 4 1-20
[30]
Chen B, Zhu L, Li G, Lu F, Fan H, and Wang S Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment IEEE Trans Circuits Syst Video Technol 2021 32 4 1903-1916
[31]
Varga D and Szirányi T No-reference video quality assessment via pretrained cnn and lstm networks SIViP 2019 13 8 1569-1576
[32]
Zhang J, Cao Y, and Wu Q Vector of locally and adaptively aggregated descriptors for image feature representation Pattern Recogn 2021 116
[33]
Sun W, Wang T, Min X, Yi F, Zhai G (2021) Deep learning based full-reference and no-reference quality assessment models for compressed ugc videos. In: 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp 1–6. IEEE Computer Society
[34]
Soundararajan R and Bovik AC Rred indices: Reduced reference entropic differencing for image quality assessment IEEE Trans Image Process 2011 21 2 517-526
[35]
Zhou W, Chen Z (2020) Deep local and global spatiotemporal feature aggregation for blind video quality assessment. In: 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp 338–341. IEEE
[36]
Hosu V, Hahn F, Jenadeleh M, Lin H, Men H, Szirányi T, Li S, Saupe D (2017) The konstanz natural video database (konvid-1k). In: 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), pp 1–6.
[37]
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
[38]
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp 248–255. IEEE.
[39]
Li H, Chen G, Li G, Yu Y (2019) Motion guided attention for video salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 7274–7283
[40]
Gao F, Yu J, Zhu S, Huang Q, and Tian Q Blind image quality prediction by exploiting multi-level deep representations Pattern Recogn 2018 81 432-442
[41]
Fawaz HI, Lucas B, Forestier G, Pelletier C, Schmidt DF, Weber J, Webb GI, Idoumghar L, Muller P-A, and Petitjean F Inceptiontime: Finding alexnet for time series classification Data Min Knowl Disc 2020 34 6 1936-1962
[42]
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1–9
[43]
Ying Z, Mandal M, Ghadiyaram D, Bovik A (2021) Patch-vq:’patching up’the video quality problem. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14019–14029
[44]
Sinno Z and Bovik AC Large-scale study of perceptual video quality IEEE Trans Image Process 2019 28 2 612-627
[45]
Ghadiyaram D, Pan J, Bovik AC, Moorthy AK, Panda P, and Yang K-C In-capture mobile video distortions: A study of subjective behavior and objective algorithms IEEE Trans Circuits Syst Video Technol 2018 28 9 2061-2077
[46]
Nuutinen M, Virtanen T, Vaahteranoksa M, Vuori T, Oittinen P, and Hökkinen J Cvd 2014 a database for evaluating no-reference video quality assessment algorithms IEEE Trans Image Process 2016 25 7 3073-3086
[47]
Seshadrinathan K, Soundararajan R, Bovik A, and Cormack L Study of subjective and objective quality assessment of video IEEE Trans Image Process 2010 19 1427-1441
[48]
Kingma DP, Ba JL (2015) Adam: A method for stochastic optimization. In: ICLR 2015 : International Conference on Learning Representations 2015
[49]
Mittal A, Soundararajan R, and Bovik AC Making a “completely blind” image quality analyzer IEEE Signal Process Lett 2013 20 3 209-212
Recommendations
Binocular perception based reduced-reference stereo video quality assessment method
A reduced-reference (RR) stereo video quality assessment method is proposed.RR frames are extracted by using motion intensity to keep temporal characteristics.Cyclopean view is generated by matching binocular fusion portions of the two views. A new ...
Video Quality Assessment Metric Based on Spatio-temporal Motion Information
DASC '13: Proceedings of the 2013 IEEE 11th International Conference on Dependable, Autonomic and Secure ComputingVideo quality assessment (VQA) plays an important role in video processing applications, e.g., compression, archiving, restoration, transmission and enhancement. Based on the video content, we design an effective and efficient objective video quality ...
Modeling motion visual perception for video quality assessment
MM '11: Proceedings of the 19th ACM international conference on MultimediaContrast sensitivity of Human Visual System (HVS) plays an important role in perceiving visual stimuli, and consequently, it has a significant impact on the perceived video quality. This paper proposes a visual perception model based on foveated vision ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 25 June 2022
Accepted: 16 June 2022
Author Tags
Qualifiers
- Research-article
Funding Sources
- the National Natural Science Foundation of China [2018]
- the Science Planning Project of Guizhou Province
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 08 Feb 2025