Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Published: 08 March 2024 Publication History

Abstract

Compared with traditional 2D images, omnidirectional images (also referred to as 360 images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

References

[1]
RECOMMENDATION ITU-R BT. 2002. Methodology for the subjective assessment of the quality of television pictures. Retrieved June 17, 2021 from http://www.itu.int
[2]
Meixu Chen, Yize Jin, Todd Goodall, Xiangxu Yu, and Alan Conrad Bovik. 2020. Study of 3D virtual reality picture quality. IEEE Journal of Selected Topics in Signal Processing 14, 1 (2020), 89–102.
[3]
Sijia Chen, Yingxue Zhang, Yiming Li, Zhenzhong Chen, and Zhou Wang. 2018. Spherical structural similarity index for objective omnidirectional video quality assessment. In Proceedings of the IEEE International Conference on Multimedia and Expo. 1–6.
[4]
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL, 1724–1734.
[5]
Igor D. D. Curcio, Henri Toukomaa, and Deepa Naik. 2017. 360-degree video streaming and its subjective quality. In Proceedings of the SMPTE 2017 Annual Technical Conference and Exhibition. 1–23.
[6]
Jia Deng, Wei Dong, Richard Socher, Lijia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conferenc on Computer Vision and Pattern Recognition. 248–255.
[7]
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. 2022. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 5 (2022), 2567–2581.
[8]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations.
[9]
Huiyu Duan, Guangtao Zhai, Xiongkuo Min, Yucheng Zhu, Yi Fang, and Xiaokang Yang. 2018. Perceptual quality assessment of omnidirectional images. In Proceedings of the IEEE International Symposium on Circuits and Systems. 1–5.
[10]
Henry Been-Lirn Duh, Donald E Parker, James O. Philips, and Thomas A. Furness. 2004. Conflicting motion cues to the visual and vestibular self-motion systems around 0.6 Hz evoke simulator sickness. Human Factors 46, 1 (2004), 142–153.
[11]
Yuming Fang, Liping Huang, Jiebin Yan, Xuelin Liu, and Yang Liu. 2022. Perceptual quality assessment of omnidirectional images. In Proceedings of the AAAI Conference on Artificial Intelligence. 1–13.
[12]
Yuming Fang, Zhaoqian Li, Jiebin Yan, Xiangjie Sui, and Hantao Liu. 2023. Study of spatio-temporal modeling in video quality assessment. IEEE Transactions on Image Processing 32 (2023), 2693–2702.
[13]
Yuming Fang, Xiangjie Sui, Jiheng Wang, Jiebin Yan, Jianjun Lei, and Patrick Le Callet. 2021. Perceptual quality assessment for asymmetrically distorted stereoscopic video by temporal binocular rivalry. IEEE Transactions on Circuits and Systems for Video Technology 31, 8 (2021), 3010–3024.
[14]
Yuming Fang, Jiebin Yan, Leida Li, Jinjian Wu, and Weisi Lin. 2017. No reference quality assessment for screen content images with both local and global feature representation. IEEE Transactions on Image Processing 27, 4 (2017), 1600–1610.
[15]
Yuming Fang, Chi Zhang, Hanqin Huang, and Jianjun Lei. 2019. Visual attention prediction for stereoscopic video by multi-module fully convolutional network. IEEE Transactions on Image Processing 28, 11 (2019), 5253–5265.
[16]
Peter Fröhlich, Sebastian Egger, Raimund Schatz, Michael Mühlegger, Kathrin Masuch, and Bruno Gardlo. 2012. QoE in 10 seconds: Are short video clip lengths sufficient for quality of experience assessment?. In Proceedings of the 4th International Workshop on Quality of Multimedia Experience. 242–247.
[17]
S. Alireza Golestaneh, Saba Dadsetan, and Kris M. Kitani. 2022. No-reference image quality assessment via transformers, relative ranking, and self-consistency. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 3989–3999.
[18]
David Hasler and Sabine E. Suesstrunk. 2003. Measuring colorfulness in natural images. In Proceedings of the Human Vision and Electronic Imaging VIII. 87–95.
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 770–778.
[20]
Mingkai Huang, Qiu Shen, Zhan Ma, Alan Conrad Bovik, Praful Gupta, Rongbing Zhou, and Xun Cao. 2018. Modeling the perceptual quality of immersive images rendered on head mounted displays: Resolution and compression. IEEE Transactions on Image Processing 27, 12 (2018), 6039–6050.
[21]
Hao Jiang, Gangyi Jiang, Mei Yu, Yun Zhang, You Yang, Zongju Peng, Fen Chen, and Qingbo Zhang. 2021. Cubemap-based perception-driven blind quality assessment for 360-degree images. IEEE Transactions on Image Processing 30 (2021), 2364–2377.
[22]
Hak Gu Kim, Heoun-Taek Lim, and Yong Man Ro. 2020. Deep virtual reality image quality assessment with human perception guider for omnidirectional image. IEEE Transactions on Circuits and Systems for Video Technology 30, 4 (2020), 917–928.
[23]
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2017).
[24]
Lukáš Krasula, Karel Fliegel, Patrick Le Callet, and Miloš Klíma. 2016. On the accuracy of objective image and video quality models: New methodology for performance evaluation. In Proceedings of the 8th International Conference on Quality of Multimedia Experience. 1–6.
[25]
Dingquan Li, Tingting Jiang, and Ming Jiang. 2019. Quality assessment of in-the-wild videos. In Proceedings of the ACM International Conference on Multimedia. 2351–2359.
[26]
Dingquan Li, Tingting Jiang, and Ming Jiang. 2020. Norm-in-norm loss with faster convergence and better performance for image quality assessment. In Proceedings of the 28th ACM International Conference on Multimedia. 789–797.
[27]
Yun Liu, Xiaohua Yin, Zuliang Wan, Guanghui Yue, and Zhi Zheng. 2023. Toward a no-reference omnidirectional image quality evaluation by using multi-perceptual features. ACM Transactions On Multimedia Computing, Communications, And Applications 19, 2 (2023), 1–19.
[28]
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21, 12 (2012), 4695–4708.
[29]
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2013. Making a completely blind image quality analyzer. IEEE Signal Processing Letters 20, 3 (2013), 209–212.
[30]
RECOMMENDATION ITU-T P. 1999. Subjective video quality assessment methods for multimedia applications. Retrieved June 17, 2021 from http://www.itu.int
[31]
Jincheol Park, Kalpana Seshadrinathan, Sanghoon Lee, and Alan Conrad Bovik. 2013. Video quality pooling adaptive to perceptual distortion severity. IEEE Transactions on Image Processing 22, 2 (2013), 610–620.
[32]
Silvia Rossi, Cagri Ozcinar, Aljosa Smolic, and Laura Toni. 2020. Do users behave similarly in VR? Investigation of the user influence on the system design. ACM Transactions on Multimedia Computing, Communications, And Applications 16, 2 (2020), 1–26.
[33]
Abderrezzaq Sendjasni, Mohamed-Chaker Larabi, and Faouzi Alaya Cheikh. 2021. Convolutional neural network for omnidirectional image quality assessment: Pre-trained or re-trained?. In Proceedings of the IEEE International Conference on Image Processing. 3413–1417.
[34]
H. R. Sheikh and Alan C. Bovik. 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 2 (2006), 430–444.
[35]
Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2018. Saliency in VR: How do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1633–1642.
[36]
Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. 2020. Blind assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3667–3676.
[37]
Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, and Zhou Wang. 2023. ScanDMM: A deep markov model of scanpath prediction for 360deg images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6989–6999.
[38]
Xiangjie Sui, Kede Ma, Yiru Yao, and Yuming Fang. 2021. Perceptual quality assessment of omnidirectional images as moving camera videos. IEEE Transactions on Visualization and Computer Graphics 28, 8 (2021), 3022–3034.
[39]
Wei Sun, Ke Gu, Guangtao Zhai, Siwei Ma, Weisi Lin, and Patrick Le Calle. 2017. CVIQD: Subjective quality evaluation of compressed virtual reality images. In Proceedings of the IEEE International Conference on Image Processing. 3450–3454.
[40]
Wei Sun, Xiongkuo Min, Guangtao Zhai, Ke Gu, Huiyu Duan, and Siwei Ma. 2020. MC360IQA: A multi-channel cnn for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing 14, 1 (2020), 64–77.
[41]
Yule Sun, Ang Lu, and Lu Yu. 2017. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Processing Letters 24, 9 (2017), 1408–1412.
[42]
Evgeniy Upenik, Martin Řeřábek, and Touradj Ebrahimi. 2016. Testbed for subjective evaluation of omnidirectional visual content. In Proceedings of the Picture Coding Symposium. 1–5.
[43]
VQEG. 2000. Final report from the video quality experts group on the validation of objective models of video quality assessment. Retrieved June 17, 2021 from http://www.vqeg.org
[44]
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
[45]
Zhou Wang and Abdul Rehman. 2017. Begin with the end in mind: A unified end-to-end quality-of-experience monitoring, optimization and management framework. In Proceedings of the SMPTE Annual Technical Conference and Exhibition. 1–11.
[46]
Tianhe Wu, Shuwei Shi, Haoming Cai, Mingdeng Cao, Jing Xiao, Yinqiang Zheng, and Yujiu Yang. 2023. Assessor360: Multi-sequence network for blind omnidirectional image quality assessment. arXiv preprint arXiv: 2305.10983 (2023).
[47]
Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2012. Recognizing scene viewpoint using panoramic place representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2695–2702.
[48]
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1492–1500.
[49]
Jiahua Xu, Chaoyi Lin, Wei Zhou, and Zhibo Chen. 2018. Subjective quality assessment of stereoscopic omnidirectional image. In Proceedings of the Pacific Rim Conference on Multimedia. 589–599.
[50]
Jiahua Xu, Ziyuan Luo, Wei Zhou, Wenyuan Zhang, and Zhibo Chen. 2019. Quality assessment of stereoscopic 360-degree images from multi-viewports. In Proceedings of the Picture Coding Symposium. 1–5.
[51]
Jiahua Xu, Wei Zhou, and Zhibo Chen. 2021. Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks. IEEE Transactions on Circuits and Systems for Video Technology 31, 5 (2021), 1724–1737.
[52]
Mai Xu, Chen Li, Shanyi Zhang, and Patrick Le Callet. 2020. State-of-the-art in 360° video/image processing: Perception, assessment and compression. IEEE Journal of Selected Topics in Signal Processing 14, 1 (2020), 5–26.
[53]
Jiebin Yan, Yuming Fang, Rengang Du, Yan Zeng, and Yifan Zuo. 2020. No reference quality assessment for 3D synthesized views by local structure variation and global naturalness change. IEEE Transactions on Image Processing 29 (2020), 7443–7453.
[54]
Jiebin Yan, Yuming Fang, Liping Huang, Xiongkuo Min, Yiru Yao, and Guangtao Zhai. 2020. Blind stereoscopic image quality assessment by deep neural network of multi-level feature fusion. In Proceedings of the IEEE International Conference on Multimedia and Expo. 1–6.
[55]
Jiebin Yan, Jing Li, Yuming Fang, Zhaohui Che, Xue Xia, and Yang Liu. 2022. Subjective and objective quality of experience of free viewpoint videos. IEEE Transactions on Image Processing 31 (2022), 3896–3907.
[56]
Jiebin Yan, Yu Zhong, Yuming Fang, Zhangyang Wang, and Kede Ma. 2021. Exposing semantic segmentation failures via maximum discrepancy competition. International Journal of Computer Vision 129, 5 (2021), 1768–1786.
[57]
Li Yang, Mai Xu, Xin Deng, and Bo Feng. 2021. Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images. In Proceedings of the IEEE International Conference on Multimedia and Expo. 1–6.
[58]
Li Yang, Mai Xu, Tie Liu, Liangyu Huo, and Xinbo Gao. 2022. TVFormer: Trajectory-guided visual quality assessment on 360° images with transformers. In Proceedings of the ACM International Conference on Multimedia. 799–808.
[59]
Yan Ye, Elena Alshina, and Jill Boyce. 2017. JVET-G1003: Algorithm description of projection format conversion and video quality metrics in 360lib version 4. Retrieved June 17, 2021 from http://phenix.it-sudparis.eu/jvet
[60]
Matt Yu, Haricharan Lakshman, and Bernd Girod. 2015. A framework to evaluate omnidirectional video coding schemes. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality. 31–36.
[61]
Vladyslav Zakharchenko, Kwang Pyo Choi, and Jeong Hoon Park. 2016. Quality metric for spherical panoramic video. In Proceedings of the Optics and Photonics for Information Processing X. 99700C.
[62]
Lin Zhang, Ying Shen, and Hongyu Li. 2014. VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image Processing 23, 10 (2014), 4270–4281.
[63]
Lin Zhang, Lei Zhang, and Alan C Bovik. 2015. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing 24, 8 (2015), 2579–2591.
[64]
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing 20, 8 (2011), 2378–2386.
[65]
Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. 2020. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology 30, 1 (2020), 36–47.
[66]
Weixia Zhang, Kede Ma, Guangtao Zhai, and Xiaokang Yang. 2021. Uncertainty-aware blind image quality assessment in the laboratory and wild. IEEE Transactions on Image Processing 30 (2021), 3474–3486.
[67]
Xuelei Zheng, Gangyi Jiang, Mei Yu, and Hao Jiang. 2020. Segmented spherical projection-based blind omnidirectional image quality assessment. IEEE Access 8 (2020), 31647–31659.
[68]
Wei Zhou, Jiahua Xu, Qiuping Jiang, and Zhibo Chen. 2022. No-reference quality assessment for 360-degree images by analysis of multifrequency information and local-global naturalness. IEEE Transactions on Circuits and Systems for Video Technology 32, 4 (2022), 1778–1791.
[69]
Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang. 2022. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions on Circuits and Systems for Video Technology 32, 4 (2022), 1767–1777.
[70]
Yufeng Zhou, Mei Yu, Hualin Ma, Hua Shao, and Gangyi Jiang. 2018. Weighted-to-spherically-uniform SSIM objective quality evaluation for panoramic video. In Proceedings of the IEEE International Conference on Signal Processing. 54–57.
[71]
Hancheng Zhu, Leida Li, Jinjian Wu, Weisheng Dong, and Guangming Shi. 2020. MetaIQA: Deep meta-learning for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14131–14140.

Cited By

View all
  • (2025)SecureIoT-FL: A Federated Learning Framework for Privacy-Preserving Real-Time Environmental Monitoring in Industrial IoT ApplicationsAlexandria Engineering Journal10.1016/j.aej.2024.11.069114(681-701)Online publication date: Mar-2025
  • (2024)The application of IoT devices in evidence collection and preservation In legal investigations In the common law legal system of Western countries2024 11th International Symposium on Telecommunications (IST)10.1109/IST64061.2024.10843627(120-124)Online publication date: 9-Oct-2024
  • (2024)Performance Evaluation of Objective Quality Assessment Methods for Omnidirectional Images Under Emerging CompressionsIEEE Access10.1109/ACCESS.2024.347879312(150419-150429)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 20, Issue 6
    June 2024
    715 pages
    EISSN:1551-6865
    DOI:10.1145/3613638
    • Editor:
    • Abdulmotaleb El Saddik
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 March 2024
    Online AM: 26 January 2024
    Accepted: 01 January 2024
    Revised: 05 December 2023
    Received: 24 August 2023
    Published in TOMM Volume 20, Issue 6

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Omnidirectional image
    2. image quality assessment
    3. non-uniform distortion
    4. viewing condition

    Qualifiers

    • Research-article

    Funding Sources

    • Natural Science Foundation of China
    • Natural Science Foundation of Jiangxi Province of China
    • Project funded by China Postdoctoral Science Foundation
    • Postgraduate Innovation Special Fund of Jiangxi Province

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)419
    • Downloads (Last 6 weeks)27
    Reflects downloads up to 26 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)SecureIoT-FL: A Federated Learning Framework for Privacy-Preserving Real-Time Environmental Monitoring in Industrial IoT ApplicationsAlexandria Engineering Journal10.1016/j.aej.2024.11.069114(681-701)Online publication date: Mar-2025
    • (2024)The application of IoT devices in evidence collection and preservation In legal investigations In the common law legal system of Western countries2024 11th International Symposium on Telecommunications (IST)10.1109/IST64061.2024.10843627(120-124)Online publication date: 9-Oct-2024
    • (2024)Performance Evaluation of Objective Quality Assessment Methods for Omnidirectional Images Under Emerging CompressionsIEEE Access10.1109/ACCESS.2024.347879312(150419-150429)Online publication date: 2024
    • (2024)Blind omnidirectional image quality assessment based on semantic information replenishmentJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104241103(104241)Online publication date: Aug-2024
    • (2024)Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-modal ManipulationInternational Journal of Computer Vision10.1007/s11263-024-02245-xOnline publication date: 7-Oct-2024

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media