Abstract
Visual content can affect viewer’s emotions, which makes sentiment analysis of visual content more and more concerned. Sentiment analysis on omni-directional images plays an important role in virtual reality (VR) applications such as user behaviour prediction, game scene modelling, psychotherapy, etc. However, due to the serious lack of validated VR emotional datasets, the research progress of sentiment analysis in VR is very slow. In this paper, firstly, we build a VR sentiment dataset containing 1,140 emotion-eliciting omni-directional images. Secondly, a pyramidal dual attention network is proposed to analyse the sentiment task. According to the characteristics of omni-directional images, this network utilizes the dual attention module to capture emotion-eliciting regions and adaptively establish the connection between them. Furthermore, objects of different scales have different contributions to evoke emotions. Therefore, the pyramidal feature hierarchy can analyse objects with different complexity by using multi-layer visual features. Finally, quantitative and qualitative experiments on the self-established dataset illustrate that the proposed network can effectively predict the regions that elicit emotions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhu, Y., Zhai, G., Min, X., Zhou, J.: Learning a deep agent to predict head movement in 360-degree images. ACM Trans. Multimed. Comput. Commun. Appl. 16(4), 1–23 (2020)
MarÃn-Morales, J., et al.: Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 8(1), 1–15 (2018)
Xu, M., Li, C., Zhang, S., Le Callet, P.: State-of-the-art in 360 video/image processing: perception, assessment and compression. IEEE J. Sel. Top. Signal Process. 14(1), 5–26 (2020)
She, D., Sun, M., Yang, J.: Learning discriminative sentiment representation from strongly-and weakly supervised cnns. ACM Trans. Multimed. Comput. Commun. Appl. 15(3s), 1–19 (2019)
Li, B.J., Bailenson, J.N., Pines, A., Greenleaf, W.J., Williams, L.M.: A public database of immersive VR videos with corresponding ratings of arousal, valence, and correlations between head movements and self report measures. Front. Psychol. Original Res. 8(2116) (2017)
Zhao, S., et al.: Affective image content analysis: two decades review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. (2021)
Yang, J., She, D., Sun, M., Cheng, M.-M., Rosin, P.L., Wang, L.: Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans. Multimed. 20(9), 2513–2525 (2018)
Yang, J., She, D., Lai, Y.-K., Rosin, P.L., Yang, M.-H.: Weakly supervised coupled networks for visual sentiment analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7584–7592 (2018)
Ou, H., Qing, C., Xu, X., Jin, J.: Multi-level context pyramid network for visual sentiment analysis. Sensors 21(6), 2136 (2021)
Li, B.J., Bailenson, J.N., Pines, A., Greenleaf, W.J., Williams, L.M.: A public database of immersive VR videos with corresponding ratings of arousal, valence, and correlations between head movements and self report measures. Front. Psychol. 8, 2116 (2017)
Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)
Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): technical manual and affective ratings. NIMH Cent. Study Emot. Attent. 1, 39–58 (1997)
De Abreu, A., Ozcinar, C., Smolic, A.: Look around you: saliency maps for omnidirectional images in VR applications. In: 2017 Ninth International Conference on Quality of Multimedia Experience, pp. 1–6 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Zhao, S., Jia, Z., Chen, H., Li, L., Ding, G., Keutzer, K.: PDANet: polarity-consistent deep attention network for fine-grained visual emotion regression. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 192–201 (2019)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision.. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
Acknowledgement
This work is partially supported by the following grants: National Natural Science Foundation of China (61972163, U1801262), Natural Science Foundation of Guangdong Province (2022A1515011555, 2023A1515012568), Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004) and Pazhou Lab, Guangzhou, 510330, China.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Huang, R., Ou, H., Qing, C., Xu, X. (2024). Visual Sentiment Analysis with a VR Sentiment Dataset on Omni-Directional Images. In: Ren, J., et al. Advances in Brain Inspired Cognitive Systems. BICS 2023. Lecture Notes in Computer Science(), vol 14374. Springer, Singapore. https://doi.org/10.1007/978-981-97-1417-9_28
Download citation
DOI: https://doi.org/10.1007/978-981-97-1417-9_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-1416-2
Online ISBN: 978-981-97-1417-9
eBook Packages: Computer ScienceComputer Science (R0)