Abstract
Visual sentiment is subjective and abstract, and it is very challenging to locate the sentiment features from images accurately. Some researchers devote themselves to extracting visual features but ignore the relation features. However, sentiment reaction is a comprehensive action of visual content, and regions may express different emotions and contribute to the image sentiment. This paper takes the abstract sentiment relation as the starting point and proposes the Weakly Supervised Interaction Discovery Network that couples detection and classification branch. Specifically, the first branch detects sentiment maps with the cross-spatial pooling strategy, which generates the representations of emotions. Then, we employ a stacked Graph Convolution Network to extract the interaction feature from the above features. The second branch utilizes both interaction and visual features for robust sentiment classification. Extensive experiments on six benchmark datasets demonstrate that the proposed method exceeds the state-of-the-art methods for image sentiment analysis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Borth, D., Ji, R., Chen, T., Breuel, T., Chang, S-F.: Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: ACM MM (2013)
Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and deep locally connected networks on graphs. In: 2nd International Conference on Learning Representations, ICLR 2014, pp. 61–80 (2014)
Chen, T., Borth, D., Darrell, T., Chang, S.-F.: Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586 (2014)
Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5177–5186 (2019)
Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, vol. 2, pp. 729–734. IEEE (2005)
Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: ACM MM (2010)
Peng, K.-C., Sadovnik, A., Gallagher, A., Chen, T.: Where do emotions come from? predicting the emotion stimuli map. In: ICIP (2016)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)
Siersdorfer, S., Minack, E., Deng, F., Hare, J.: Analyzing and predicting sentiment of images on the social web. In: Proceedings of the 18th ACM international conference on Multimedia, pp. 715–718 (2010)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2015)
Sun, M., Yang, J., Wang, K., Shen, H.: Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. In: ICME (2016)
Wu, L., Qi, M., Jian, M., Zhang, H.: Visual sentiment analysis by combining global and local information. Neural Process. Lett. 51(3), 2063–2075 (2019). https://doi.org/10.1007/s11063-019-10027-7
Yang, J., She, D., Lai, Y.-K., Rosin, P.L., Yang, M.-H.: Weakly supervised coupled networks for visual sentiment analysis. In: CVPR (2018)
Yang, J., She, D., Sun, M., Cheng, M.-M., Rosin, P.L., Wang, L.: Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans. Multimedia 20(9), 2513–2525 (2018)
You, Q., Jin, H., Luo, J.: Visual sentiment analysis by attending on local image regions. In: AAAI (2017)
You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: AAAI (2015)
You, Q., Luo, J., Jin, H., Yang, J.: Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: AAAI (2016)
Acknowledgement
This work was supported in part by Beijing Municipal Education Committee Science Foundation (KM201910005024), Beijing Postdoctoral Research Fundation (Q6042001202101).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, L., Zhang, H., Shi, G., Deng, S. (2022). Weakly Supervised Interaction Discovery Network for Image Sentiment Analysis. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13188. Springer, Cham. https://doi.org/10.1007/978-3-031-02375-0_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-02375-0_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-02374-3
Online ISBN: 978-3-031-02375-0
eBook Packages: Computer ScienceComputer Science (R0)