research-article

Learning deep features for image emotion classification

Authors:

Jan P. AllebachAuthors Info & Claims

2015 IEEE International Conference on Image Processing (ICIP)

Pages 4491 - 4495

https://doi.org/10.1109/ICIP.2015.7351656

Published: 01 September 2015 Publication History

Abstract

Images can both express and affect people's emotions. It is intriguing and important to understand what emotions are conveyed and how they are implied by the visual content of images. Inspired by the recent success of deep convolutional neural networks (CNN) in visual recognition, we explore two simple, yet effective deep learning-based methods for image emotion analysis. The first method uses off-the-shelf CNN features directly for classification. For the second method, we fine-tune a CNN that is pre-trained on a large dataset, i.e. ImageNet, on our target dataset first. Then we extract features using the fine-tuned CNN at different location at multiple levels to capture both the global and local information. The features at different location are aggregated using the Fisher Vector for each level and concatenated to form a compact representation. From our experimental results, both the deep learning-based methods outperforms traditional methods based on generic image descriptors and hand-crafted features.

5. References

[1]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recopnition, 2009. IEEE Conference on. IEEE, 2009, pp. 248–255.

[2]

Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo, “Aesthetics and emotions in images,” Signal Processing Magazine, IEEE, vol. 28, no. 5, pp. 94–115, 2011.

[3]

Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming, “Image retrieval by emotional semantics: A study of emotional space and feature extraction,” in Systems, Man and Cybernetics, 2006. SMC'06. IEEE International Conference on. IEEE, 2006, vol. 4, pp. 3534–3539.

[4]

Jana Machajdik and Allan Hanbury, “Affective image classification using features inspired by psychology and art theory,” in Proceedings of the international conference on Multimedia. ACM, 2010, pp. 83–92.

[5]

Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, and Jie Tang, “Can we understand van gogh's mood?: learning to infer affects from images in social networks,” in Proceedings of the 20th ACM international conference on Multimedia ACM, 2012, pp. 857–860.

[6]

Ritendra Datta, Jia Li, and James Ze Wang, “Algorith-mic inferencing of aesthetics and emotion in natural images: An exposition,” in Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on. IEEE, 2008, pp. 105–108.

[7]

Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun, “Exploring principles-of-art features for image emotion recognition,” in Proceedings of the ACM International Conference on Multimedia ACM, 2014, pp. 47–56.

[8]

Sicheng Zhao, Hongxun Yao, You Yang, and Yanhao Zhang, “Affective image retrieval via multi-graph learning,” in Proceedings of the ACM International Conference on Multimedia. ACM, 2014, pp. 1025–1028.

[9]

Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson, “Cnn features off-the-shelf: an astounding baseline for recognition,” in Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on. IEEE, 2014, pp. 512–519.

[10]

Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller, “Recognizing image style,” arXiv preprint arXiv:1311.3715, 2013.

[11]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.

Digital Library

[12]

Yangqing Jia, “Caffe: An open source convolutional architecture for fast feature embedding,” http://caffe.berkeleyvision.org, 2013.

[13]

Yunchao Gong, Liwei Wang, Ruigi Guo, and Svetlana Lazebnik, “Multi-scale orderless pooling of deep convolutional activation features,” in Computer Vision-ECCV 2014, pp. 392–407. Springer, 2014.

[14]

Florent Perronnin, Jorge Sanchez, and Thomas Mensink, “Improving the fisher kernel for large-scale image classification,” in Computer Vision-ECCV 2010, pp. 143–156. Springer, 2010.

Digital Library

[15]

Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka, “Assessing the aesthetic quality of photographs using generic image descriptors,” in Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 2011, pp. 1784–1791.

[16]

Sagnik Dhar, Vicente Ordonez, and Tamara L Berg, “High level describable attributes for predicting aesthetics and interestingness,” in Computec Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011, pp. 1657–1664.

[17]

Jahanian Ali, Quantifying Aesthetics of Visual Design Applied to Automatic Design, Ph.D. thesis, Purdue University, 2014.

[18]

Ming Chen and Jan Allebach, “Aesthetic quality inference for online fashion shopping,” in IS&TISPIE Electronic Imaging. International Society for Optics and Photonics, 2014, pp. 902703–902703.

Cited By

Ruotsalo TMäkelä KSpapé MLeiva LChen HDuh WHuang HKato MMothe JPoblete B(2023)Affective Relevance: Inferring Emotional Responses via fNIRS NeuroimagingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591946(1796-1800)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591946
Wang XChen CLan RLiu LLiu ZZhou HLuo X(2022)Binary Representation via Jointly Personalized Sparse HashingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355876918:3s(1-20)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.1145/3558769
Song JHan KKim S(2022)“I Have No Text in My Post”: Using Visual Hints to Model User Emotions in Social MediaProceedings of the ACM Web Conference 202210.1145/3485447.3512009(2888-2896)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512009
Show More Cited By

Index Terms

Learning deep features for image emotion classification
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

2015 IEEE International Conference on Image Processing (ICIP)

5242 pages

Copyright © 2015.

Publisher

IEEE Press

Publication History

Published: 01 September 2015

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ruotsalo TMäkelä KSpapé MLeiva LChen HDuh WHuang HKato MMothe JPoblete B(2023)Affective Relevance: Inferring Emotional Responses via fNIRS NeuroimagingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591946(1796-1800)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591946
Wang XChen CLan RLiu LLiu ZZhou HLuo X(2022)Binary Representation via Jointly Personalized Sparse HashingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355876918:3s(1-20)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.1145/3558769
Song JHan KKim S(2022)“I Have No Text in My Post”: Using Visual Hints to Model User Emotions in Social MediaProceedings of the ACM Web Conference 202210.1145/3485447.3512009(2888-2896)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512009
Song JHan KLee DKim SHung CCerny TShin DBechini A(2020)Understanding emotions in SNS images from posters' perspectivesProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3373923(450-457)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3373923

View Options

View options

Figures

Tables

Media

View Table of Conten