article

Discovering Beautiful Attributes for Aesthetic Image Analysis

Authors:

Luca Marchesotti,

Florent PerronninAuthors Info & Claims

International Journal of Computer Vision, Volume 113, Issue 3

Pages 246 - 266

https://doi.org/10.1007/s11263-014-0789-2

Published: 01 July 2015 Publication History

Abstract

Aesthetic image analysis is the study and assessment of the aesthetic properties of images. Current computational approaches to aesthetic image analysis either provide accurate or interpretable results. To obtain both accuracy and interpretability by humans, we advocate the use of learned and nameable visual attributes as mid-level features. For this purpose, we propose to discover and learn the visual appearance of attributes automatically, using a recently introduced database, called AVA, which contains more than 250,000 images together with their aesthetic scores and textual comments given by photography enthusiasts. We provide a detailed analysis of these annotations as well as the context in which they were given. We then describe how these three key components of AVA--images, scores, and comments--can be effectively leveraged to learn visual attributes. Lastly, we show that these learned attributes can be successfully used in three applications: aesthetic quality prediction, image tagging and retrieval.

References

[1]

"aesthetics" E. (2012). The American Heritage® Dictionary of the English Language, Fourth Edition.

[2]

Akata, Z., Perronnin, F., Harchaoui, Z., & Schmid, C. (2014). Good practice in large-scale learning for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3), 507-520.

Digital Library

[3]

Bekkerman, R., & Allan, J. (2004). Using bigrams in text categorization. Technical Report IR-408 Department of Computer Science, University of Massachusetts, Amherst, MA.

[4]

Berg, A. C., Berg, T. L., Daume, H., Dodge, J., Goyal, A., Han, X., Mensch, A., Mitchell, M., Sood, A., & Stratos, K., et al. (2012). Understanding and predicting importance in images. In CVPR, pp. 3562-3569.

[5]

Berg, T., Berg, A., & Shih, J. (2010). Automatic attribute discovery and characterization from noisy web data. In ECCV.

[6]

Bottou, L., & Bousquet, O. (2007). The tradeoffs of large scale learning. In NIPS.

[7]

Chatfield, K., Lempitsky, V., Vedaldi, A., & Zisserman, A. (2011). The devil is in the details: An evaluation of recent feature encoding methods. In BMVC.

[8]

Chatterjee, A. (2011). Neuroaesthetics: A coming of age story. Journal of Cognitive Neuroscience, 23(1), 53-62.

Digital Library

[9]

Clinchant, S., Csurka, G., Perronnin, F., & Renders, J. M. (2007). Xrce participation to ImageEval. In ImageEval Workshop at CVIR.

[10]

Cramer, D., & Howitt, D. (2004). The SAGE dictionary of statistics, 1st Edn. SAGE, p. 21 (entry "ceiling effect"), p. 67 (entry "floor effect").

[11]

Crammer, K., & Singer, Y. (2002). On the algorithmic implementation of multiclass kernel-based vector machines. The Journal of Machine Learning Research, 2, 265-292.

Digital Library

[12]

Csurka, G., Dance, C., Fan, L., Willamowski, J., & Bray, C. (2004). Visual categorization with bags of keypoints. In ECCV SLCV Workshop.

[13]

Datta, R., & Wang, J. Z. (2010). Acquine: Aesthetic quality inference engine¿real-time automatic rating of photo aesthetics. In MIR.

[14]

Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2006). Studying aesthetics in photographic images using a computational approach. In ECCV.

[15]

Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2008). Algorithmic inferencing of aesthetics and emotion in natural images: An exposition. In ICIP.

[16]

Dhar, S., Ordonez, V., & Berg, T. (2011). High level describable attributes for predicting aesthetics and interestingness. In CVPR.

[17]

Donahue, J., & Grauman, K. (2011). Annotator rationales for visual recognition. In ICCV.

[18]

Duan, K., Parikh, D., Crandall, D., & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In CVPR.

[19]

Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009). Describing objects by their attributes. In CVPR.

[20]

Ferrari, V., & Zisserman, A. (2007). Learning visual attributes. In NIPS.

[21]

Geng, B., Yang, L., Xu, C., Hua, X., & Li, S. (2011). The role of attractiveness in web image search. In ACM-MM.

[22]

Gracyk, T. (2011). Hume's aesthetics. In: E. N. Zalta (Ed.) The Stanford encyclopedia of philosophy, winter 2011 edn.

[23]

Hammermeister, K. (2002). The German aesthetic tradition. Cambridge, MA: Cambridge University Press.

[24]

Hofmann, T. (2001). Unsupervised learning by probabilistic latent semantic analysis. Machine Learning, 42, 177-196.

[25]

Isola, P., Parikh, D., Torralba, A., & Oliva, A. (2011). Understanding the intrinsic memorability of images. In NIPS.

[26]

Jacobson, E., & Ostwald, W. (1946). The color harmony manual, large chip edition. Chicago: Container Corporation.

[27]

Jégou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search. IEEE TPAMI.

[28]

Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In ECML.

[29]

Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q., Wang, J., Li, J., et al. (2011). Aesthetics and emotions in images. IEEE on Signal Processing Magazine, 28(5), 94-115.

[30]

Ke, Y., Tang, X., & Jing, F. (2006). The design of high-level features for photo quality assessment. In CVPR.

[31]

Kodak. (1987). How to take good pictures: A photo guide (35th ed.). New York, NY: Ballantine Books.

[32]

Krages, B. (2005). Photography: The art of composition. New York, US: Allworth Press.

[33]

Lampert, C., Nickisch, H., & Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In CVPR.

[34]

Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR.

Digital Library

[35]

Leder, H., Belke, B., Oeberst, A., & Augustin, D. (2004). A model of aesthetic appreciation and aesthetic judgments. British Journal of Psychology, 95(4), 489-508.

[36]

Li, C., Loui, A. C., & Chen, T. (2010). Towards aesthetics: A photo quality assessment and photo selection system. In ACM-MM.

[37]

Lowe, D. (1999). Object recognition from local scale-invariant features. In ICCV.

[38]

Luo, W., Wang, X., & Tang, X. (2011). Content-based photo quality assessment. In ICCV.

[39]

Luo, Y., & Tang, X. (2008). Photo and video quality evaluation: Focusing on the subject. In ECCV.

[40]

Machajdik, J., & Hanbury, A. (2010). Affective image classification using features inspired by psychology and art theory. In ACM MM, New York, NY, USA.

[41]

Marchesotti, L., & Perronnin, F. (2013). Learning beautiful (and ugly) attributes. In BMVC.

[42]

Marchesotti, L., Perronnin, F., Larlus, D., & Csurka, G. (2011). Assessing the aesthetic quality of photographs using generic image descriptors. In ICCV.

[43]

Müller, H., Clough, P., Deselaers, T., & Caputo, B. (2010). ImageCLEF: Experimental evaluation in visual information retrieval (Vol. 32). Berlin: Springer.

[44]

Murray, N., Marchesotti, L., & Perronnin, F. (2012a). AVA: A large-scale database for aesthetic visual analysis. In CVPR.

[45]

Murray, N., Marchesotti, L., & Perronnin, F. (2012b) Learning to rank images using semantic and aesthetic labels. In BMVC.

[46]

Ng, A. Y., Jordan, M. I., & Weiss, Y., et al. (2002). On spectral clustering: Analysis and an algorithm. In NIPS.

[47]

Obrador, P., Schmidt-Hackenberg, L., & Oliver, N. (2010). The role of image composition in image aesthetics. In ICIP.

[48]

Obrador, P., Saad, M., Suryanarayan, P., & Oliver, N. (2012). Towards category-based aesthetic models of photographs. Advances in Multimedia Modeling, pp. 63-76.

[49]

Oliva, A., & Torralba, A. (2001). Modeling the shape of the scene: A holistic representation of the spatial envelope. In IJCV.

[50]

Orendovici, R., & Wang, J. (2010). Training data collection system for a learning-based photographic aesthetic quality inference engine. In ACM-MM.

[51]

Pang, B., Lee, L., & Vaithyanathan, S. (2012). Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing.

[52]

Parikh, D., & Grauman, K. (2011a). Interactively building a discriminative vocabulary of nameable attributes. In CVPR.

[53]

Parikh, D., & Grauman, K. (2011b). Relative attributes. In ICCV.

[54]

Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In CVPR.

[55]

Perronnin, F., Sánchez, J., & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In ECCV.

Digital Library

[56]

Riloff, E., Patwardhan, S., & Wiebe, J., et al. (2006). Feature subsumption for opinion analysis. In Proceedings of the 2006 conference on empirical methods in natural language processing.

[57]

Rohrbach, M., Stark, M., Szarvas, G., Gurevych, I., & Schiele, B. (2010). What helps where-and why? Semantic relatedness for knowledge transfer. In CVPR.

[58]

Russell, J. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178.

[59]

San Pedro, J., Yeh, T., & Oliver, N. (2012). Leveraging user comments for aesthetic aware image search reranking. In WWW.

[60]

Shelley, J. (2012a). 18th century british aesthetics. In: E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy, summer 2012 edn.

[61]

Shelley, J. (2012b). The concept of the aesthetic. In: E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy, spring 2012 edn.

[62]

Sivic, J., & Zisserman, A. (2003). Video Google: A text retrieval approach to object matching in videos. In ICCV.

Digital Library

[63]

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B (Methodological), 58, 267-288.

[64]

Wang, J., Markert, K., & Everingham, M. (2009). Learning models for object recognition from natural language descriptions. In BMVC.

[65]

Yanai, K., & Barnard, K. (2005). Image region entropy: A measure of visualness of web images associated with one concept. In ACM-MM.

[66]

Yao, L., Suryanarayan, P., Qiao, M., Wang, J., & Li, J. (2012). On-site composition and aesthetics feedback through exemplars for photographers. In IJCV, Oscar.

[67]

Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, 67, 301-320.

Cited By

He SXiao YMing AMa H(2025)Prompt-guided image color aesthetics assessmentInformation Fusion10.1016/j.inffus.2024.102706114:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.inffus.2024.102706
Debnath SRoy RChangder S(2024)A novel approach using deep convolutional neural network to classify the photographs based on leading-line by fine-tuning the pre-trained VGG16 neural networkMultimedia Tools and Applications10.1007/s11042-022-13338-583:1(3189-3214)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-022-13338-5
Chen WSu JSong WXu JZhu GLi YJi YLiu C(2024)Quality evaluation methods of handwritten Chinese characters: a comprehensive surveyMultimedia Systems10.1007/s00530-024-01396-830:4Online publication date: 4-Jul-2024
https://dl.acm.org/doi/10.1007/s00530-024-01396-8
Show More Cited By

Recommendations

Learning image aesthetic subjectivity from attribute-aware relational reasoning network
Highlights
- An attribute-aware relational reasoning network for image aesthetic assessment.
- People’s various preferences lead to subjectivity in aesthetic ratings on images.
- The diversity of aesthetic attributes in images affects people’s ...
Abstract
People’s various visual preferences lead to subjectivity in their aesthetic perception of images. Hence, learning image aesthetic subjectivity has attracted great interest in the computer vision community. People with different subjectivity often ...
Personality modeling from image aesthetic attribute-aware graph representation learning
Abstract
Recently, inferring users’ personality traits on social media has attracted extensive attention. Existing studies have shown that users’ personality traits can be inferred from their preferences for images. However, since users’ ...
Highlights
- We propose an attribute-aware graph representation learning for personality modeling.
Neural aesthetic image reviewer

Recently, there is a rising interest in perceiving image aesthetics. The existing works deal with image aesthetics as a classification or regression problem. To extend the cognition from rating to reasoning, a deeper understanding of aesthetics should be ...

Comments

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision

International Journal of Computer Vision Volume 113, Issue 3

July 2015

106 pages

ISSN:0920-5691

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

He SXiao YMing AMa H(2025)Prompt-guided image color aesthetics assessmentInformation Fusion10.1016/j.inffus.2024.102706114:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.inffus.2024.102706
Debnath SRoy RChangder S(2024)A novel approach using deep convolutional neural network to classify the photographs based on leading-line by fine-tuning the pre-trained VGG16 neural networkMultimedia Tools and Applications10.1007/s11042-022-13338-583:1(3189-3214)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-022-13338-5
Chen WSu JSong WXu JZhu GLi YJi YLiu C(2024)Quality evaluation methods of handwritten Chinese characters: a comprehensive surveyMultimedia Systems10.1007/s00530-024-01396-830:4Online publication date: 4-Jul-2024
https://dl.acm.org/doi/10.1007/s00530-024-01396-8
Brizzi MBattisti FCarli MNeri A(2023)Selective video enhancement in the Laguerre–Gauss domainImage Communication10.1016/j.image.2022.116876110:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.image.2022.116876
Lin XCheng JWang SSong KWang JPeng Z(2022)Exploring Users’ Text and Image Posting Behaviors in Weibo Mental Health CommunitiesProceedings of the Tenth International Symposium of Chinese CHI10.1145/3565698.3565797(277-281)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1145/3565698.3565797
Bian JShen X(2021)Sentiment Analysis of Chinese Paintings Based on Lightweight Convolutional Neural NetworkWireless Communications & Mobile Computing10.1155/2021/60972952021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6097295
Tang YHuang HShi XMao X(2020)Picture News Collection: A Dataset for Automatic Picture News Thumbnail SelectionWeb Information Systems Engineering – WISE 201910.1007/978-3-030-34223-4_29(458-472)Online publication date: 19-Jan-2020
https://dl.acm.org/doi/10.1007/978-3-030-34223-4_29
Cui CLiu HLian TNie LZhu LYin Y(2019)Distribution-Oriented Aesthetics Assessment With Semantic-Aware Hybrid NetworkIEEE Transactions on Multimedia10.1109/TMM.2018.287535721:5(1209-1220)Online publication date: 23-Apr-2019
https://dl.acm.org/doi/10.1109/TMM.2018.2875357
Zhang LYao YLu ZShao L(2019)Aesthetics-Guided Graph Clustering With Absent Modalities ImputationIEEE Transactions on Image Processing10.1109/TIP.2019.289794028:7(3462-3476)Online publication date: 21-May-2019
https://dl.acm.org/doi/10.1109/TIP.2019.2897940
Wang GYan JQin Z(2018)Collaborative and attentive learning for personalized image aesthetic assessmentProceedings of the 27th International Joint Conference on Artificial Intelligence10.5555/3304415.3304551(957-963)Online publication date: 13-Jul-2018
https://dl.acm.org/doi/10.5555/3304415.3304551
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents