research-article

Beauty Is in the Eye of the Beholder: Demographically Oriented Analysis of Aesthetics in Photographs

Authors:

Magzhan Kairanbay,

Lai-Kuan WongAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 15, Issue 2s

Article No.: 63, Pages 1 - 21

https://doi.org/10.1145/3328993

Published: 25 July 2019 Publication History

Abstract

Aesthetics is a subjective concept that is likely to be perceived differently among people of different ages, genders, and cultural backgrounds. While techniques that directly compute this concept in images has seen increasing attention by the multimedia and machine-learning community, there are very few attempts at encoding the influences from the photographer’s viewpoint. This work demonstrates how the aesthetic quality of photos can be better learned by accounting for the demographic background of a photographer. A new AVA-PD (Photographer Demographic) dataset is created to supplement the AVA dataset by providing photographers’ age, gender and location attributes. Two deep convolutional neural network (CNN) architectures are proposed to utilize demographic information for aesthetic prediction of photos; both are shown to yield better prediction capabilities compared to most existing approaches. By leveraging on AVA-PD meta-data, we also present some additional machine-learnable tasks such as identifying the photographer and predicting photography styles from a person’s gallery of photos.

References

[1]

Yaniv Bar, Noga Levy, and Lior Wolf. 2014. Classification of artistic styles using binarized features derived from a deep neural network. In Proceedings of the Workshop at the European Conference on Computer Vision. 71--84.

[2]

Peter Niclas Broer, Sabrina Juran, Yuen-Jong Liu, Katie Weichman, Neil Tanna, Marc E. Walker, Reuben Ng, and John A. Persing. 2014. The impact of geographic, ethnic, and demographic dynamics on the perception of beauty. J. Cranio. Surg. 25, 2 (2014), e157--e161.

[3]

Camilo J. Cela-Conde, Francisco J. Ayala, Enric Munar, Fernando Maestú, Marcos Nadal, Miguel A. Capó, David del Río, Juan J. López-Ibor, Tomás Ortiz, Claudio Mirasso et al. 2009. Sex-related similarities and differences in the neural correlates of beauty. Proc. Nat. Acad. Sci. 106, 10 (2009), 3847--3852.

[4]

Wei-Ta Chu and Yi-Ling Wu. 2016. Deep correlation features for image style classification. In Proceedings of the ACM on Multimedia Conference. ACM, 402--406.

Digital Library

[5]

Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Proceedings of the European Conference on Computer Vision. Springer, 288--301.

Digital Library

[6]

Ritendra Datta, Jia Li, and James Z. Wang. 2007. Learning the consensus on visual quality for next-generation image management. In Proceedings of the 15th ACM International Conference on Multimedia. ACM, 533--536.

Digital Library

[7]

Sagnik Dhar, Vicente Ordonez, and Tamara L. Berg. 2011. High level describable attributes for predicting aesthetics and interestingness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 1657--1664.

Digital Library

[8]

Zhe Dong, Xu Shen, Houqiang Li, and Xinmei Tian. 2015. Photo quality assessment with DCNN that understands image well. In Proceedings of the Conference on Multimedia Modeling (MMM’15). 524--535.

[9]

Yong-Lian Hii, John See, Magzhan Kairanbay, and Lai-Kuan Wong. 2017. Multigap: Multi-pooled inception network with text augmentation for aesthetic prediction of photographs. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 1722--1726.

[10]

Thomas Jacobsen. 2010. Beauty and the brain: Culture, history and individual differences in aesthetic appreciation. J. Anatomy 216, 2 (2010), 184--191.

[11]

Xin Jin, Jingying Chi, Siwei Peng, Yulu Tian, Chaochen Ye, and Xiaodong Li. 2016. Deep image aesthetics classification using inception modules and fine-tuning connected layer. In Proceedings of the 8th International Conference on Wireless Communications 8 Signal Processing (WCSP’16). IEEE, 1--6.

[12]

Magzhan Kairanbay, John See, and Lai-Kuan Wong. 2016. Aesthetic evaluation of facial portraits using compositional augmentation for deep CNNs. In Proceedings of the Asian Conference on Computer Vision. Springer, 462--474.

[13]

Magzhan Kairanbay, John See, and Lai-Kuan Wong. 2018. Towards demographic-based photographic aesthetics prediction for portraitures. In Proceedings of the International Conference on Multimedia Modeling. Springer, 531--543.

[14]

Magzhan Kairanbay, John See, Lai-Kuan Wong, and Yong-Lian Hii. 2017. Filling the gaps: Reducing the complexity of networks for multi-attribute image aesthetic prediction. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 3051--3055.

[15]

Yueying Kao, Ran He, and Kaiqi Huang. 2016a. Visual aesthetic quality assessment with multi-task deep learning. Retrieved from arXiv preprint arXiv:1604.049705 (2016).

[16]

Yueying Kao, Ran He, and Kaiqi Huang. 2017. Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Proc. 26, 3 (2017), 1482--1495.

Digital Library

[17]

Yueying Kao, Kaiqi Huang, and Steve Maybank. 2016b. Hierarchical aesthetic quality assessment using deep convolutional neural networks. Sig. Proc.: Image Commun. 47 (2016), 500--510.

Digital Library

[18]

Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller. 2013. Recognizing image style. Retrieved from: arXiv preprint arXiv:1311.3715.

[19]

Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of high-level features for photo quality assessment. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 1. IEEE, 419--426.

Digital Library

[20]

Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 662--679.

[21]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 1097--1105.

Digital Library

[22]

Challenging Technologies LLC. 2018. DpChallenge dataset. Retrieved from: http://www.dpchallenge.com/.

[23]

Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2014. Rapid: Rating pictorial aesthetics using deep learning. In Proceedings of the 22nd ACM International Conference on Multimedia. ACM, 457--466.

Digital Library

[24]

Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2015a. Rating image aesthetics using deep learning. IEEE Trans. Multimed. 17, 11 (2015), 2021--2034.

Digital Library

[25]

Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z. Wang. 2015b. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In Proceedings of the IEEE International Conference on Computer Vision. 990--998.

Digital Library

[26]

Long Mai, Hailin Jin, and Feng Liu. 2016. Composition-preserving deep photo aesthetics assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 497--506.

[27]

Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’11). IEEE, 1784--1791.

Digital Library

[28]

Takahiko Masuda, Richard Gonzalez, Letty Kwan, and Richard E. Nisbett. 2008. Culture and aesthetic preference: Comparing the attention to context of East Asians and Americans. Person. Soc. Psych. Bull. 34, 9 (2008), 1260--1275.

[29]

Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 2408--2415.

Digital Library

[30]

Pere Obrador, Xavier Anguera, Rodrigo de Oliveira, and Nuria Oliver. 2009. The role of tags and image aesthetics in social image search. In Proceedings of the 1st SIGMM Workshop on Social Media. ACM, 65--72.

Digital Library

[31]

Pere Obrador, Ludwig Schmidt-Hackenberg, and Nuria Oliver. 2010. The role of image composition in image aesthetics. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP’10). IEEE, 3185--3188.

[32]

Miriam Redi, Nikhil Rasiwasia, Gaurav Aggarwal, and Alejandro Jaimes. 2015. The beauty of capturing faces: Rating the quality of digital portraits. In Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’15), Vol. 1. IEEE, 1--8.

[33]

Jian Ren, Xiaohui Shen, Zhe Lin, Radomír Mech, and David J. Foran. 2017. Personalized image aesthetics. In Proceedings of the IEEE International Conference on Computer Vision. 638--647.

[34]

Andreas E. Savakis, Stephen P. Etz, and Alexander C. P. Loui. 2000. Evaluation of image appeal in consumer photography. In Proceedings of the Conference on Human Vision and Electronic Imaging V, Vol. 3959. International Society for Optics and Photonics, 111--121.

[35]

Katharina Schwarz, Patrick Wieschollek, and Hendrik P. A. Lensch. 2018. Will people like your image? Learning the aesthetic space. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV’18). 2048--2057.

[36]

Tiancheng Sun, Yulong Wang, Jian Yang, and Xiaolin Hu. 2017. Convolution neural networks with two pathways for image style recognition. IEEE Trans. Image Proc. 26, 9 (2017), 4102--4113.

Digital Library

[37]

Xiaoshuai Sun, Hongxun Yao, Rongrong Ji, and Shaohui Liu. 2009. Photo assessment based on computational visual attention model. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 541--544.

Digital Library

[38]

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818--2826.

[39]

Christopher Thomas and Adriana Kovashka. 2016. Seeing behind the camera: Identifying the authorship of a photograph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3494--3502.

[40]

Stanford University. 2018. ImageNet dataset. Retrieved from: http://www.image-net.org/.

[41]

Guolong Wang, Junchi Yan, and Zheng Qin. 2018. Collaborative and attentive learning for personalized image aesthetic assessment. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’18). 957--963.

Digital Library

[42]

Weining Wang, Mingquan Zhao, Li Wang, Jiexiong Huang, Chengjia Cai, and Xiangmin Xu. 2016. A multi-scene deep learning model for image aesthetic evaluation. Sig. Proc.: Image Commun. 47 (2016), 511--518.

Digital Library

[43]

Zhangyang Wang, Ding Liu, Shiyu Chang, Florin Dolcos, Diane Beck, and Thomas Huang. 2017. Image aesthetics assessment using Deep Chatterjee’s machine. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 941--948.

[44]

Lai-Kuan Wong and Kok-Lim Low. 2009. Saliency-enhanced image aesthetics class prediction. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP’09). IEEE, 997--1000.

Digital Library

[45]

Luming Zhang, Yue Gao, Roger Zimmermann, Qi Tian, and Xuelong Li. 2014. Fusion of multichannel local and global structural cues for photo aesthetics evaluation. IEEE Trans. Image Proc. 23, 3 (2014), 1419--1429.

Digital Library

Cited By

Shi TChen CWu ZHao AFang Y(2024)Improving Image Aesthetic Assessment via Multiple Image Joint LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368712820:11(1-24)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3687128
Jin XLi XLou HFan CDeng QXiao CCui SSingh A(2023)Aesthetic Attribute Assessment of Images Numerically on Mixed Multi-attribute DatasetsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/354714418:3s(1-16)Online publication date: 20-Mar-2023
https://dl.acm.org/doi/10.1145/3547144
Maître H(2023)ReferencesAesthetics in Digital Photography10.1002/9781394225972.refs(271-294)Online publication date: 14-Jul-2023
https://doi.org/10.1002/9781394225972.refs
Show More Cited By

Index Terms

Beauty Is in the Eye of the Beholder: Demographically Oriented Analysis of Aesthetics in Photographs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Recommendations

Web Menus with Beauty and Brains with Cdrom
The Photographer's Eye Field Guide: The essential handbook for traveling with your digital SLR camera
In the eye of the beholder - perception of indeterminate art
Computational Aesthetics'07: Proceedings of the Third Eurographics conference on Computational Aesthetics in Graphics, Visualization and Imaging

How do we interpret an object - a scene - a painting? Perception research and art illuminate from different angles how the vast amount of information in our visually perceived environment is processed by the viewer to form a coherent and consistent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 15, Issue 2s

Special Section on Cross-Media Analysis for Visual Question Answering, Special Section on Big Data, Machine Learning and AI Technologies for Art and Design and Special Section on MMSys/NOSSDAV 2018

April 2019

381 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3343360

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2019

Accepted: 01 April 2019

Revised: 01 February 2019

Received: 01 July 2018

Published in TOMM Volume 15, Issue 2s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

MMU-GRA Scheme, Multimedia University
Ministry of Higher Education, Malaysia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
439
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shi TChen CWu ZHao AFang Y(2024)Improving Image Aesthetic Assessment via Multiple Image Joint LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368712820:11(1-24)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3687128
Jin XLi XLou HFan CDeng QXiao CCui SSingh A(2023)Aesthetic Attribute Assessment of Images Numerically on Mixed Multi-attribute DatasetsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/354714418:3s(1-16)Online publication date: 20-Mar-2023
https://dl.acm.org/doi/10.1145/3547144
Maître H(2023)ReferencesAesthetics in Digital Photography10.1002/9781394225972.refs(271-294)Online publication date: 14-Jul-2023
https://doi.org/10.1002/9781394225972.refs
Leder HHakala JPeltoketo VValuch CPelowski M(2022)Swipes and Saves: A Taxonomy of Factors Influencing Aesthetic Assessments and Perceived Beauty of Mobile Phone PhotographsFrontiers in Psychology10.3389/fpsyg.2022.78697713Online publication date: 28-Feb-2022
https://doi.org/10.3389/fpsyg.2022.786977
Lima AGresse von Wangenheim C(2021)Assessing the Visual Esthetics of User Interfaces: A Ten-Year Systematic MappingInternational Journal of Human–Computer Interaction10.1080/10447318.2021.192611838:2(144-164)Online publication date: 13-Jun-2021
https://doi.org/10.1080/10447318.2021.1926118
Valenzise GKang CDufaux F(2021)Advances and Challenges in Computational Image AestheticsHuman Perception of Visual Information10.1007/978-3-030-81465-6_6(133-181)Online publication date: 22-Jul-2021
https://doi.org/10.1007/978-3-030-81465-6_6
Ching JSee JWong L(2020)Learning Image Aesthetics by Learning Inpainting2020 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP40778.2020.9191130(2246-2250)Online publication date: Oct-2020
https://doi.org/10.1109/ICIP40778.2020.9191130
Ghosh H(2020)ReferencesComputational Models for Cognitive Vision10.1002/9781119527886.refs(187-213)Online publication date: 6-Jul-2020
https://doi.org/10.1002/9781119527886.refs

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents