research-article

Increasing Image Memorability with Neural Style Transfer

Authors:

Aliaksandr Siarohin,

Cveta Majtanovic,

Xavier Alameda-Pineda,

Elisa Ricci, and

Nicu SebeAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 15, Issue 2

Article No.: 42, Pages 1 - 22

https://doi.org/10.1145/3311781

Published: 05 June 2019 Publication History

Abstract

Recent works in computer vision and multimedia have shown that image memorability can be automatically inferred exploiting powerful deep-learning models. This article advances the state of the art in this area by addressing a novel and more challenging issue: “Given an arbitrary input image, can we make it more memorable?” To tackle this problem, we introduce an approach based on an editing-by-applying-filters paradigm: given an input image, we propose to automatically retrieve a set of “style seeds,” i.e., a set of style images that, applied to the input image through a neural style transfer algorithm, provide the highest increase in memorability. We show the effectiveness of the proposed approach with experiments on the publicly available LaMem dataset, performing both a quantitative evaluation and a user study. To demonstrate the flexibility of the proposed framework, we also analyze the impact of different implementation choices, such as using different state-of-the-art neural style transfer methods. Finally, we show several qualitative results to provide additional insights on the link between image style and memorability.

References

[1]

Peter P. Aitken. 1974. Judgments of pleasingness and interestingness as functions of visual complexity.J. Exper. Psychol. 103, 2 (1974), 240.

[2]

Afsheen Rafaqat Ali and Mohsen Ali. 2017. Automatic image transformation for inducing affect. In Proceedings of the British Machine Vision Conference (BMVC’17).

[3]

Daniel E. Berlyne. 1960. Conflict, Arousal, and Curiosity. McGraw-Hill Book Company.

[4]

Daniel E. Berlyne. 1963. Complexity and incongruity variables as determinants of exploratory choice and evaluative ratings.Canadian J. Psychol./Revue 17, 3 (1963), 274.

[5]

Zoya Bylinskii, Phillip Isola, Constance Bainbridge, Antonio Torralba, and Aude Oliva. 2015. Intrinsic and extrinsic effects on image memorability. Vision Res. 116 (2015), 165--178.

[6]

Alex J. Champandard. 2016. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016).

[7]

Russell Eisenman. 1966. Pleasing and interesting visual complexity: Support for Berlyne. Percept. Motor Skills 23, 3 suppl. (1966), 1167--1170.

[8]

Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).

[9]

Alvin G. Goldstein and June E. Chance. 1971. Visual recognition memory for complex configurations. Atten. Percept. Psychophys. 9, 2 (1971), 237--241.

[10]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press.

Digital Library

[11]

Helmut Grabner, Fabian Nater, Michel Druey, and Luc Van Gool. 2013. Visual interestingness in image sequences. In Proceedings of the ACM Multimedia Conference.

Digital Library

[12]

Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool. 2013. The interestingness of images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). 1633--1640.

Digital Library

[13]

Raisa Halonen, Stina Westman, and Pirkko Oittinen. 2011. Naturalness and interestingness of test images for visual quality evaluation. In Proceedings of the IS&T/SPIE Electronic Imaging Conference. International Society for Optics and Photonics, 78670Z--78670Z.

[14]

Li He, Hairong Qi, and Russell Zaretzki. 2015. Image color transfer to evoke different emotions based on color combinations. Signal, Image Video Process. 9, 8 (2015), 1965--1973.

[15]

Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).

[16]

Juan Huo. 2016. An image complexity measurement algorithm with visual memory capacity and an EEG study. In Proceedings of the SAI Computing Conference (SAI’16). IEEE, 264--268.

[17]

Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011. Understanding the intrinsic memorability of images. In Advances in Neural Information Processing Systems. MIT Press.

Digital Library

[18]

Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. 2014. What makes a photograph memorable?IEEE Trans. Pattern Anal. Mach. Intell. 36, 7 (2014), 1469--1482.

Digital Library

[19]

Aditya Khosla. 2017. Predicting human behavior using visual media. http://hdl.handle.net/1721.1/109001.

[20]

Aditya Khosla, Wilma Bainbridge, Antonio Torralba, and Aude Oliva. 2013. Modifying the memorability of face photographs. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13).

Digital Library

[21]

Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva. 2015. Understanding and predicting image memorability at a large scale. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15).

Digital Library

[22]

Aditya Khosla, Jianxiong Xiao, Phillip Isola, Antonio Torralba, and Aude Oliva. 2012. Image memorability and visual inception. In Proceedings of the Conference and Exhibition on Computer Graphics 8 Interactive Techniques in Asia (SIGGRAPHAsia’12). ACM.

Digital Library

[23]

Hye-Rin Kim, Henry Kang, and In-Kwon Lee. 2016. Image recoloring with valence-arousal emotion model. In Proceedings of the Computer Graphics Forum, vol. 35. 209--216.

Digital Library

[24]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. MIT Press.

Digital Library

[25]

James L. McGaugh and Larry Cahill. 1995. A novel demonstration of enhanced memory associated with emotional arousal. Consciousness and Cognition 4, 4 (1995), 410--421.

[26]

David R. Lide. 2018. Handbook of mathematical functions. In A Century of Excellence in Measurements, Standards, and Technology. CRC Press, 135--139.

[27]

Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep photo style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).

[28]

Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia.

Digital Library

[29]

Stephen Maren. 1999. Long-term potentiation in the amygdala: A mechanism for emotional learning and memory. Trends Neurosci. 22, 12 (1999), 561--567.

[30]

Weijie Mao, Mengjuan Fei, and Wei Jiang. 2018. Creating memorable video summaries that satisfy the user’s intention for taking the videos. Neurocomputing 275 (2018), 1911--1920.

Digital Library

[31]

Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C. Gallagher. 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15).

[32]

Elizabeth A. Phelps. 2004. Human emotion and memory: Interactions of the amygdala and hippocampal complex. Curr. Opin. Neurobiol. 14, 2 (2004), 198--202.

[33]

Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2018. Artistic style transfer for videos and spherical images. Int. J. Comput. Vision 126, 11 (2018), 1199--1219.

Digital Library

[34]

Andreza Sartori, Victoria Yanulevskaya, Almila Akdag Salah, Jasper Uijlings, Elia Bruni, and Nicu Sebe. 2015. Affective analysis of professional and amateur abstract paintings using statistical analysis and art theory. ACM Trans. Interact. Intell. Syst. 5, 2 (2015), 8.

Digital Library

[35]

Sumit Shekhar, Srinivasa Madhava Phaneendra Angara, Manav Kedia, Dhruv Singal, and Akhil Sathyaprakash Shetty. 2017. Techniques for enhancing content memorability of user generated video content. U.S. Patent 9,805,269.

[36]

Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. 2018. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).

[37]

Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-Pineda, Elisa Ricci, and Nicu Sebe. 2017. How to make an image more memorable? A deep style transfer approach. In Proceedings of the International Conference on Multimedia Retrieval (ICMR’17).

Digital Library

[38]

Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Machine Learning(ICLR’15).

[39]

Mohammad Soleymani. 2015. The quest for visual interest. In Proceedings of the ACM International Conference on Multimedia.

Digital Library

[40]

Lionel Standing. 1973. Learning 10,000 pictures. Quart. J. Exp. Psychol. 25, 2 (1973), 207--222.

[41]

Lionel Standing, Jerry Conezio, and Ralph Norman Haber. 1970. Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonom. Sci. 19, 2 (1970), 73--74.

[42]

Noah Sulman and Thomas Sanocki. 2011. Color relations increase the capacity of visual short-term memory. Perception 40, 6 (2011).

[43]

Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture networks: Feed-forward synthesis of textures and stylized images. In Proceedings of the International Conference on Machine Learning (ICML’16).

Digital Library

[44]

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2017. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).

[45]

Wenguan Wang and Jianbing Shen. 2017. Deep cropping via attention box prediction and aesthetics assessment. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).

[46]

Hang Zhang and Kristin Dana. 2018. Multi-style generative network for real-time transfer. In Proceedings of the European Conference on Computer Vision Workshops (ECCV’18).

[47]

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In Advances in Neural Information Processing Systems. MIT Press.

Digital Library

Cited By

Liu JZheng ZYang ZYang Y(2024)High Fidelity Makeup via 2D and 3D Identity Preservation NetACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365647520:8(1-24)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3656475
Ma YZhao CHuang BLi XBasu A(2023)RAST: Restorable Arbitrary Style TransferACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363877020:5(1-21)Online publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1145/3638770
Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Show More Cited By

Index Terms

Increasing Image Memorability with Neural Style Transfer
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Edge Enhanced Image Style Transfer via Transformers
ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

In recent years, arbitrary image style transfer has attracted more and more attention. Given a pair of content and style images, a stylized one is hoped that retains the content from the former while catching style patterns from the latter. However, it ...
Read More
Aesthetic-Aware Image Style Transfer
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Style transfer aims to synthesize an image which inherits the content of one image while preserving a similar style of the other one. The "style'' of an image usually refers to its unique feeling conveyed from visual features, which is highly related to ...
Read More
Multi-Attention Network for Arbitrary Style Transfer
Neural Information Processing
Abstract
Arbitrary style transfer task is to synthesize a new image with the content of an image and the style of another image. With the development of deep learning, the effect and efficiency of arbitrary style transfer have been greatly improved. ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 15, Issue 2

May 2019

375 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3339884

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2019

Accepted: 01 February 2019

Revised: 01 February 2019

Received: 01 May 2018

Published in TOMM Volume 15, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
399
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Other Metrics

View Author Metrics

Citations

Cited By

Liu JZheng ZYang ZYang Y(2024)High Fidelity Makeup via 2D and 3D Identity Preservation NetACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365647520:8(1-24)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3656475
Ma YZhao CHuang BLi XBasu A(2023)RAST: Restorable Arbitrary Style TransferACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363877020:5(1-21)Online publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1145/3638770
Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Wang QLi SZhang XFeng GEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Rethinking Neural Style Transfer: Generating Personalized and Watermarked Stylized ImagesProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612202(6928-6937)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612202
Cummins SSweeney LSmeaton A(2022)Analysing the Memorability of a Procedural Crime-Drama TV Series, CSIProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549592(174-180)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549592
Wang QLi SZhang XFeng G(2022)Multi-granularity Brushstrokes Network for Universal Style TransferACM Transactions on Multimedia Computing, Communications, and Applications10.1145/350671018:4(1-17)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3506710
Lahrache SEl Ouazzani R(2022)A Survey on Image Memorability Prediction: From Traditional to Deep Learning Models2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)10.1109/IRASET52964.2022.9738055(1-10)Online publication date: 3-Mar-2022
https://doi.org/10.1109/IRASET52964.2022.9738055
Jing PShang YNie LSu YLiu JWang M(2021)Learning Low-Rank Sparse Representations With Robust Relationship Inference for Image Memorability PredictionIEEE Transactions on Multimedia10.1109/TMM.2020.300948523(2259-2272)Online publication date: 2021
https://doi.org/10.1109/TMM.2020.3009485
Bylinskii ZGoetschalckx LNewman AOliva A(2021)Memorability: An Image-Computable Measure of Information UtilityHuman Perception of Visual Information10.1007/978-3-030-81465-6_8(207-239)Online publication date: 22-Jul-2021
https://doi.org/10.1007/978-3-030-81465-6_8
Lu JXu MYang RWang Z(2020)Understanding and Predicting the Memorability of Outdoor Natural ScenesIEEE Transactions on Image Processing10.1109/TIP.2020.297595729(4927-4941)Online publication date: 2020
https://doi.org/10.1109/TIP.2020.2975957
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents