Detecting and Explaining Emotions in Video Advertisements
Pages 2734 - 2738
Abstract
The use of video advertisements is a common marketing strategy in today's digital age. Extensive research is conducted by companies to comprehend the emotions conveyed in video advertisements, as they play a crucial role in crafting memorable commercials. Understanding and explaining these abstract concepts in videos is an unsolved problem. There is a large body of work that tries to predict human emotion or activity from videos, however, this is not sufficient. In this paper, we propose a novel framework for detecting and, most importantly, explaining emotions in video advertisements. Our framework consists of two main stages: emotion detection and explanation generation. We use a deep learning model to detect the underlying emotions of a video advertisement and generate visual explanations to give insight into our model's predictions. We demonstrate our system on a dataset of video advertisements and show that our framework can accurately detect and explain emotions in video advertisements. Our results suggest that our novel algorithm has the potential to explain decisions from any video classification model.
References
[1]
Sarah Adel Bargal, Emad Barsoum, Cristian Canton Ferrer, and Cha Zhang. 2016. Emotion recognition in the wild from videos using images. In Proceedings of the 18th ACM international conference on multimodal interaction. 433--436.
[2]
Lei Chen, Su-Youn Yoon, Chee Wee Leong, Michelle Martin, and Min Ma. 2014. An Initial Analysis of Structured Video Interviews by Using Multimodal Emotion Detection. In Proceedings of the 2014 Workshop on Emotion Representation and Modelling in Human-Computer-Interaction-Systems (Istanbul, Turkey) (ERM4HCI '14). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/2668056.2668057
[3]
Yen-Liang Chen, Chia-Ling Chang, and Chin-Sheng Yeh. 2017. Emotion classification of YouTube videos. Decision Support Systems, Vol. 101 (2017), 40--50.
[4]
Samira Ebrahimi Kahou, Vincent Michalski, Kishore Konda, Roland Memisevic, and Christopher Pal. 2015. Recurrent neural networks for emotion recognition in video. In Proceedings of the 2015 ACM on international conference on multimodal interaction. 467--474.
[5]
Prashant Gohel, Priyanka Singh, and Manoranjan Mohanty. 2021. Explainable AI: current status and future directions. CoRR, Vol. abs/2107.07045 (2021). showeprint[arXiv]2107.07045 https://arxiv.org/abs/2107.07045
[6]
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018. ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2594--2604. https://doi.org/10.18653/v1/D18--1280
[7]
Andreas Holzinger, Anna Saranti, Christoph Molnar, Przemyslaw Biecek, and Wojciech Samek. 2022. Explainable AI Methods - A Brief Overview. Springer International Publishing, Cham, 13--38. https://doi.org/10.1007/978--3-031-04083--2_2
[8]
Fatima Hussain, Rasheed Hussain, and Ekram Hossain. 2021. Explainable Artificial Intelligence (XAI): An Engineering Perspective. CoRR, Vol. abs/2101.03613 (2021). showeprint[arXiv]2101.03613 https://arxiv.org/abs/2101.03613
[9]
Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. 2017. Automatic understanding of image and video advertisements. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1705--1715.
[10]
Maximilian A. Köhl, Kevin Baum, Markus Langer, Daniel Oster, Timo Speith, and Dimitri Bohlender. 2019. Explainability as a Non-Functional Requirement. In 2019 IEEE 27th International Requirements Engineering Conference (RE). 363--368. https://doi.org/10.1109/RE.2019.00046
[11]
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. CoRR, Vol. abs/1705.07874 (2017). showeprint[arXiv]1705.07874 http://arxiv.org/abs/1705.07874
[12]
Scott McCoy, Andrea Everard, Peter Polak, and Dennis F Galletta. 2007. The effects of online advertising. Commun. ACM, Vol. 50, 3 (2007), 84--88.
[13]
M Miró-Nicolau, A Jaume-i Capó, and G Moyà-Alcover. 2023. Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasets. arXiv preprint arXiv:2311.01961 (2023).
[14]
Miquel Miró-Nicolau, Antoni Jaume-i Capó, and Gabriel Moyà-Alcover. 2024. A comprehensive study on fidelity metrics for XAI. arXiv preprint arXiv:2401.10640 (2024).
[15]
Karen Nelson-Field, Erica Riebe, and Kellie Newstead. 2013. The emotions that drive viral video. Australasian Marketing Journal, Vol. 21, 4 (2013), 205--211.
[16]
Yagya Raj Pandeya, Bhuwan Bhattarai, and Joonwhoan Lee. 2021. Deep-learning-based multimodal emotion classification for music videos. Sensors, Vol. 21, 14 (2021), 4927.
[17]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. CoRR, Vol. abs/2103.00020 (2021). showeprint[arXiv]2103.00020 https://arxiv.org/abs/2103.00020
[18]
Munaf Rashid, SAR Abu-Bakar, and Musa Mokji. 2013. Human emotion recognition from videos using spatio-temporal and audio features. The Visual Computer, Vol. 29 (2013), 1269--1275.
[19]
Atiq Rehman and Samir Brahim Belhaouari. 2023. Deep learning for video classification: A review. Authorea Preprints (2023).
[20]
Marco Tú lio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. CoRR, Vol. abs/1602.04938 (2016). showeprint[arXiv]1602.04938 http://arxiv.org/abs/1602.04938
[21]
Mirka Saarela and Lilia Geogieva. 2022. Robustness, stability, and fidelity of explanations for a deep skin cancer classification model. Applied Sciences, Vol. 12, 19 (2022), 9545.
[22]
P Tamil Selvi, P Vyshnavi, R Jagadish, Shravan Srikumar, and S Veni. 2017. Emotion recognition from videos using facial expressions. In Artificial Intelligence and Evolutionary Computations in Engineering Systems: Proceedings of ICAIECES 2016. Springer, 565--576.
[23]
Ugur Unal, I??l Yenido?an, Hasan Dag, and Aykut Cayir. 2020. Use Case Study: Data Science Application for Microsoft Malware Prediction Competition on Kaggle.
[24]
Valentin Vielzeuf, Stéphane Pateux, and Frédéric Jurie. 2017. Temporal multimodal fusion for video emotion classification in the wild. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 569--576.
[25]
Jin Wang. 2021. Classifying Emotion Types in Advertisement Videos by a Deep Learning Approach. Master's thesis. University of Glasgow.
Index Terms
- Detecting and Explaining Emotions in Video Advertisements
Recommendations
Spam detection in online classified advertisements
WebQuality '11: Proceedings of the 2011 Joint WICOW/AIRWeb Workshop on Web QualityOnline classified advertisements have become an essential part of the advertisement market. Popular online classified advertisement sites such as Craigslist, Ebay Classifieds, and Oodle have attracted a huge number of posts and visits. Due to its high ...
Designing online banner advertisements: should we animate?
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsA common medium for advertising on the Internet is the use of banner ads. This study investigates recall and recognition of animated banner advertisements in an attempt to identify design guidelines. It was hypothesized that animation would increase ...
Detecting variation of emotions in online activities
Detect wide spectrum of emotions from online text sources.Reveal social emotions and affective states from online social networks content.A case study where explicit and implicit emotion experiences are monitored.Online text sources features for emotion ...
Comments
Information & Contributors
Information
Published In
July 2024
3164 pages
ISBN:9798400704314
DOI:10.1145/3626772
- General Chairs:
- Grace Hui Yang,
- Hongning Wang,
- Sam Han,
- Program Chairs:
- Claudia Hauff,
- Guido Zuccon,
- Yi Zhang
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 July 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
SIGIR 2024
Sponsor:
SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 14 - 18, 2024
Washington DC, USA
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 59Total Downloads
- Downloads (Last 12 months)59
- Downloads (Last 6 weeks)27
Reflects downloads up to 13 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in