Abstract
This study aims to predict humor in a binary label, i.e., the presence or absence of humor in video recordings. The challenge here is to predict the variable in a cross-cultural manner, where the training data is in German, and the testing is done on the recordings of English-language-speaking football coaches. The novelty of this paper lies in exploring audio and textual features to predict humor in a cross-cultural setting. It is interesting to study audio and text-based features due to the cross-cultural nature of the problem, which remains largely unexplored when studying pose and facial features. The paper explores several audio (mms-lid, wav2vec 2.0) and textual (LaBSE, multilingual-e5-base) features and then uses them to train both RNN and Transformer encoder architectures. Experiments have been performed on the transformer encoder architectures with and without position encoding to study the effects of the absence of positional encoding in those features for humor detection. Late fusion has also been studied with combinations of all three modalities. We achieved our best AUC Score of 0.9251 and 0.8245 for the development and test set, respectively, out of five given submissions.
This research was supported by the Ministry of Science and ICT (MSIT) Korea under the National Research Foundation (NRF) Korea (NRF-2022R1A2C4001270), by the MSIT Korea Korea under the India-Korea Joint Programme of Cooperation in Science & Technology (NRF-2020K1A3A1A68093469), and by the ITRC (Information Technology Research Center) support program (IITP-2022-2020-0-01602) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)
Christ, L., et al.: The muse 2023 multimodal sentiment analysis challenge: mimicked emotions, cross-cultural humour, and personalisation. arXiv preprint arXiv:2305.03369 (2023)
Christ, L., et al.: The muse 2022 multimodal sentiment analysis challenge: humor, emotional reactions, and stress. In: Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pp. 5–14 (2022)
Christ, L., Amiriparian, S., Kathan, A., Müller, N., König, A., Schuller, B.W.: Multimodal prediction of spontaneous humour: a novel dataset and first results. arXiv preprint arXiv:2209.14272 (2022)
Christ, L., Amiriparian, S., Kathan, A., Müller, N., König, A., Schuller, B.W.: Multimodal prediction of spontaneous humour: a novel dataset and first results (2022)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
Ekman, P., Friesen, W.V.: Facial action coding system. Environ. Psychol. Nonverbal Behav. (1978)
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic bert sentence embedding (2022)
Pratap, V., et al.: Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516 (2023)
Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using Siamese bert-networks (2019)
Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
Wang, L., et al.: Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Singh, A.K., Ghosh, S., Kumar, A., Choi, B.J. (2024). Exploring Multimodal Features to Understand Cultural Context for Spontaneous Humor Prediction. In: Choi, B.J., Singh, D., Tiwary, U.S., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2023. Lecture Notes in Computer Science, vol 14531. Springer, Cham. https://doi.org/10.1007/978-3-031-53827-8_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-53827-8_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53826-1
Online ISBN: 978-3-031-53827-8
eBook Packages: Computer ScienceComputer Science (R0)