Abstract
Multimodal sentiment analysis aims to predict human sentiment polarity with multiple modalities. Most existing methods focus on directly integrating original modal features into multimodal fusion, ignoring the redundancy and heterogeneity across modalities. In this paper, we propose a simple but efficient Adaptive Token Selection and Fusion Network (ATSFN) to mitigate the effect of redundancy and heterogeneity. ATSFN employs adaptive trainable tokens to extract unimodal informative tokens and perform dynamic multimodal token fusion. Specifically, we first integrate critical information from original features into adaptive selection tokens through token selection transformers. Sentiment features flow through these smaller sequences of tokens to capture important information while reducing redundancy. Next, we introduce a token fusion transformer to fuse multimodal features dynamically. It adaptively estimates the unique contribution of each modality to sentiment tendencies through learnable fusion tokens. Experiments on two benchmark datasets demonstrate that our proposed approach achieves competitive performance and significant improvements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baltrušaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE (2016)
Carion, Nicolas, Massa, Francisco, Synnaeve, Gabriel, Usunier, Nicolas, Kirillov, Alexander, Zagoruyko, Sergey: End-to-End Object Detection with Transformers. In: Vedaldi, Andrea, Bischof, Horst, Brox, Thomas, Frahm, Jan-Michael. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Degottex, G., Kane, J., Drugman, T., Raitio, T., Scherer, S.: Covarep-a collaborative voice analysis repository for speech technologies. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2014)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv preprint. arXiv:1810.04805 (2018)
Du, Pengfei, Gao, Yali, Li, Xiaoyong: Bi-attention Modal Separation Network for Multimodal Video Fusion. In: Þór Jónsson, Björn., Gurrin, Cathal, Tran, Minh-Triet., Dang-Nguyen, Duc-Tien., Hu, Anita Min-Chun., Huynh Thi Thanh, Binh, Huet, Benoit (eds.) MMM 2022. LNCS, vol. 13141, pp. 585–598. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98358-1_46
Han, W., Chen, H., Gelbukh, A., Zadeh, A., Morency, L.P., Poria, S.: Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 6–15 (2021)
Han, W., Chen, H., Poria, S.: Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Nov 2021)
Hazarika, D., Zimmermann, R., Poria, S.: Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (2020)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Iashin, V., Xie, W., Rahtu, E., Zisserman, A.: Sparse in space and time: Audio-visual synchronisation with trainable selectors. In: 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November, pp. 21–24, 2022. BMVA Press (2022)
Jin, T., Huang, S., Li, Y., Zhang, Z.: Dual low-rank multimodal fusion. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 377–387 (2020)
Kumar, A., Vepa, J.: Gated mechanism for attention based multi modal sentiment analysis. In: ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2020)
Lin, Z., et al.: Modeling intra-and inter-modal relations: Hierarchical graph contrastive learning for multimodal sentiment analysis. In: Proceedings of the 29th International Conference on Computational Linguistics (2022)
Morency, L.P., Mihalcea, R., Doshi, P.: Towards multimodal sentiment analysis: Harvesting opinions from the web. In: Proceedings of the 13th International Conference on Multimodal Interfaces, pp. 169–176 (2011)
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., Sun, C.: Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems 34 (2021)
Nguyen, D., Nguyen, K., Sridharan, S., Dean, D., Fookes, C.: Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition. Comput. Vis. Image Underst. 174, 33–42 (2018)
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., Morency, L.P.: Deep multimodal fusion for persuasiveness prediction. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction (2016)
Poria, S., Chaturvedi, I., Cambria, E., Hussain, A.: Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE (2016)
Rahman, W., et al.: Integrating multimodal information in large pretrained transformers. In: Proceedings of the Conference. Association for Computational Linguistics. Meeting. vol. 2020. NIH Public Access (2020)
Sun, H., Wang, H., Liu, J., Chen, Y.W., Lin, L.: CubeMLP: An MLP-based model for multimodal sentiment analysis and depression estimation. In: Proceedings of the 30th ACM International Conference on Multimedia (2022)
Sun, Z., Sarma, P., Sethares, W., Liang, Y.: Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34 (2020)
Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., Salakhutdinov, R.: Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the Conference. Association for Computational Linguistics. Meeting. vol. 2019. NIH Public Access (2019)
Tsai, Y.H.H., Liang, P.P., Zadeh, A., Morency, L.P., Salakhutdinov, R.: Learning factorized multimodal representations. In: International Conference on Learning Representations (2019)
Vaswani, A., Shazeer, et al.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
Wang, Y., Shen, Y., Liu, Z., Liang, P.P., Zadeh, A., Morency, L.P.: Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33 (2019)
Wu, C.H., Liang, W.B.: Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels. IEEE Trans. Affect. Comput. 2(1), 10–21 (2010)
Yu, W., Xu, H., Yuan, Z., Wu, J.: Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35 (2021)
Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.P.: Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017)
Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016)
Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., Morency, L.P.: Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2018)
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., Kong, X.: Multimodal sentiment analysis based on fusion methods: a survey. Inf. Fusion 95, 306–325 (2023)
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No.62272025 and No.U22B2021) and the Fund of the State Key Laboratory of Software Development Environment.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, X., Lu, M., Guo, Z., Zhang, X. (2024). Adaptive Token Selection and Fusion Network for Multimodal Sentiment Analysis. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14556. Springer, Cham. https://doi.org/10.1007/978-3-031-53311-2_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-53311-2_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53310-5
Online ISBN: 978-3-031-53311-2
eBook Packages: Computer ScienceComputer Science (R0)