research-article

Emotion recognition using bimodal data fusion

Authors:

Dragoş Datcu,

Léon J. M. RothkrantzAuthors Info & Claims

CompSysTech '11: Proceedings of the 12th International Conference on Computer Systems and Technologies

Pages 122 - 128

https://doi.org/10.1145/2023607.2023629

Published: 16 June 2011 Publication History

Get Access

Abstract

This paper proposes a bimodal system for emotion recognition that uses face and speech analysis. Hidden Markov models - HMMs are used to learn and to describe the temporal dynamics of the emotion clues in the visual and acoustic channels. This approach provides a powerful method enabling to fuse the data we extract from separate modalities. The paper presents the best performing models and the results of the proposed recognition system.

References

[1]

Caridakis, G., L. Malatesta, L. Kessous, N. Amir, A. Raouzaiou, K. Karpouzis. Modeling naturalistic affective states via facial and vocal expressions recognition. In ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces, pages 146--154, New York, NY, USA, 2006.

Digital Library

Google Scholar

[2]

Ekman, P., W. V. Friesen. Facial action coding system: investigator's guide. Consulting Psychologists Press, Palo Alto, 1978.

Google Scholar

[3]

Fellenz, W. A., J. G. Taylor, R. Cowie, E. D. Cowie, F. Piat, S. D. Kollias, C. Orovas, B. Apolloni. On emotion recognition of faces and of speech using neuralnetworks, fuzzy logic and the assess system. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN'00, 2000.

Digital Library

Google Scholar

[4]

Freund, Y., R. E. Schapire. A decision-theoretic generalization of online learning and an application to boosting. Journal of Computer and System Science, 55:119--139, 1997.

Digital Library

Google Scholar

[5]

Han, M. J., J. H. Hsu, K. T. Song, F. Y. Chang. A new information fusion method for bimodal robotic emotion recognition. JCP, 3(7):39--47, 2008.

Google Scholar

[6]

Martin, O., I. Kotsia, B. Macq, I. Pita. The eNTERFACE' 05 Audio-Visual Emotion Database. In 22nd International Conference on Data Engineering Workshops. ICDEW'06, page 8, 2006.

Digital Library

Google Scholar

[7]

Ojala, T., M. Pietikainen, T. Maenpaa. Multiresolution grayscale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971--987, 2002.

Digital Library

Google Scholar

[8]

Paleari, M., C. L. Lisetti. Toward multimodal fusion of affective cues. In Proceedings of the 1st ACM international workshop on Human-centered multimedia, pages 99--108, 2006.

Digital Library

Google Scholar

[9]

Paleari, M., B. Huet. Toward emotion indexing of multimedia excerpts. In Content-Based Multimedia Indexing, 2008. CBMI 2008. International Workshop on, pages 425--432, June 2008.

Crossref

Google Scholar

[10]

Sebe N., I. Cohen, T. Gevers, T. S. Huang. Emotion recognition based on joint visual and audio cues. Pattern Recognition, International Conference on, 1:1136--1139, 2006.

Digital Library

Google Scholar

[11]

Song, M., J. Bu, C. Chen, N. Li. Audio-visual based emotion recognition, a new approach. Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, 2:1020--1025, 2004.

Digital Library

Google Scholar

[12]

Wimmer, M., B. Schuller, D. Arsic, B. Radig, G. Rigoll. Low-level fusion of audio and video feature for multi-modal emotion recognition. In 3rd International Conference on Computer Vision Theory and Applications. VISAPP, volume 2, pages 145--151, Madeira, Portugal, January 2008.

Google Scholar

[13]

Zeng, Z., Y. Hu, G. I. Roisman, Z. Wen, Y. Fu, T. S. Huang. Audio-visual spontaneous emotion recognition. In Artifical Intelligence for Human Computing, volume 4451 of Lecture Notes in Computer Science, pages 72--90. Springer, 2007.

Digital Library

Google Scholar

Cited By

View all

Cheng KTian LLi C(2024)LaWNet: Audio-Visual Emotion Recognition by Listening and Watching2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651101(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651101
Lu QSun XGao ZLong YFeng JZhang H(2024)Coordinated-joint translation fusion framework with sentiment-interactive graph convolutional networks for multimodal sentiment analysisInformation Processing & Management10.1016/j.ipm.2023.10353861:1(103538)Online publication date: Jan-2024
https://doi.org/10.1016/j.ipm.2023.103538
Cerna PUtutalum CEvangelista RDarkis AAsiri MMuallam-Darkis J(2023)An IOT-based Language Recognition System for Indigenous Languages using Integrated CNN and RNN2023 3rd International Conference on Smart Data Intelligence (ICSMDI)10.1109/ICSMDI57622.2023.00086(451-456)Online publication date: Mar-2023
https://doi.org/10.1109/ICSMDI57622.2023.00086
Show More Cited By

Index Terms

Emotion recognition using bimodal data fusion

Recommendations

Analysis of emotion recognition using facial expressions, speech and multimodal information
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on ...
Multi-actor emotion recognition in movies using a bimodal approach
MMM'11: Proceedings of the 17th international conference on Advances in multimedia modeling - Volume Part II

Approaches for emotion recognition in movie scenes using high level features, consider emotion of only a single actor. The contribution of this paper is to analyze using emotional information from multiple actors present in the scene instead of just one ...
Bimodal Emotion Recognition Based on Convolutional Neural Network
ICMLC '19: Proceedings of the 2019 11th International Conference on Machine Learning and Computing

Computer emotion recognition plays an important role in the field of artificial intelligence and is a key technology to realize human-machine interaction. Aiming at a cross-modal fusion problem of two nonlinear features of facial expression image and ...

Comments

Information & Contributors

Information

Published In

CompSysTech '11: Proceedings of the 12th International Conference on Computer Systems and Technologies

June 2011

688 pages

ISBN:9781450309172

DOI:10.1145/2023607

Editors:
Boris Rachev
Technical University of Varna, Varna, Bulgaria
,
Angel Smrikarov
University of Ruse, Ruse, Bulgaria

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CompSysTech '11

Sponsor:

TELECVB
Austrian Comp Soc
BPCSB
IOMAIBB
NBUBB
IOIACTBB
TSFPS
ERSVB
FOSEUB
UORB
BBPSB
CASTUVTB
TECHUVB
LLLPET
IEEEBSB

CompSysTech '11: 12th International Conference on Computer Systems and Technologies

June 16 - 17, 2011

Vienna, Austria

Acceptance Rates

Overall Acceptance Rate 241 of 492 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
361
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Cheng KTian LLi C(2024)LaWNet: Audio-Visual Emotion Recognition by Listening and Watching2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651101(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651101
Lu QSun XGao ZLong YFeng JZhang H(2024)Coordinated-joint translation fusion framework with sentiment-interactive graph convolutional networks for multimodal sentiment analysisInformation Processing & Management10.1016/j.ipm.2023.10353861:1(103538)Online publication date: Jan-2024
https://doi.org/10.1016/j.ipm.2023.103538
Cerna PUtutalum CEvangelista RDarkis AAsiri MMuallam-Darkis J(2023)An IOT-based Language Recognition System for Indigenous Languages using Integrated CNN and RNN2023 3rd International Conference on Smart Data Intelligence (ICSMDI)10.1109/ICSMDI57622.2023.00086(451-456)Online publication date: Mar-2023
https://doi.org/10.1109/ICSMDI57622.2023.00086
Sharma ASharma KKumar A(2022)Real-time emotional health detection using fine-tuned transfer networks with multimodal fusionNeural Computing and Applications10.1007/s00521-022-06913-235:31(22935-22948)Online publication date: 22-Jan-2022
https://doi.org/10.1007/s00521-022-06913-2
Lai HWu KLi L(2021)Multimodal emotion recognition with hierarchical memory networksIntelligent Data Analysis10.3233/IDA-20518325:4(1031-1045)Online publication date: 9-Jul-2021
https://doi.org/10.3233/IDA-205183
Matsane LJadhav AAjoodha R(2020)The use of Automatic Speech Recognition in Education for Identifying Attitudes of the Speakers2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE)10.1109/CSDE50874.2020.9411528(1-7)Online publication date: 16-Dec-2020
https://doi.org/10.1109/CSDE50874.2020.9411528
Lai HChen HWu S(2020)Different Contextual Window Sizes Based RNNs for Multimodal Emotion Detection in Interactive ConversationsIEEE Access10.1109/ACCESS.2020.30056648(119516-119526)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3005664
Shoumy NAng LMotiur Rahaman D(2019)Multimodal Big Data Affective AnalyticsMultimodal Analytics for Next-Generation Big Data Technologies and Applications10.1007/978-3-319-97598-6_3(45-71)Online publication date: 19-Jul-2019
https://doi.org/10.1007/978-3-319-97598-6_3
D'Mello SBosch NChen H(2018)Multimodal-multisensor affect detectionThe Handbook of Multimodal-Multisensor Interfaces10.1145/3107990.3107998(167-202)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1145/3107990.3107998
Hazarika DGorantla SPoria SZimmermann R(2018)Self-Attentive Feature-Level Fusion for Multimodal Emotion Detection2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR.2018.00043(196-201)Online publication date: Apr-2018
https://doi.org/10.1109/MIPR.2018.00043
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Analysis of emotion recognition using facial expressions, speech and multimodal information

Multi-actor emotion recognition in movies using a bimodal approach

Bimodal Emotion Recognition Based on Convolutional Neural Network