research-article

Discovering emotion and reasoning its flip in multi-party conversations using masked memory network and transformer

Authors:

Anubhav Shrimal,

Md Shad Akhtar,

Tanmoy ChakrabortyAuthors Info & Claims

Volume 240, Issue C

https://doi.org/10.1016/j.knosys.2021.108112

Published: 15 March 2022 Publication History

Abstract

Efficient discovery of a speaker’s emotional states in a multi-party conversation is significant to design human-like conversational agents. During a conversation, the cognitive state of a speaker often alters due to certain past utterances, which may lead to a flip in their emotional state. Therefore, discovering the reasons (triggers) behind the speaker’s emotion-flip during a conversation is essential to explain the emotion labels of individual utterances. In this paper, along with addressing the task of emotion recognition in conversations (ERC), we introduce a novel task – Emotion-Flip Reasoning (EFR), that aims to identify past utterances which have triggered one’s emotional state to flip at a certain time. We propose a masked memory network to address the former and a Transformer-based network for the latter task. To this end, we consider MELD, a benchmark emotion recognition dataset in multi-party conversations for the task of ERC, and augment it with new ground-truth labels for EFR. An extensive comparison with five state-of-the-art models suggests improved performances of our models for both the tasks. We further present anecdotal evidence and both qualitative and quantitative error analyses to support the superiority of our models compared to the baselines.

References

[1]

Ekman P., An argument for basic emotions, Cogn. Emot. (1992) 169–200.

[2]

Picard R.W., Affective Computing, MIT Press, Cambridge, MA, USA, 1997.

[3]

M. Abdul-Mageed, L. Ungar, Emonet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, 2017, pp. 718–728.

[4]

A. Chatterjee, K.N. Narahari, M. Joshi, P. Agrawal, SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text, in: Proceedings of the 13th international Workshop on Semantic Evaluation, Minneapolis, Minnesota, USA, 2019, pp. 39–48.

[5]

Akhtar M.S., Ghosal D., Ekbal A., Bhattacharyya P., Kurohashi S., All-in-one: Emotion, sentiment and intensity prediction using a multi-task ensemble framework, IEEE Trans. Affect. Comput. (2019) 1.

[6]

N. Gupta, M. Gilbert, G. Di Fabbrizio, Emotion Detection in Email Customer Care, in: Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA, 2010, pp. 10–16.

[7]

L. Dini, a. Bittar, Emotion Analysis on Twitter: The Hidden Challenge, in: Proceedings of the Tenth international Conference on Language Resources and Evaluation, LREC’16, Portorož, Slovenia, 2016, pp. 3953–3958.

[8]

Akhtar M.S., Ekbal A., Cambria E., How intense are you? Predicting intensities of emotions and sentiments using stacked ensemble [application notes], IEEE Comput. intell. Mag. 15 (1) (2020) 64–75.

Digital Library

[9]

H. Khanpour, C. Caragea, Fine-Grained Emotion Detection in Health-Related online Posts, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018, pp. 1160–1166.

[10]

D. Hazarika, S. Poria, A. Zadeh, E. Cambria, L.-P. Morency, R. Zimmermann, Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, 2018, pp. 2122–2132.

[11]

Lin Z., Madotto a., Shin J., Xu P., Fung P., Moel: Mixture of empathetic listeners, 2019, arXiv preprint arXiv:1908.07687.

[12]

Shin J., Xu P., Madotto a., Fung P., Generating empathetic responses by looking ahead the user’s sentiment, in: ICASSP 2020-2020 IEEE international Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2020, pp. 7989–7993.

[13]

Ma Y., Nguyen K.L., Xing F.Z., Cambria E., A survey on empathetic dialogue systems, Inf. Fusion 64 (2020) 50–70.

[14]

Young t., Pandelea V., Poria S., Cambria E., Dialogue systems with audio context, Neurocomputing 388 (2020) 102–109.

Digital Library

[15]

S. Sukhbaatar, A. Sszlam, J. Weston, R. Fergus, End-to-End Memory Networks, in: Proceedings of the Advances in Neural information Processing Systems, 2015, pp. 2440–2448.

[16]

S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, R. Mihalcea, MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019, pp. 527–536.

[17]

S.Y.M. Lee, Y. Chen, C.-R. Huang, A Text-driven Rule-based System for Emotion Cause Detection, in: Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA, 2010, pp. 45–53.

[18]

L. Gui, D. Wu, R. Xu, Q. Lu, Y. Zhou, Event-Driven Emotion Cause Extraction with Corpus Construction, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 2016, pp. 1639–1649.

[19]

Poria S., Cambria E., Hazarika D., Vij P., A deeper look into sarcastic tweets using deep convolutional neural networks, 2016, arXiv preprint arXiv:1610.08815.

[20]

Mencattini A., Martinelli E., Costantini G., todisco M., Basile B., Bozzali M., Di Natale C., Speech emotion recognition using amplitude modulation parameters and a combined feature selection procedure, Knowl.-Based Syst. 63 (2014) 68–81,. URL https://www.sciencedirect.com/science/article/pii/S0950705114001087.

Digital Library

[21]

Zhang L., Mistry K., Neoh S.C., Lim C.P., Intelligent facial emotion recognition using moth-firefly optimization, Knowl.-Based Syst. 111 (2016) 248–267,. URL https://www.sciencedirect.com/science/article/pii/S0950705116302799.

Digital Library

[22]

Cui H., Liu A., Zhang X., Chen X., Wang K., Chen X., EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network, Knowl.-Based Syst. 205 (2020),. URL https://www.sciencedirect.com/science/article/pii/S0950705120304433.

[23]

D. Hazarika, S. Poria, R. Mihalcea, E. Cambria, R. Zimmermann, ICon: Interactive Conversational Memory Network for Multimodal Emotion Detection, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018, pp. 2594–2604.

[24]

Hazarika D., Poria S., Zimmermann R., Mihalcea R., Emotion recognition in conversations with transfer learning from generative conversation modeling, 2019, arXiv preprint arXiv:1910.04980.

[25]

P. Zhong, D. Wang, C. Miao, Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and The 9th international Joint Conference on Natural Language Processing, EMNLP-IJCNLP, Hong Kong, China, 2019, pp. 165–176.

[26]

D. Ghosal, N. Majumder, S. Poria, N. Chhaya, A. Gelbukh, DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and The 9th international Joint Conference on Natural Language Processing, EMNLP-IJCNLP, Hong Kong, China, 2019, pp. 154–164.

[27]

Li W., Shao W., Ji S., Cambria E., BiERU: bidirectional emotional recurrent unit for conversational sentiment analysis, 2020, arXiv preprint arXiv:2006.00492.

[28]

Zhang Y., Li Q., Song D., Zhang P., Wang P., Quantum-inspired interactive networks for conversational sentiment analysis, 2019.

[29]

Y. Wang, J. Zhang, J. Ma, S. Wang, J. Xiao, Contextualized emotion recognition in conversation as sequence tagging, in: Proceedings of the 21th Annual Meeting of the Special interest Group on Discourse and Dialogue, 2020, pp. 186–195.

[30]

Lazarus R.S., Folkman S., Stress, Appraisal, and Coping, 1984.

[31]

Krippendorff K., Computing Krippendorff’s alpha-reliability, 2011.

[32]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, in: Advances in Neural information Processing Systems, 2017, pp. 5998–6008.

[33]

D. Ma, S. Li, X. Zhang, H. Wang, interactive Attention Networks for Aspect-Level Sentiment Classification, in: Proceedings of the Twenty-Sixth international Joint Conference on Artificial intelligence, IJCAI 2017, Melbourne, Australia, 2017, pp. 4068–4074.

[34]

Jiao W., Lyu M.R., King I., Real-time emotion recognition via attention gated hierarchical memory network, 2019, arXiv preprint arXiv:1911.09075.

[35]

Vinyals O., fortunato M., Jaitly N., Pointer networks, 2015, arXiv preprint arXiv:1506.03134.

[36]

Devlin J., Chang M.-W., Lee K., toutanova K., Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv preprint arXiv:1810.04805.

[37]

Busso C., Bulut M., Lee C.-C., Kazemzadeh A., Mower E., Kim S., Chang J.N., Lee S., Narayanan S.S., IEMOCAP: Interactive emotional dyadic motion capture database, Language Resources and Evaluation 42 (4) (2008) 335–359.

Cited By

Kim YLee DLiang PAlghowinem SBreazeal CPark H(2023)HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory TransformerProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614122(314-325)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614122

Index Terms

Discovering emotion and reasoning its flip in multi-party conversations using masked memory network and transformer
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
  2. Machine learning
    1. Learning paradigms

Index terms have been assigned to the content through auto-classification.

Recommendations

Multimodal Turn Analysis and Prediction for Multi-party Conversations
ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction

This paper presents a computational study to analyze and predict turns (i.e., turn-taking and turn-keeping) in multiparty conversations. Specifically, we use a high-fidelity hybrid data acquisition system to capture a large-scale set of multi-modal ...
Telepresence robot that exaggerates non-verbal cues for taking turns in multi-party teleconferences
HAI '14: Proceedings of the second international conference on Human-agent interaction

In this paper, we propose a telepresence robot that exaggerates non-verbal cues for taking turns in multi- party teleconferences. In multi-party teleconferences, it is more difficult that the remote participants to take their turns than face-to-face. ...
Automatic Speech Recognition and Natural Language Understanding for Emotion Detection in Multi-party Conversations
MuCAI ?20: Proceedings of the 1st International Workshop on Multimodal Conversational AI

Conversational emotion and sentiment analysis approaches rely on Natural Language Understanding (NLU) and audio processing components to achieve the goal of detecting emotions and sentiment based on what is being said. While there has been marked ...

Comments

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 240, Issue C

Mar 2022

830 pages

ISSN:0950-7051

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 15 March 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim YLee DLiang PAlghowinem SBreazeal CPark H(2023)HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory TransformerProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614122(314-325)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614122

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents