research-article

Understanding the Performance of AI Algorithms in Text-Based Emotion Detection for Conversational Agents

Authors:

Sheetal D. Kusal,

Shruti G. Patil,

Jyoti Choudrie,

Ketan V. KotechaAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 23, Issue 8

Article No.: 121, Pages 1 - 26

https://doi.org/10.1145/3643133

Published: 08 August 2024 Publication History

Abstract

Current industry trends demand automation in every aspect, where machines could replace humans. Recent advancements in conversational agents have grabbed a lot of attention from industries, markets, and businesses. Building conversational agents that exhibit human communication characteristics is a need in today's marketplace. Thus, by accumulating emotions, we can build emotionally aware conversational agents. Emotion detection in text-based dialogues has turned into a pivotal component of conversational agents, enhancing their ability to understand and respond to users’ emotional states. This article extensively compares various artificial intelligence techniques adapted to text-based emotion detection for conversational agents. The study covers a wide range of methods, from machine learning models to cutting-edge pre-trained models and deep learning models. We evaluate the performance of these techniques on the benchmark unbalanced Topical-Chat and balanced Empathetic Dialogue datasets. This article offers an overview of the practical implications of emotion detection techniques in conversational systems and their impact on user response. The outcomes of this work contribute to the ongoing development of empathetic conversational agents, emphasizing natural human-machine interactions.

References

[1]

R. Bavaresco, Diorgenes Silveira, Eduardo Reis, Jorge Barbosa, Rodrigo Righi, Cristiano Costa, Rodolfo Antunes, Marcio Gomes, Clauter Gatti, Mariangela Vanzin, Saint Clair Junior, Elton Silva, and Carlos Moreira. 2020. Conversational agents in business: A systematic literature review and future research directions. Computer Science Review 36 (2020), 100239. DOI:

[2]

M. Nuruzzaman and O. K. Hussain. 2018. A survey on chatbot implementation in customer service industry through deep neural networks. In Proceedings of the 2018 IEEE 15th International Conference on e-Business Engineering (ICEBE ’18). IEEE, 54–61. DOI:

[3]

S. Hobert and R. Meyer von Wolff. 2019. Say hello to your new automated tutor—A structured literature review on pedagogical conversational agents. In Proceedings of the 14th International Conference on Wirtschaftsinformatik. 301–314.

[4]

J. Fraser, I. Papaioannou, and O. Lemon. 2018. Spoken conversational AI in video games: Emotional dialogue management increases user engagement. In Proceedings of the 18th International Conference on Intelligent Virtual Agents (IVA ’18). 179–184. DOI:

Digital Library

[5]

J. L. Z. Montenegro, C. A. da Costa, and R. da Rosa Righi. 2019. Survey of conversational agents in health. Expert Systems with Applications 129 (2019), 56–67. DOI:

Digital Library

[6]

R. W. Picard. 2000. Affective Computing. MIT Press, Cambridge, MA.

Digital Library

[7]

H. Prendinger and M. Ishizuka. 2005. The empathic companion: A character-based interface that addresses users’ affective states. Applied Artificial Intelligence 19, 3-4 (March 2005), 267–285. DOI:

[8]

J. Tao. 2004. Context based emotion detection from text input. In Proceedings of the 8th International Conference on Spoken Language Processing (INTERSPEECH ’04).

[9]

O. Udochukwu and Y. He. 2015. A rule-based approach to implicit emotion detection in text. In Natural Language Processing and Information Systems. Lecture Notes in Computer Science, Vol. 9013. Springer, 197–203.

[10]

IEEE. 2018. 2018 IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE). IEEE.

[11]

M. E. Basiri, S. Nemati, M. Abdar, E. Cambria, and U. R. Acharya. 2021. ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis. Future Generation Computer Systems 115 (Feb. 2021), 279–294. DOI:

Digital Library

[12]

F. A. Acheampong, C. Wenyu, and H. Nunoo-Mensah. 2020. Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports 2, 7 (July 2020), e12189. DOI:

[13]

M. Mnasri. 2019. Recent advances in conversational NLP: Towards the standardization of Chatbot building. arXiv:1903.09025 (2019). http://arxiv.org/abs/1903.09025

[14]

K. M. Colby, S. Weber, and F. D. Hilf. 1971. Artificial paranoia. Artificial Intelligence 2, 1 (1971), 1–25.

Digital Library

[15]

M. Skowron. 2010. Affect listeners: Acquisition of affective states by means of conversational systems. In Development of Multimodal Interfaces: Active Listening and Synchrony. Lecture Notes in Computer Science, Vol. 5967. Springer, 169–181.

Digital Library

[16]

N. v Kolekar, P. Gauri Rao, S. Dey, M. Mane, V. Jadhav, and S. Patil. 2016. Sentiment analysis and classification using lexicon-based approach and addressing polarity shift problem. Journal of Theoretical and Applied Information Technology 90, 1 (2016), 118–125.

[17]

A. Basile, M. Franco-Salvador, N. Pawar, S. Sanjaštajner, M. C. Rios, and Y. Benajiba. 2019. SymantoResearch at SemEval-2019 Task 3: Combined neural models for emotion classification in human-chatbot conversations. In Proceedings of the 13th International Workshop on Semantic Evaluation. 330–334.

[18]

A. Adikari, D. de Silva, D. Alahakoon, and X. Yu. 2019. A cognitive model for emotion awareness in industrial chatbots. In Proceedings of the 2019 IEEE 17th International Conference on Industrial Informatics (INDIN ’19).

[19]

M. Feidakis, P. Kasnesis, E. Giatraki, C. Giannousis, C. Patrikakis, and P. Monachelis. 2019. Building pedagogical conversational agents, affectively correct. In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU ’19). 100–107. DOI:

[20]

Y. Xie and P. Pu. 2021. Empathetic dialog generation with fine-grained intents. In Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL ’21). 133–147. http://arxiv.org/abs/2105.06829

[21]

S. Patil, V. M. Mudaliar, P. Kamat, and S. Gite. 2020. LSTM based ensemble network to enhance the learning of long-term dependencies in chatbot. International Journal for Simulation and Multidisciplinary Design Optimization 11 (2020), 1–17. DOI:

[22]

C. Huang, O. R. Zaıane, A. Trabelsi, and N. Dziri. 2018. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 49–54. http://www.cs.ualberta.ca/

[23]

H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence Conference, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (AAAI/IAAI/EAAI ’18). 730–738. http://arxiv.org/abs/1704.01074

[24]

Y. Sun and Y. Zhang. 2018. Conversational recommender system. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’18). ACM, 235–244. DOI:

Digital Library

[25]

S. Ghosh, D. Varshney, A. Ekbal, and P. Bhattacharyya. 2021. Context and knowledge enriched transformer framework for emotion recognition in conversations. In Proceedings of the International Joint Conference on Neural Networks. IEEE. DOI:

[26]

A. Ezen-Can and E. F. Can. 2018. RNN for Affects at SemEval-2018 Task 1: Formulating affect identification as a binary classification problem. In Proceedings of the 12th International Workshop on Semantic Evaluation. 162–166.

[27]

Y. Shou, T. Meng, W. Ai, S. Yang, and K. Li. 2022. Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis. Neurocomputing 501 (Aug. 2022), 629–639. DOI:

Digital Library

[28]

G. Tu, J. Wen, C. Liu, D. Jiang, and E. Cambria. 2022. Context- and sentiment-aware networks for emotion recognition in conversation. IEEE Transactions on Artificial Intelligence 3, 5 (Oct. 2022), 699–708. DOI:

[29]

N. T. Thomas. 2016. An e-business chatbot using AIML and LSA. In Proceedings of the 2016 International Conference on Advances in Computing, Communications, and Informatics (ICACCI ’16). IEEE, 2740–2742. DOI:

[30]

Y. C. Chang and Y. C. Hsing. 2021. Emotion-infused deep neural network for emotionally resonant conversation. Applied Soft Computing 113 (Dec. 2021), 107861. DOI:

Digital Library

[31]

L. De Bruyne, O. De Clercq, and V. Hoste. 2018. LT3 at SemEval-2018 Task 1: A classifier chain to detect emotions in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation. 123–127.

[32]

M. Suhasini and B. Srinivasu. 2020. Emotion detection framework for Twitter data using supervised classifers. In Data Engineering and Communication Technology. Advances in Intelligent Systems and Computing, Vol. 1079. Springer, 565–576.

[33]

Merav Allouch, Amos Azaria, Rina Azoulay, Ester Ben-Izchak, and Moti Zwilling. 2018. Automatic detection of insulting sentences in conversation. In Proceedings of the 2018 IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE ’18).

[34]

N. Nguyen, T.-H. Nguyen, Y.-N. Nguyen, D. Doan, M. Nguyen, and V.-H. Nguyen. 2023. Machine learning-based model for customer emotion detection in hotel booking services. Journal of Hospitality and Tourism Insights. Published July 17, 2023.

[35]

A. Basile, M. Franco-Salvador, N. Pawar, S. Sanjaštajner, M. C. Rios, and Y. Benajiba. 2019. SymantoResearch at SemEval-2019 Task 3: Combined neural models for emotion classification in human-chatbot conversations. In Proceedings of the 13th International Workshop on Semantic Evaluation.

[36]

K. Shrivastava, S. Kumar, and D. K. Jain. 2019. An effective approach for emotion detection in multimedia text data using sequence based convolutional neural network. Multimedia Tools and Applications 78, 20 (Oct. 2019), 29607–29639. DOI:

Digital Library

[37]

L. Claudio Diogo Reis, F. Cristina Bernardini, S. Bacellar Leal Ferreira, and C. Cappelli. 2021. ICT governance in Brazilian smart cities: An integrative approach in the context of digital transformation. In Proceedings of the 22nd Annual International Conference on Digital Government Research. 302–316. DOI:

Digital Library

[38]

J. Xiao. 2019. Figure Eight at SemEval-2019 Task 3: Ensemble of transfer learning methods for contextual emotion detection. In Proceedings of the 13th International Workshop on Semantic Evaluation. https://github.com/fastai/fastai

[39]

José Ángel González, Lluís-F. Hurtado, and Ferran Pla. 2020. Transformer based contextualization of pre-trained word embeddings for irony detection in Twitter. Information Processing & Management 57, 4 (2020), 102262.

[40]

D. Cortiz. 2021. Exploring Transformers in emotion recognition: A comparison of BERT, DistilBERT, RoBERTa, XLNet and ELECTRA. arXiv:2104.02041 (2021).

[41]

S. Kusal, S. Patil, K. Kotecha, R. Aluvalu, and V. Varadarajan. 2021. AI based emotion detection for textual big data: Techniques and contribution. Big Data and Cognitive Computing 5, 3 (2021), 43. DOI:

[42]

S. Kusal, S. Patil, J. Choudrie, K. Kotecha, S. Mishra, and A. Abraham. 2022. AI-based conversational agents: A scoping review from technologies to future directions. IEEE Access 10 (2022), 92337–92356. DOI:

[43]

K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwarta, A. Venkatesh, R. Gabriel, and D. Hakkani-Tur. 2019. Topical-Chat: Towards knowledge-grounded open-domain conversations. In Proceedings of the 8th International Conference on Spoken Language Processing (INTERSPEECH ’19).

[44]

H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv:1811.00207 (2018). http://arxiv.org/abs/1811.00207

[45]

N. Anantrasirichai and D. Bull. 2022. Artificial intelligence in the creative industries: A review. Artificial Intelligence Review 55, 1 (Jan. 2022), 589–656. DOI:

Digital Library

[46]

Lovejit Singh, Sarbjeet Singh, and Naveen Aggarwal. 2018. Two-stage text feature selection method for human emotion recognition. In Proceedings of the 2nd International Conference on Communication, Computing and Networking. Lecture Notes in Networks and Systems, Vol. 46. Springer, 531–538.

[47]

S. Aman and S. Szpakowicz. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue. Lecture Notes in Computer Science, Vol. 4629. Springer, 196–205. DOI:

[48]

C. Baziotis, N. Pelekis, and C. Doulkeridis. 2017. DataStories at SemEval-2017 Task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval ’17). 747–754.

[49]

A. Balahur, J. M. Hermida, and A. Montoyo. 2012. Building and exploiting EmotiNet, a knowledge base for emotion detection based on the appraisal theory model. IEEE Transactions on Affective Computing 3, 1 (Jan. 2012), 88–101. DOI:

Digital Library

[50]

F. Huang, X. Li, C. Yuan, S. Zhang, J. Zhang, and S. Qiao. 2021. Attention-emotion-enhanced convolutional LSTM for sentiment analysis. IEEE Transactions on Neural Networks and Learning Systems 33, 9 (2021), 4332–4345.

[51]

Z. Ahmad, R. Jindal, A. Ekbal, and P. Bhattachharyya. 2020. Borrow from rich cousin: Transfer learning for emotion detection using cross lingual embedding. Expert Systems with Applications 139 (2020), 112851.

Digital Library

[52]

M. G. Salido Ortega, L.-F. Rodriguez, and J. Octavio Gutierrez-Garcia. 2020. Towards emotion recognition from contextual information using machine learning. Journal of Ambient Intelligence and Humanized Computing 11 (2020), 3187–3207.

[53]

Y. Lai, L. Zhang, D. Han, R. Zhou, and G. Wang. 2020. Fine-grained emotion classification of Chinese microblogs based on graph convolution networks. World Wide Web 23 (2020), 2771–2787.

Digital Library

[54]

F. A. Acheampong, H. Nunoo-Mensah, and W. Chen. 2021. Transformer models for text-based emotion detection: A review of BERT-based approaches. Artificial Intelligence Review 54 (2021), 5789–5829.

Digital Library

[55]

C. Huang, A. Trabelsi, and O. R. Zaïane. 2019. ANA at SemEval-2019 Task 3: Contextual emotion detection in conversations through hierarchical LSTMs and BERT. In Proceedings of the 13th International Workshop on Semantic Evaluation. 49–53. http://arxiv.org/abs/1904.00132

[56]

Y.-H. Huang, S.-R. Lee, M.-Y. Ma, Y.-H. Chen, Y.-W. Yu, and Y.-S. Chen. 2019. EmotionX-IDEA: Emotion BERT—An affectional model for conversation. arXiv:1908.06264 (2019). http://nlp.mathcs.emory.edu

[57]

S. M. Mohammad. 2016. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. Emotion Measurement 2016 (2016), 201–237.

[58]

S. Sun, C. Luo, and J. Chen. 2017. A review of natural language processing techniques for opinion mining systems. Information Fusion 36 (2017), 10–25.

Digital Library

[59]

A. Kumar and G. Garg. 2019. Empirical study of shallow and deep learning models for sarcasm detection using context in benchmark datasets. Journal of Ambient Intelligence and Humanized Computing 14 (2019), 5327–5342.

[60]

S. Zhang, X. Zhang, J. Chan, and P. Rosso. 2019. Irony detection via sentiment-based transfer learning. Information Processing & Management 56, 5 (2019), 1633–1644.

Digital Library

Cited By

Kusal SPanchal PPatil S(2024)Pre-Trained Networks and Feature Fusion for Enhanced Multimodal Sentiment Analysis2024 MIT Art, Design and Technology School of Computing International Conference (MITADTSoCiCon)10.1109/MITADTSoCiCon60330.2024.10574938(1-7)Online publication date: 25-Apr-2024
https://doi.org/10.1109/MITADTSoCiCon60330.2024.10574938

Index Terms

Understanding the Performance of AI Algorithms in Text-Based Emotion Detection for Conversational Agents
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Information extraction

Recommendations

Enhancing Conversational Agents with Empathic Abilities
IVA '21: Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents

Conversational agents are getting increasingly popular and find applications in health and customer services. Conversations in these fields are often emotionally charged. It is, therefore, necessary to handle the conversation with some degree of empathy ...
Natural Language, Mixed-initiative Personal Assistant Agents
IMCOM '18: Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication

The increasing popularity and use of personal voice assistant technologies, such as Siri and Google Now, is driving and expanding progress toward the long-term and lofty goal of using artificial intelligence to build human-computer dialog systems ...
Young Children's Reading and Learning with Conversational Agents
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Young children increasingly interact with voice-driven interfaces, such as conversational agents (CAs). The social nature of CAs makes them good learning partners for children. We have designed a storytelling CA to engage children in book reading ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 23, Issue 8

August 2024

343 pages

EISSN:2375-4702

DOI:10.1145/3613611

Editor:
Imed Zitouni
Google, USA
,
Guest Editors:
Deepak Kumar Jain,
Thierry Boumans,
Stefano Berretti

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 August 2024

Online AM: 31 January 2024

Accepted: 03 January 2024

Revised: 24 October 2023

Received: 19 January 2023

Published in TALLIP Volume 23, Issue 8

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
416
Total Downloads

Downloads (Last 12 months)416
Downloads (Last 6 weeks)65

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kusal SPanchal PPatil S(2024)Pre-Trained Networks and Feature Fusion for Enhanced Multimodal Sentiment Analysis2024 MIT Art, Design and Technology School of Computing International Conference (MITADTSoCiCon)10.1109/MITADTSoCiCon60330.2024.10574938(1-7)Online publication date: 25-Apr-2024
https://doi.org/10.1109/MITADTSoCiCon60330.2024.10574938

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents