research-article

Open access

Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot Interactions

Authors:

Anouck Braggaar,

Marjolijn Antheunis,

Reinout WiersAuthors Info & Claims

CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces

Article No.: 9, Pages 1 - 14

https://doi.org/10.1145/3640794.3665536

Published: 08 July 2024 Publication History

All formats PDF

Abstract

Monitoring and maintaining user engagement in human-chatbot interactions is challenging. Researchers often use cues observed in the interactions as indicators to infer engagement. However, evaluation of these cues is lacking. In this study, we collected an inventory of potential textual engagements cues from the literature, including linguistic features, utterance features, and interaction features. These cues were subsequently used to annotate a dataset of 291 user-chatbot interactions, and we examined which of these cues predicted self-reported user engagement. Our results show that engagement can indeed be recognized at the level of individual utterances. Notably, words indicating cognitive thinking processes and motivational utterances were strong indicators of engagement. An overall negative tone could also predict engagement, highlighting the importance of nuanced interpretation and contextual awareness of user utterances. Our findings demonstrated initial feasibility of recognizing utterance-level cues and using them to infer user engagement, although further validation is needed across different content-domains.

References

[1]

Alaa Ali Abd-Alrazaq, Asma Rababeh, Mohannad Alajlani, Bridgette M. Bewick, and Mowafa Househ. 2020. Effectiveness and safety of using chatbots to improve mental health: systematic review and meta-analysis. Journal of medical Internet research 22, 7 (2020), e16021.

[2]

Abhishek Aggarwal, Cheuk Chi Tam, Dezhi Wu, Xiaoming Li, and Shan Qiao. 2023. Artificial Intelligence–Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. Journal of Medical Internet Research 25, (2023), e40789.

[3]

Tarek Ait Baha, Mohamed El Hajji, Youssef Es-Saady, and Hammou Fadili. 2023. The power of personalization: A systematic review of personality-adaptive chatbots. SN Computer Science 4, 5 (2023), 661.

Digital Library

[4]

Deepali Aneja, Daniel McDuff, and Mary Czerwinski. 2020. Conversational Error Analysis in Human-Agent Interaction. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA ’20), October 19, 2020. Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3383652.3423901

Digital Library

[5]

Sandeep Avula, Gordon Chadwick, Jaime Arguello, and Robert Capra. 2018. SearchBots: User Engagement with ChatBots during Collaborative Search. In Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR ’18, 2018. ACM Press, New Brunswick, NJ, USA, 52–61. https://doi.org/10.1145/3176349.3176380

Digital Library

[6]

Erkan Basar, Divyaa Balaji, Linwei He, Iris Hendrickx, Emiel Krahmer, Gert-Jan de Bruijn, and Tibor Bosse. 2023. HyLECA: A Framework for Developing Hybrid Long-term Engaging Controlled Conversational Agents. In Proceedings of the 5th International Conference on Conversational User Interfaces, 2023. 1–5.

Digital Library

[7]

Timothy Bickmore, Laura Pfeifer, and Daniel Schulman. 2011. Relational agents improve engagement and learning in science museum visitors. In Intelligent Virtual Agents: 10th International Conference, IVA 2011, Reykjavik, Iceland, September 15-17, 2011. Proceedings 11, 2011. Springer, 55–67.

[8]

Timothy Bickmore, Daniel Schulman, and Langxuan Yin. 2010. Maintaining Engagement in Long-Term Interventions with Relational Agents. Applied Artificial Intelligence 24, 6 (July 2010), 648–666. https://doi.org/10.1080/08839514.2010.492259

Digital Library

[9]

Stacey Birkett, Adam Galpin, Simon Cassidy, Lynne Marrow, and Sarah Norgate. 2011. How revealing are eye-movements for understanding web engagement in young children. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, May 07, 2011. ACM, Vancouver BC Canada, 2251–2256. https://doi.org/10.1145/1979742.1979900

Digital Library

[10]

Ned Block. 1981. Psychologism and behaviorism. The Philosophical Review 90, 1 (1981), 5–43.

[11]

Ryan L. Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W. Pennebaker. 2022. The development and psychometric properties of LIWC-22. Austin, TX: University of Texas at Austin (2022), 1–47.

[12]

Egon Brunswik. 1956. Perception and the representative design of psychological experiments. University of California Press, Berkley.

[13]

Jonathan Carlton, Andy Brown, Caroline Jay, and John Keane. 2021. Using Interaction Data to Predict Engagement with Interactive Media. In Proceedings of the 29th ACM International Conference on Multimedia (MM ’21), October 17, 2021. Association for Computing Machinery, New York, NY, USA, 1258–1266. https://doi.org/10.1145/3474085.3475631

Digital Library

[14]

Jacky Casas, Marc-Olivier Tricot, Omar Abou Khaled, Elena Mugellini, and Philippe Cudré-Mauroux. 2020. Trends & Methods in Chatbot Evaluation. In Companion Publication of the 2020 International Conference on Multimodal Interaction, October 25, 2020. ACM, Virtual Event Netherlands, 280–286. https://doi.org/10.1145/3395035.3425319

Digital Library

[15]

Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How should my chatbot interact? A survey on social characteristics in human–chatbot interaction design. International Journal of Human–Computer Interaction 37, 8 (2021), 729–758.

[16]

Jason Ingyu Choi, Ali Ahmadvand, and Eugene Agichtein. 2019. Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, November 03, 2019. ACM, Beijing China, 1281–1290. https://doi.org/10.1145/3357384.3358047

Digital Library

[17]

Martin Colbert and Angela Boodoo. 2011. Does’ Letting Go of the Words’ increase engagement: A traffic study. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. 655–667.

[18]

M. Csikszentmihalyi. 1990. Flow. The Psychology of Optimal Experience. New York (HarperPerennial) 1990. (1990).

[19]

Sara Dalzel-Job, Craig Nicol, and Jon Oberlander. 2008. Comparing behavioural and self-report measures of engagement with an embodied conversational agent: a first report on eye tracking in Second Life. In Proceedings of the 2008 symposium on Eye tracking research & applications (ETRA ’08), March 26, 2008. Association for Computing Machinery, New York, NY, USA, 83–85. https://doi.org/10.1145/1344471.1344493

Digital Library

[20]

Benner Dennis, Sofia Schöbel, Andreas Janson, and Jan Marco Leimeister. 2024. Engaging Minds–How Gamified Chatbots can Support and Motivate Learners in Digital Education. In Hawaii International Conference on System Sciences (HICSS), 2024. .

[21]

Kevin Doherty and Gavin Doherty. 2019. Engagement in HCI: Conception, Theory and Measurement. ACM Comput. Surv. 51, 5 (September 2019), 1–39. https://doi.org/10.1145/3234149

Digital Library

[22]

Michelle Drouin, Susan Sprecher, Robert Nicola, and Taylor Perkins. 2022. Is chatting with a sophisticated chatbot as good as chatting online or FTF with a stranger? Computers in Human Behavior 128, (March 2022), 107100. https://doi.org/10.1016/j.chb.2021.107100

Digital Library

[23]

Asbjørn Følstad, Marita Skjuve, and Petter Bae Brandtzaeg. 2019. Different chatbots for different purposes: towards a typology of chatbots to understand interaction design. In Internet Science: INSCI 2018 International Workshops, St. Petersburg, Russia, October 24–26, 2018, Revised Selected Papers 5, 2019. Springer, 145–156.

[24]

Fotos Frangoudes, Marios Hadjiaros, Eirini C. Schiza, Maria Matsangidou, Olia Tsivitanidou, and Kleanthis Neokleous. 2021. An overview of the use of chatbots in medical and healthcare education. In International Conference on Human-Computer Interaction, 2021. Springer, 170–184.

Digital Library

[25]

Hannah Gaffney, Warren Mansell, and Sara Tai. 2019. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR mental health 6, 10 (2019), e14166.

[26]

Kai Guo, Yuchun Zhong, Danling Li, and Samuel Kai Wah Chu. 2023. Investigating students’ engagement in chatbot-supported classroom debates. Interactive Learning Environments (2023), 1–17.

[27]

Linwei He, Divyaa Balaji, Reinout W. Wiers, Marjolijn L. Antheunis, and Emiel Krahmer. 2023. Effectiveness and acceptability of conversational agents for smoking cessation: A systematic review and meta-analysis. Nicotine and Tobacco Research 25, 7 (2023), 1241–1250.

[28]

He, Linwei, Erkan Basar, Emiel Krahmer, Reinout W. Wiers, and Marjolijn L. Antheunis. Effectiveness and user experience of a smoking cessation chatbot: A mixed-methods study comparing motivational interviewing and confrontational counselling.

[29]

Linwei He, Erkan Basar, Reinout W. Wiers, Marjolijn L. Antheunis, and Emiel Krahmer. 2022. Can chatbots help to motivate smoking cessation? A study on the effectiveness of motivational interviewing on engagement and therapeutic alliance. BMC Public Health 22, 1 (2022), 726.

[30]

Khe Foon Hew, Weijiao Huang, Jiahui Du, and Chengyuan Jia. 2021. Using chatbots in flipped learning online sessions: perceived usefulness and ease of use. In Blended Learning: Re-thinking and Re-defining the Learning Process. 14th International Conference, ICBL 2021, Nagoya, Japan, August 10–13, 2021, Proceedings 14, 2021. Springer, 164–175.

[31]

Jennifer Hill, W. Randolph Ford, and Ingrid G. Farreras. 2015. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in Human Behavior 49, (August 2015), 245–250. https://doi.org/10.1016/j.chb.2015.02.026

Digital Library

[32]

Robert Ho. 2006. Handbook of univariate and multivariate data analysis and interpretation with SPSS. CRC press.

[33]

Ryo Ishii and Yukiko I. Nakano. 2008. Estimating user's conversational engagement based on gaze behaviors. In Intelligent Virtual Agents: 8th International Conference, IVA 2008, Tokyo, Japan, September 1-3, 2008. Proceedings 8, 2008. Springer, 200–207.

Digital Library

[34]

Carolina Islas Sedano, Verona Leendertz, Mikko Vinni, Erkki Sutinen, and Suria Ellis. 2013. Hypercontextualized learning games: Fantasy, motivation, and engagement in reality. Simulation & Gaming 44, 6 (2013), 821–845.

Digital Library

[35]

Klaus Krippendorff. 2009. Testing the reliability of content analysis data. The content analysis reader (2009), 350–357.

[36]

Alexander J. Kull, Marisabel Romero, and Lisa Monahan. 2021. How may I help you? Driving brand engagement through the warmth of an initial chatbot message. Journal of Business Research 135, (October 2021), 840–850. https://doi.org/10.1016/j.jbusres.2021.03.005

[37]

Effie Lai-Chong Law, Asbjørn Følstad, and Nena Van As. 2022. Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service. In Nordic Human-Computer Interaction Conference, 2022. 1–13.

[38]

Chi-Hsun Li, Ken Chen, and Yung-Ju Chang. 2019. When There is No Progress with a Task-Oriented Chatbot: A Conversation Analysis. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, October 2019. ACM, Taipei Taiwan, 1–6. https://doi.org/10.1145/3338286.3344407

Digital Library

[39]

Weixin Liang, Kai-Hui Liang, and Zhou Yu. 2021. HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations. Retrieved September 21, 2023 from http://arxiv.org/abs/2106.00162

[40]

Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016).

[41]

Yukiko I. Nakano and Ryo Ishii. 2010. Estimating user's engagement from eye-gaze behaviors in human-agent conversations. In Proceedings of the 15th international conference on Intelligent user interfaces, 2010. 139–148.

[42]

Johanna Nurmi, Keegan Knittle, Todor Ginchev, Fida Khattak, Christopher Helf, Patrick Zwickl, Carmina Castellano-Tejedor, Pilar Lusilla-Palacios, Jose Costa-Requena, and Niklas Ravaja. 2020. Engaging users in the behavior change process with digitalized motivational interviewing and gamification: development and feasibility testing of the precious app. JMIR mHealth and uHealth 8, 1 (2020), e12884.

[43]

Heather L. O'Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112, (2018), 28–39.

[44]

Heather L. O'Brien and Elaine G. Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American society for Information Science and Technology 59, 6 (2008), 938–955.

Digital Library

[45]

Robert M. O'brien. 2007. A caution regarding rules of thumb for variance inflation factors. Quality & quantity 41, (2007), 673–690.

[46]

Catharine Oertel, Ginevra Castellano, Mohamed Chetouani, Jauwairia Nasir, Mohammad Obaid, Catherine Pelachaud, and Christopher Peters. 2020. Engagement in human-agent interaction: An overview. Frontiers in Robotics and AI 7, (2020), 92.

[47]

Olga Perski, Ann Blandford, Robert West, and Susan Michie. 2017. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Translational behavioral medicine 7, 2 (2017), 254–267.

[48]

Minjin Rheu, Yue Dai, Jingbo Meng, and Wei Peng. 2024. When a Chatbot Disappoints You: Expectancy Violation in Human-Chatbot Interaction in a Social Support Context. Communication Research (2024), 00936502231221669.

[49]

Hanan Salam and Mohamed Chetouani. 2015. Engagement detection based on mutli-party cues for human robot interaction. In 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), 2015. IEEE, 341–347.

Digital Library

[50]

Henrik Schoenau-Fog. 2011. Hooked!–evaluating engagement as continuation desire in interactive narratives. In Interactive Storytelling: Fourth International Conference on Interactive Digital Storytelling, ICIDS 2011, Vancouver, Canada, November 28–1 December, 2011. Proceedings 4, 2011. Springer, 219–230.

Digital Library

[51]

Huma Shah, Kevin Warwick, Jordi Vallverdú, and Defeng Wu. 2016. Can machines talk? Comparison of Eliza with modern dialogue systems. Computers in Human Behavior 58, (May 2016), 278–295. https://doi.org/10.1016/j.chb.2016.01.004

Digital Library

[52]

Annika Silvervarg and Arne Jonsson. Subjective and Objective Evaluation of Conversational Agents in Learning Environments for Young Teenagers.

[53]

Marita Skjuve, Asbjørn Følstad, and Petter Bae Brandtzaeg. 2023. The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users. In Proceedings of the 5th International Conference on Conversational User Interfaces, July 19, 2023. ACM, Eindhoven Netherlands, 1–10. https://doi.org/10.1145/3571884.3597144

Digital Library

[54]

Sinarwati Mohamad Suhaili, Naomie Salim, and Mohamad Nazim Jambli. 2021. Service chatbots: A systematic review. Expert Systems with Applications 184, (2021), 115461.

[55]

S. Shyam Sundar, Saraswathi Bellur, Jeeyun Oh, Qian Xu, and Haiyan Jia. 2014. User experience of on-screen interaction techniques: An experimental investigation of clicking, sliding, zooming, hovering, dragging, and flipping. Human–Computer Interaction 29, 2 (2014), 109–152.

[56]

Yla R. Tausczik and James W. Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29, 1 (2010), 24–54.

[57]

Ha Trinh, Ameneh Shamekhi, Everlyne Kimani, and Timothy W. Bickmore. 2018. Predicting User Engagement in Longitudinal Interventions with Virtual Agents. In Proceedings of the 18th International Conference on Intelligent Virtual Agents, November 05, 2018. ACM, Sydney NSW Australia, 9–16. https://doi.org/10.1145/3267851.3267909

Digital Library

[58]

B. Troy Frensley, Marc J. Stern, and Robert B. Powell. 2020. Does student enthusiasm equal learning? The mismatch between observed and self-reported student engagement and environmental literacy outcomes in a residential setting. The Journal of Environmental Education 51, 6 (November 2020), 449–461. https://doi.org/10.1080/00958964.2020.1727404

[59]

Alexandria Katarina Vail, Joseph F. Grafsgaard, Joseph B. Wiggins, James C. Lester, and Kristy Elizabeth Boyer. 2014. Predicting Learning and Engagement in Tutorial Dialogue: A Personality-Based Model. In Proceedings of the 16th International Conference on Multimodal Interaction, November 12, 2014. ACM, Istanbul Turkey, 255–262. https://doi.org/10.1145/2663204.2663276

Digital Library

[60]

Niels Van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017. The experience sampling method on mobile devices. ACM Computing Surveys (CSUR) 50, 6 (2017), 1–40.

Digital Library

[61]

Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, Rahul Goel, Shaohua Yang, and Anirudh Raju. 2018. On Evaluating and Comparing Open Domain Dialog Systems. Retrieved September 21, 2023 from http://arxiv.org/abs/1801.03625

[62]

Marsha White and Steve M. Dorman. 2001. Receiving social support online: implications for health education. Health Education Research 16, 6 (December 2001), 693–707. https://doi.org/10.1093/her/16.6.693

[63]

Zhou Yu, Vikram Ramanarayanan, Patrick Lange, and David Suendermann-Oeft. 2019. An Open-Source Dialog System with Real-Time Engagement Tracking for Job Interview Training Applications. In Advanced Social Interaction with Agents: 8th International Workshop on Spoken Dialog Systems, Maxine Eskenazi, Laurence Devillers and Joseph Mariani (eds.). Springer International Publishing, Cham, 199–207. https://doi.org/10.1007/978-3-319-92108-2_21

[64]

Ran Zhao, Oscar J. Romero, and Alex Rudnicky. 2018. SOGO: A Social Intelligent Negotiation Dialogue System. In Proceedings of the 18th International Conference on Intelligent Virtual Agents, November 05, 2018. ACM, Sydney NSW Australia, 239–246. https://doi.org/10.1145/3267851.3267880

Digital Library

[65]

Qi Zhou, Bin Li, Lei Han, and Min Jou. 2023. Talking to a bot or a wall? How chatbots vs. human agents affect anticipated communication quality. Computers in Human Behavior 143, (June 2023), 107674. https://doi.org/10.1016/j.chb.2023.107674

Digital Library

[66]

LWT Chatbots. Retrieved February 29, 2024 from https://lwt.cls.ru.nl/chat/home

Cited By

He LBasar EKrahmer EWiers RAntheunis M(2024)Effectiveness and User Experience of a Smoking Cessation Chatbot: Mixed Methods Study Comparing Motivational Interviewing and Confrontational CounselingJournal of Medical Internet Research10.2196/5313426(e53134)Online publication date: 6-Aug-2024
https://doi.org/10.2196/53134

Index Terms

Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot Interactions
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Inferring User Engagement from Interaction Data
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

This paper presents preliminary results of a study designed to quantify users' engagement levels with interactive media content, through self-reported measures and interaction data. The broad hypothesis of the study is that interaction data can be used ...
Using Interaction Data to Predict Engagement with Interactive Media
MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is ...
Exploring the User Engagement in Interacting with Game Interfaces
IVIC 2013: Third International Visual Informatics Conference on Advances in Visual Informatics - Volume 8237

This research investigates and identifies the factors that influence positive user engagement while interacting with games interfaces. In this paper we provide a finding on our experiences with two case studies; 1) A study on the variation of flow ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces

July 2024

616 pages

ISBN:9798400705113

DOI:10.1145/3640794

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NWO

Conference

CUI '24

Sponsor:

SIGCHI

CUI '24: ACM Conversational User Interfaces 2024

July 8 - 10, 2024

Luxembourg, Luxembourg

Acceptance Rates

Overall Acceptance Rate 34 of 100 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
187
Total Downloads

Downloads (Last 12 months)187
Downloads (Last 6 weeks)93

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

He LBasar EKrahmer EWiers RAntheunis M(2024)Effectiveness and User Experience of a Smoking Cessation Chatbot: Mixed Methods Study Comparing Motivational Interviewing and Confrontational CounselingJournal of Medical Internet Research10.2196/5313426(e53134)Online publication date: 6-Aug-2024
https://doi.org/10.2196/53134

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents