research-article

User Expectations of Conversational Chatbots Based on Online Reviews

Authors:

Ekaterina Svikhnushina,

Alexandru Placinta,

Pearl PuAuthors Info & Claims

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

Pages 1481 - 1491

https://doi.org/10.1145/3461778.3462125

Published: 28 June 2021 Publication History

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

User Expectations of Conversational Chatbots Based on Online Reviews

Pages 1481 - 1491

Abstract
References

Abstract

Open-domain chatbots that can engage in a conversation on any topic received significant attention in the last several years, which opened opportunities for studying user interaction with them. Drawing from reviews of chatbots posted on Google Play, we explore user experience and expectations of these agents in a mixed-method study. Results of statistical analysis reveal which social qualities of chatbots are the most significant for user satisfaction. Further, we employ natural language processing and qualitative methods to identify how users wish their chatbots to evolve in the future. While currently users mostly value the entertaining component of their experience, their expectations call for more human-like behavior of chatbots. The most prominent expectations include chatbots’ abilities to treat and express emotions and be more attentive to the user. Based on these findings, we conclude with design implications, discussing the directions for developing social skills of open-domain chatbots.

References

[1]

Hamza Aldabbas, Abdullah Bajahzar, Meshrif Alruily, Ali Adil Qureshi, Rana M. Amir Latif, and Muhammad Farhan. 2021. Google Play Content Scraping and Knowledge Engineering using Natural Language Processing Techniques with the Analysis of User Reviews. Journal of Intelligent Systems 30, 1 (2021), 192–208. https://doi.org/10.1515/jisys-2019-0197

[2]

Amazon Alexa. 2021. What is the Alexa Skills Kit?https://developer.amazon.com/en-US/docs/alexa/ask-overviews/what-is-the-alexa-skills-kit.html.

[3]

Rana M. Amir Latif, M. Talha Abdullah, Syed Umair Aslam Shah, Muhammad Farhan, Farah Ijaz, and Abdul Karim. 2019. Data Scraping from Google Play Store and Visualization of its Content for Analytics. In 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). 1–8. https://doi.org/10.1109/ICOMET.2019.8673523

[4]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research 3 (2003), 993–1022. https://doi.org/10.1162/jmlr.2003.3.4-5.993

[5]

Asbjørn Brandtzaeg, Petter Baeand Følstad. 2017. Why People Use Chatbots. In Internet Science, Ioannis Kompatsiaris, Jonathan Cave, Anna Satsiou, Georg Carle, Antonella Passani, Efstratios Kontopoulos, Sotiris Diplaris, and Donald McMillan (Eds.). Springer International Publishing, Cham, 377–392. https://doi.org/10.1007/978-3-319-70284-1_30

Digital Library

[6]

Holly P. Branigan, Martin J. Pickering, Jamie Pearson, and Janet F. McLean. 2010. Linguistic alignment between people and computers. Journal of Pragmatics 42, 9 (2010), 2355–2368. https://doi.org/10.1016/j.pragma.2009.12.012

[7]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa

[8]

Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. International Journal of Human–Computer Interaction 37, 8(2021), 729–758. https://doi.org/10.1080/10447318.2020.1841438

[9]

Peng-Yu Chen and Von-Wun Soo. 2018. Humor Recognition Using Deep Learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, 113–117. https://doi.org/10.18653/v1/N18-2018

[10]

Minji Cho, Sang-su Lee, and Kun-Pyo Lee. 2019. Once a Kind Friend is Now a Thing: Understanding How Conversational Agents at Home Are Forgotten. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS ’19). Association for Computing Machinery, New York, NY, USA, 1557–1569. https://doi.org/10.1145/3322276.3322332

Digital Library

[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

[12]

Interaction Design Foundation. 2020. Understanding Early Adopters and Customer Adoption Patterns. https://www.interaction-design.org/literature/article/understanding-early-adopters-and-customer-adoption-patterns.

[13]

Zuohui Fu, Yikun Xian, Yongfeng Zhang, and Yi Zhang. 2021. IUI 2021 Tutorial on Conversational Recommendation Systems. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 1–2. https://doi.org/10.1145/3397482.3450621

Digital Library

[14]

Jianfeng Gao, Michel Galley, and Lihong Li. 2019. Neural Approaches to Conversational AI: Question Answering, Task-oriented Dialogues and Social Chatbots. Now Foundations and Trends. https://doi.org/10.1561/1500000074

Digital Library

[15]

Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16710

[16]

Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 209, 11 pages. https://doi.org/10.1145/3290605.3300439

Digital Library

[17]

Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 415, 12 pages. https://doi.org/10.1145/3173574.3173989

Digital Library

[18]

C. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media 8, 1 (May 2014). https://ojs.aaai.org/index.php/ICWSM/article/view/14550

[19]

Mohit Jain, Pratyush Kumar, Ramachandra Kota, and Shwetak N. Patel. 2018. Evaluating and Informing the Design of Chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). ACM, New York, NY, USA, 895–906. https://doi.org/10.1145/3196709.3196735

Digital Library

[20]

S. Katayama, A. Mathur, M. van den Broeck, T. Okoshi, J. Nakazawa, and F. Kawsar. 2019. Situation-Aware Emotion Regulation of Conversational Agents with Kinetic Earables. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). 725–731. https://doi.org/10.1109/ACII.2019.8925449

[21]

Junhan Kim, Yoojung Kim, Byungjoon Kim, Sukyung Yun, Minjoon Kim, and Joongseek Lee. 2018. Can a Machine Tend to Teenagers’ Emotional Needs? A Study with Conversational Agents. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188548

Digital Library

[22]

Andrei P. Kirilenko and Svetlana Stepchenkova. 2016. Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa. PLOS ONE 11, 3 (03 2016), 1–14. https://doi.org/10.1371/journal.pone.0149787

[23]

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. https://doi.org/10.2307/2529310

[24]

Q. Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael Muller, N. Sadat Shami, and Werner Geyer. 2018. All Work and No Play?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 3, 13 pages. https://doi.org/10.1145/3173574.3173577

Digital Library

[25]

Andrea López, Alissa Detz, Neda Ratanawongsa, and Urmimala Sarkar. 2012. What patients say about their doctors online: a qualitative content analysis. Journal of general internal medicine 27, 6 (2012), 685–692. https://doi.org/10.1007/s11606-011-1958-4

[26]

Ewa Luger and Abigail Sellen. 2016. “Like Having a Really Bad PA”: The Gulf Between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288

Digital Library

[27]

Robert R McCrae and Paul T Costa. 1987. Validation of the five-factor model of personality across instruments and observers.Journal of personality and social psychology 52, 1(1987), 81. https://doi.org/10.1037/0022-3514.52.1.81

[28]

Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica 22, 3 (Oct. 2012), 276–282. https://doi.org/10.11613/BM.2012.031

[29]

Indrani Medhi Thies, Nandita Menon, Sneha Magapu, Manisha Subramony, and Jacki O’Neill. 2017. How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians. In Human-Computer Interaction - INTERACT 2017, Regina Bernhaupt, Girish Dalvi, Anirudha Joshi, Devanuj K. Balkrishan, Jacki O’Neill, and Marco Winckler (Eds.). Springer International Publishing, Cham, 441–459. https://doi.org/10.1007/978-3-319-67744-6_28

[30]

Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 845–854. https://doi.org/10.18653/v1/P19-1081

[31]

Robert J Moore and Raphael Arar. 2019. Conversational UX Design: A Practitioner’s Guide to the Natural Conversation Framework. ACM. https://doi.org/10.1145/3304087

Digital Library

[32]

Susan M Mudambi and David Schuff. 2010. Research note: What makes a helpful online review? A study of customer reviews on Amazon. com. MIS quarterly (2010), 185–200. https://doi.org/10.2307/20721420

[33]

Andreea Muresan and Henning Pohl. 2019. Chats with Bots: Balancing Imitation and Engagement. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). ACM, New York, NY, USA, Article LBW0252, 6 pages. https://doi.org/10.1145/3290607.3313084

Digital Library

[34]

Rosalind W Picard. 2000. Affective computing. MIT press.

[35]

Google Play. 2020. Google Play Terms of Service. https://play.google.com/intl/en_US/about/play-terms/.

[36]

Digital Media Law Project. 2021. Fair Use. http://www.dmlp.org/legal-guide/fair-use.

[37]

Amanda Purington, Jessie G. Taft, Shruti Sannon, Natalya N. Bazarova, and Samuel Hardman Taylor. 2017. “Alexa is my new BFF”: Social Roles, User Satisfaction, and Personification of the Amazon Echo. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’17. ACM Press, Denver, Colorado, USA, 2853–2859. https://doi.org/10.1145/3027063.3053246

Digital Library

[38]

Ben Sheehan, Hyun Seung Jin, and Udo Gottlieb. 2020. Customer service chatbots: Anthropomorphism and adoption. Journal of Business Research 115 (2020), 14 – 24. https://doi.org/10.1016/j.jbusres.2020.04.030

[39]

Ekaterina Svikhnushina and Pearl Pu. 2021. Key Qualities of Conversational Chatbots – the PEACE Model. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 520–530. https://doi.org/10.1145/3397481.3450643

Digital Library

[40]

D.H. Vu, K.M. Muttaqi, and A.P. Agalgaonkar. 2015. A variance inflation factor and backward elimination based robust regression model for forecasting monthly electricity demand using climatic variables. Applied Energy 140(2015), 385 – 394. https://doi.org/10.1016/j.apenergy.2014.12.011

[41]

Anuradha Welivita and Pearl Pu. 2020. A Taxonomy of Empathetic Response Intents in Human Social Conversations. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 4886–4899. https://doi.org/10.18653/v1/2020.coling-main.429

[42]

Christopher G Wetzel, Timothy D Wilson, and James Kort. 1981. The halo effect revisited: Forewarned is not forearmed. Journal of Experimental Social Psychology 17, 4 (1981), 427–439. https://doi.org/10.1016/0022-1031(81)90049-4

[43]

Jennifer Zamora. 2017. “I’m Sorry, Dave, I’m Afraid I Can’T Do That”: Chatbot Perception and Expectations. In Proceedings of the 5th International Conference on Human Agent Interaction (Bielefeld, Germany) (HAI ’17). ACM, New York, NY, USA, 253–260. https://doi.org/10.1145/3125739.3125766

Digital Library

Cited By

Greiner DLemoine J(2025)Bridging the gap: user expectations for conversational AI services with consideration of user expertiseJournal of Services Marketing10.1108/JSM-02-2024-005639:2(76-94)Online publication date: 13-Jan-2025
https://doi.org/10.1108/JSM-02-2024-0056
Mubassira THasan MNoor JIslam A(2024)Enhancing EmoBot: An In-Depth Analysis of User Expectation and Satisfaction in an Emotion-Aware ChatbotProceedings of the 11th International Conference on Networking, Systems, and Security10.1145/3704522.3704527(65-71)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3704522.3704527
Namvarpour MRazi AFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)Uncovering Contradictions in Human-AI Interactions: Lessons Learned from User Reviews of ReplikaCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681909(579-586)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681909
Show More Cited By

Recommendations

Evaluating and Informing the Design of Chatbots
DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not ...
Exploring Effects of Conversational Fillers on User Perception of Conversational Agents
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Through technological advancements in various areas of our lives, Conversational Agents progressed in their human-likeness. In the field of HCI, however, the use of conversational fillers (e.g., "um," "uh," etc.) by Conversational Agents have not been ...
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and Design
Abstract
In this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

June 2021

2082 pages

ISBN:9781450384766

DOI:10.1145/3461778

Editors:
Wendy Ju
Jacobs Technion-Cornell Institute at Cornell Tech
,
Lora Oehlberg
University of Calgary
,
Sean Follmer
Stanford University
,
Sarah Fox
Carnegie Mellon University
,
Stacey Kuznetsov
Arizona State University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Swiss National Science Foundation

Conference

DIS '21

Sponsor:

SIGCHI

DIS '21: Designing Interactive Systems Conference 2021

June 28 - July 2, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Upcoming Conference

DIS '25

Sponsor:
sigchi

Designing Interactive Systems Conference

July 5 - 9, 2025

Funchal , Portugal

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
1,080
Total Downloads

Downloads (Last 12 months)199
Downloads (Last 6 weeks)16

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Greiner DLemoine J(2025)Bridging the gap: user expectations for conversational AI services with consideration of user expertiseJournal of Services Marketing10.1108/JSM-02-2024-005639:2(76-94)Online publication date: 13-Jan-2025
https://doi.org/10.1108/JSM-02-2024-0056
Mubassira THasan MNoor JIslam A(2024)Enhancing EmoBot: An In-Depth Analysis of User Expectation and Satisfaction in an Emotion-Aware ChatbotProceedings of the 11th International Conference on Networking, Systems, and Security10.1145/3704522.3704527(65-71)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3704522.3704527
Namvarpour MRazi AFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)Uncovering Contradictions in Human-AI Interactions: Lessons Learned from User Reviews of ReplikaCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681909(579-586)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681909
Rashik MJasim MKucher KSarvghad AMahyar N(2024)Beyond Text and Speech in Conversational Agents: Mapping the Design Space of AvatarsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661563(1875-1894)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661563
Shin DWang LHsieh G(2024)From Paper to Card: Transforming Design Implications with Generative AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642266(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642266
Karthikeya MAnand S(2024)User Feedback Module for Women Self Help Group Chatbot for Increased Usability2024 5th International Conference for Emerging Technology (INCET)10.1109/INCET61516.2024.10593339(1-7)Online publication date: 24-May-2024
https://doi.org/10.1109/INCET61516.2024.10593339
Richter F(2024)From human-system interaction to human-system co-action and back: ethical assessment of generative AI and mutual theory of mindAI and Ethics10.1007/s43681-024-00626-z5:1(19-28)Online publication date: 4-Dec-2024
https://doi.org/10.1007/s43681-024-00626-z
Kusal SPatil SChoudrie JKotecha KVora D(2024)Transfer learning for emotion detection in conversational text: a hybrid deep learning approach with pre-trained embeddingsInternational Journal of Information Technology10.1007/s41870-024-02027-1Online publication date: 3-Jul-2024
https://doi.org/10.1007/s41870-024-02027-1
Huțul TPopescu AKarner‐Huțuleac AHolman AHuțul A(2024)Who's willing to lay on the virtual couch? Attitudes, anthropomorphism and need for human interaction as factors of intentions to use chatbots for psychotherapyCounselling and Psychotherapy Research10.1002/capr.12794Online publication date: 24-Jun-2024
https://doi.org/10.1002/capr.12794
de Sá Siqueira MMüller BBosse T(2023)When Do We Accept Mistakes from Chatbots? The Impact of Human-Like Communication on User Experience in Chatbots That Make MistakesInternational Journal of Human–Computer Interaction10.1080/10447318.2023.217515840:11(2862-2872)Online publication date: 16-Feb-2023
https://doi.org/10.1080/10447318.2023.2175158
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten