How Does Conversation Length Impact User’s Satisfaction? A Case Study of Length-Controlled Conversations with LLM-Powered Chatbots
Authors: 
Shih-Hong Huang, Ya-Fang Lin, Zeyu He, Chieh-Yang Huang, Ting-Hao Kenneth HuangAuthors Info & Claims
Shih-Hong Huang, Ya-Fang Lin, Zeyu He, Chieh-Yang Huang, Ting-Hao Kenneth HuangAuthors Info & ClaimsArticle No.: 188, Pages 1 - 13
Abstract
Users can discuss a wide range of topics with large language models (LLMs), but they do not always prefer solving problems or getting information through lengthy conversations. This raises an intriguing HCI question: How does instructing LLMs to engage in longer or shorter conversations affect conversation quality? In this paper, we developed two Slack chatbots using GPT-4 with the ability to vary conversation lengths and conducted a user study. Participants asked the chatbots both highly and less conversable questions, engaging in dialogues with 0, 3, 5, and 7 conversational turns. We found that the conversation quality does not differ drastically across different conditions, while participants had mixed reactions. Our study demonstrates LLMs’ ability to change conversation length and the potential benefits for users resulting from such changes, but we caution that changes in text form may not necessarily imply changes in quality or content.
Supplemental Material
MP4 File - Video Preview
Video Preview
- Download
- 9.65 MB
- Transcript
MP4 File
Talk Video
- Download
- 59.10 MB
- Transcript
References
[1]
Theo Araujo. 2018. Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Computers in Human Behavior 85 (2018), 183–189.
[2]
Ana Paula Chaves, Jesse Egbert, Toby Hocking, Eck Doerry, and Marco Aurelio Gerosa. 2022. Chatbots language design: The influence of language variation on user experience with tourist assistant chatbots. ACM Transactions on Computer-Human Interaction 29, 2 (2022), 1–38.
[3]
Zhifa Chen, Yichen Lu, Mika P Nieminen, and Andrés Lucero. 2020. Creating a chatbot for and with migrants: chatbot personality drives co-design activities. In Proceedings of the 2020 acm designing interactive systems conference. 219–230.
[4]
Ela Elsholz, Jon Chamberlain, and Udo Kruschwitz. 2019. Exploring language style in chatbots to increase perceived product value and user engagement. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. 301–305.
[5]
Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2017. Towards Designing Cooperative and Social Conversational Agents for Customer Service. In ICIS. 1–13.
[6]
Herbert P Grice. 1975. Logic and conversation. In Speech acts. Brill, 41–58.
[7]
Jennifer Hill, W Randolph Ford, and Ingrid G Farreras. 2015. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in human behavior 49 (2015), 245–250.
[8]
Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch your heart: A tone-aware chatbot for customer care on social media. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.
[9]
Shih-Hong Huang, Chieh-Yang Huang, Ya-Fang Lin, and Ting-Hao Kenneth Huang. 2023. What Types of Questions Require Conversation to Answer? A Case Study of AskReddit Questions. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–9.
[10]
Ting-Hao Huang, Joseph Chee Chang, and Jeffrey P Bigham. 2018. Evorus: A crowd-powered conversational assistant built to automate itself over time. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
[11]
Ana Jakic, Maximilian Oskar Wagner, and Anton Meyer. 2017. The impact of language style accommodation during social media interactions on brand trust. Journal of Service Management 28, 3 (2017), 418–441.
[12]
Yashal Shakti Kanungo, Gyanendra Das, and Sumit Negi. 2022. COBART: Controlled, optimized, bidirectional and auto-regressive transformer for ad headline generation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3127–3136.
[13]
Crowd AI Lab. 2023. What Types of Questions Require Conversation to Answer? AskReddit Questions Dataset. https://github.com/Crowd-AI-Lab/AskReddit.
[14]
Kellie Morrissey and Jurek Kirakowski. 2013. ‘Realness’ in chatbots: establishing quantifiable criteria. In Human-Computer Interaction. Interaction Modalities and Techniques: 15th International Conference, HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings, Part IV 15. Springer, 87–96.
[15]
Vidya Setlur and Melanie Tory. 2022. How do you converse with an analytical chatbot? revisiting gricean maxims for designing analytical conversational behavior. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–17.
[16]
Xin Sun, Emiel Krahmer, Jan De Wit, Reinout Wiers, and Jos A Bosch. 2023. Plug and Play Conversations: The Micro-Conversation Scheme for Modular Development of Hybrid Conversational Agent. In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing. 50–55.
[17]
Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed. 2018. The ethnobot: Gathering ethnographies in the age of IoT. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
[18]
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. Comput. Surveys 56, 3 (2023), 1–37.
Index Terms
- How Does Conversation Length Impact User’s Satisfaction? A Case Study of Length-Controlled Conversations with LLM-Powered Chatbots
Recommendations
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
CSCWLarge language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal such as collecting self-...
User Engagement with Chatbots: A Discursive Psychology Approach
CUI '20: Proceedings of the 2nd Conference on Conversational User InterfacesConversational agents have transcended into multiple industries with increased ability for user engagement in intelligent conversation. Conversations with chatbots are different from interpersonal communication in terms of turn-taking, intentions, and ...
Exploring how politeness impacts the user experience of chatbots for mental health support
AbstractPoliteness is important in human–human interaction when asking people to engage in sensitive conversations. If politeness manifests similarly in human–chatbot interaction, it may play an important role in the design of sensitive chatbot ...
Highlights- Politeness can both positively and negatively impact the chatbot user experience.
- The Personal politeness chatbot was experienced as caring and encouraging.
- The Passive politeness chatbot was experienced as too apologetic and ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Data Availability
Video Preview: Video Preview https://dl.acm.org/doi/10.1145/3613905.3650823#3613905.3650823-preview.mp4
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 281Total Downloads
- Downloads (Last 12 months)281
- Downloads (Last 6 weeks)98
Reflects downloads up to 12 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format