research-article

When There is No Progress with a Task-Oriented Chatbot: A Conversation Analysis

Authors:

MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

Article No.: 59, Pages 1 - 6

https://doi.org/10.1145/3338286.3344407

Published: 01 October 2019 Publication History

Get Access

Abstract

Task-oriented chatbots are increasingly prevalent in our daily life. Research effort has been devoted to advancing our understanding of users' interaction with conversational agents, including conversation breakdowns. However, most research attempts were limited to obversions from a relatively short duration of user interaction with chatbots, where users were aware of being studied. In this study, we conducted a conversation analysis on a three-month conversation log of users conversing with a chatbot of a banking institution. The log consisted of 1,837 users' conversations with this chatbot with 19,449 message exchanges. From this analysis, we show that users more often failed to make a progress in a conversation when they requested information than when they provided information. Furthermore, we uncovered five kinds of intention gaps unexpected to the chatbot, and five major behaviors users adopted to cope with non-progress.

References

[1]

Paul M. Aoki, Margaret H. Szymanski, Luke Plurkowski, James D. Thornton, Allison Woodruff, Weilie Yi, 2006. Where's the "party" in "multiparty"?: analyzing the structure of small-group sociable talk. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006.

Google Scholar

[2]

Zahra Ashktorab, Mohit Jain, Q. Vera Liao, & Justin D. Weisz. 2019. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 1--1

Digital Library

Google Scholar

[3]

Jonathan Grudin, Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. (CHI '19), 1--11.

Digital Library

Google Scholar

[4]

Mohit Jain, Ramachandra Kota, Pratyush Kumar, & Shwetak Patel. (2018). Convey: Exploring the Use of a Context View for Chatbots. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 1--6.

Digital Library

Google Scholar

[5]

Jiepu Jiang, Wei Jeng, & Daqing He. 2013. How Do Users Respond to Voice Input Errors?: Lexical and Phonetic Query Reformulation in Voice Search. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, 143--152.

Digital Library

Google Scholar

[6]

Jane Lockwood. 2017. An analysis of web-chat in an outsourced customer service account in the Philippines, English for Specific Purposes, 47, 26--39.

Crossref

Google Scholar

[7]

Matthew Lombard, Jennifer Snyder-Duch, Cheryl Campanella Bracken. 2010. Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects.

Google Scholar

[8]

Ewa Luger, Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents, In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 5286--5297.

Digital Library

Google Scholar

[9]

Joanne Meredith. 2017. Analysing technological affordances of online interactions using conversation analysis. Journal of Pragmatics, 115, 42--55.

Crossref

Google Scholar

[10]

Vera Q. Liao, Muhammed Masud Hussain, Praveen Chandar, Matthew Davis, Marco Crasso, Dakuo Wang, Michael Muller, Sadat N. Shami, and Werner Geyer. 2018. All Work and no Play? Conversations with a Question-and-Answer Chatbot in the Wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)

Google Scholar

[11]

Filip Radlinski, Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (CHIR '17), 117--126.

Digital Library

Google Scholar

[12]

Wyke Stommel, Trena M. Paulus, David P. Atkins, "Here's the link: Hyperlinking in service-focused chat interaction," Journal of Pragmatics 115 (2017) pp. 56--57.

Google Scholar

Cited By

View all

Liesenfeld ADingemanse M(2024)Interactive probes: Towards action-level evaluation for dialogue systemsDiscourse & Communication10.1177/17504813241267071Online publication date: 12-Sep-2024
https://doi.org/10.1177/17504813241267071
He LBraggaar ABasar EKrahmer EAntheunis MWiers R(2024)Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot InteractionsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665536(1-14)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665536
Hasani MPurwandari KIbrahim MArifin SSafira WPrabaswara B(2024)Utterance Intent Recognition for Online Retail2024 3rd International Conference on Digital Transformation and Applications (ICDXA)10.1109/ICDXA61007.2024.10470915(199-204)Online publication date: 29-Jan-2024
https://doi.org/10.1109/ICDXA61007.2024.10470915
Show More Cited By

Recommendations

A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Task-oriented chatbots are becoming popular alternatives for fulfilling users' needs, but few studies have investigated how users cope with conversational 'non-progress' (NP) in their daily lives. Accordingly, we analyzed a three-month conversation log ...
Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking
CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

Use of chatbots for emotional exchange is recently increasing in various domains. However, as existing chatbots have been considered in terms of natural language processing techniques for interaction with text-based chatting, chatbot interaction with ...
Action-a-Bot: Exploring Human-Chatbot Conversations for Actionable Instruction Giving and Following
CSCW'22 Companion: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing

Conversation serves as one critical mechanism for knowledge-sharing and instruction-giving in collaborative work. Conversation allows people to take turns to make contributions, plan joint actions, align shared understanding of work status and resolve ...

Comments

Information & Contributors

Information

Published In

MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

October 2019

646 pages

ISBN:9781450368254

DOI:10.1145/3338286

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ministry of Science and Technology, Taiwan

Conference

MobileHCI '19

Sponsor:

SIGCHI

MobileHCI '19: 21st International Conference on Human-Computer Interaction with Mobile Devices and Services

October 1 - 4, 2019

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
946
Total Downloads

Downloads (Last 12 months)176
Downloads (Last 6 weeks)18

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Liesenfeld ADingemanse M(2024)Interactive probes: Towards action-level evaluation for dialogue systemsDiscourse & Communication10.1177/17504813241267071Online publication date: 12-Sep-2024
https://doi.org/10.1177/17504813241267071
He LBraggaar ABasar EKrahmer EAntheunis MWiers R(2024)Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot InteractionsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665536(1-14)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665536
Hasani MPurwandari KIbrahim MArifin SSafira WPrabaswara B(2024)Utterance Intent Recognition for Online Retail2024 3rd International Conference on Digital Transformation and Applications (ICDXA)10.1109/ICDXA61007.2024.10470915(199-204)Online publication date: 29-Jan-2024
https://doi.org/10.1109/ICDXA61007.2024.10470915
Liu YMartens J(2024)Conversation-based hybrid UI for the repertory grid techniqueInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103227184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103227
da Silva Fernandes URodrigues Barbosa GAzevedo Chagas BJunqueira Barbosa GJunqueira Barbosa SOliveira Prates R(2023)Lessons Learned from Modeling the Interaction with Conversational AgentsInteraction Design and Architecture(s)10.55612/s-5002-055-007(139-173)Online publication date: 1-Feb-2023
https://doi.org/10.55612/s-5002-055-007
Salehzadeh Niksirat KKorka DHarkous HHuguenin KCherubini M(2023)On the Potential of Mediation Chatbots for Mitigating Multiparty Privacy Conflicts - A Wizard-of-Oz StudyProceedings of the ACM on Human-Computer Interaction10.1145/35796187:CSCW1(1-33)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579618
Hasani MHeryadi YArifin YLukas Suparta W(2023)Density Based Spatial Clustering of Applications with Noise and Sentence Bert Embedding for Indonesian Utterance Clustering2023 International Conference on Computer Science, Information Technology and Engineering (ICCoSITE)10.1109/ICCoSITE57641.2023.10127683(386-391)Online publication date: 16-Feb-2023
https://doi.org/10.1109/ICCoSITE57641.2023.10127683
Wube HEsubalew SWeldesellasie FDebelee T(2022)Text-Based Chatbot in Financial Sector: A Systematic Literature ReviewData Science in Finance and Economics10.3934/DSFE.20220112:3(232-259)Online publication date: 2022
https://doi.org/10.3934/DSFE.2022011
Lee LBraud THosio SHui P(2021)Towards Augmented Reality Driven Human-City Interaction: Current Research on Mobile Headsets and Future ChallengesACM Computing Surveys10.1145/346796354:8(1-38)Online publication date: 4-Oct-2021
https://dl.acm.org/doi/10.1145/3467963
Khilji AManna RLaskar SPakray PDas DBandyopadhyay SGelbukh A(2021)CookingQA: Answering Questions and Recommending Recipes Based on IngredientsArabian Journal for Science and Engineering10.1007/s13369-020-05236-5Online publication date: 7-Jan-2021
https://doi.org/10.1007/s13369-020-05236-5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot

Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking

Action-a-Bot: Exploring Human-Chatbot Conversations for Actionable Instruction Giving and Following

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot

Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking

Action-a-Bot: Exploring Human-Chatbot Conversations for Actionable Instruction Giving and Following

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations