short-paper

Open access

Conversational Agents Trust Calibration: A User-Centred Perspective to Design

Authors:

Mateusz Dubiel,

Sylvain Daronnat,

Luis A. LeivaAuthors Info & Claims

CUI '22: Proceedings of the 4th Conference on Conversational User Interfaces

Article No.: 30, Pages 1 - 6

https://doi.org/10.1145/3543829.3544518

Published: 15 September 2022 Publication History

All formats PDF

Abstract

Previous work identified trust as one of the key requirements for adoption and continued use of conversational agents (CAs). Given recent advances in natural language processing and deep learning, it is currently possible to execute simple goal-oriented tasks by using voice. As CAs start to provide a gateway for purchasing products and booking services online, the question of trust and its impact on users’ reliance and agency becomes ever-more pertinent. This paper collates trust-related literature and proposes four design suggestions that are illustrated through example conversations. Our goal is to encourage discussion on ethical design practices to develop CAs that are capable of employing trust-calibration techniques that should, when relevant, reduce the user’s trust in the agent. We hope that our reflections, based on the synthesis of insights from the fields of human-agent interaction, explainable ai, and information retrieval, can serve as a reminder of the dangers of excessive trust in automation and contribute to more user-centred CA design.

References

[1]

Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977(2020).

[2]

Kamran Alipour, Jurgen P Schulze, Yi Yao, Avi Ziskind, and Giedrius Burachas. 2020. A study on multimodal and interactive explanations for visual question answering. arXiv preprint arXiv:2003.00431(2020).

[3]

Jorge A. Alvarado-Valencia and Lope H. Barrero. 2014. Reliance, trust and heuristics in judgmental forecasting. Computers in Human Behavior 36 (2014), 102–113. https://doi.org/10.1016/j.chb.2014.03.047

Digital Library

[4]

Tawfiq Ammari, Jofish Kaye, Janice Y Tsai, and Frank Bentley. 2019. Music, Search, and IoT: How People (Really) Use Voice Assistants.ACM Trans. Comput. Hum. Interact. 26, 3 (2019), 17–1.

Digital Library

[5]

Salvatore Andolina, Valeria Orso, Hendrik Schneider, Khalil Klouche, Tuukka Ruotsalo, Luciano Gamberini, and Giulio Jacucci. 2018. Investigating Proactive Search Support in Conversations. In Proceedings of the 2018 Designing Interactive Systems Conference. ACM, 1295–1307.

Digital Library

[6]

Matthew P Aylett, Selina Jeanne Sutton, and Yolanda Vazquez-Alvarez. 2019. The right kind of unnatural: designing a robot voice. In Proceedings of the 1st International Conference on Conversational User Interfaces. 1–2.

Digital Library

[7]

Marcia J Bates. 1990. Where should the person stop and the information search interface start?Information Processing & Management 26, 5 (1990), 575–591.

[8]

Rodrigo Bavaresco, Diórgenes Silveira, Eduardo Reis, Jorge Barbosa, Rodrigo Righi, Cristiano Costa, Rodolfo Antunes, Marcio Gomes, Clauter Gatti, Mariangela Vanzin, Saint Clair Junior, Elton Silva, and Carlos Moreira. 2020. Conversational agents in business: A systematic literature review and future research directions. Computer Science Review 36 (2020), 100239. https://doi.org/10.1016/j.cosrev.2020.100239

[9]

Jessie Y Chen, Katelyn Procci, Michael Boyce, Julia Wright, Andre Garcia, and Michael Barnes. 2014. Situation awareness-based agent transparency. Technical Report. Army research lab aberdeen proving ground md human research and engineering ….

[10]

Aleksandr Chuklin, Aliaksei Severyn, Johanne Trippas, Enrique Alfonseca, Hanna Silen, and Damiano Spina. 2018. Prosody modifications for question-answering in voice-only settings. arXiv preprint arXiv:1806.03957(2018).

[11]

Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, 2019. What makes a good conversation? Challenges in designing truly conversational agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[12]

Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. ” What can i help you with?” infrequent users’ experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–12.

Digital Library

[13]

Ewart J De Visser, Samuel S Monfort, Ryan McKendrick, Melissa AB Smith, Patrick E McKnight, Frank Krueger, and Raja Parasuraman. 2016. Almost human: Anthropomorphism increases trust resilience in cognitive agents.Journal of Experimental Psychology: Applied 22, 3 (2016), 331.

[14]

Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.

[15]

Mateusz Dubiel, Martin Halvey, Leif Azzopardi, Damien Anderson, and Sylvain Daronnat. 2020. Conversational strategies: impact on search performance in a goal-oriented task. In The Third International Workshop on Conversational Approaches to Information Retrieval.

[16]

Mateusz Dubiel, Martin Halvey, Pilar Oplustil Gallegos, and Simon King. 2020. Persuasive synthetic speech: Voice perception and user behaviour. In Proceedings of the 2nd Conference on Conversational User Interfaces. 1–9.

Digital Library

[17]

Justin Edwards and Elaheh Sanoubari. 2019. A need for trust in conversational interface research. In Proceedings of the 1st International Conference on Conversational User Interfaces. 1–3.

Digital Library

[18]

Aaron C Elkins and Douglas C Derrick. 2013. The sound of trust: voice as a measurement of trust during interactions with embodied conversational agents. Group decision and negotiation 22, 5 (2013), 897–913.

[19]

Xiaocong Fan, Sooyoung Oh, Michael McNeese, John Yen, Haydee Cuevas, Laura Strater, and Mica R. Endsley. 2008. The influence of agent reliability on trust in human-agent collaboration. Proceedings of the 15th European conference on Cognitive ergonomics the ergonomics of cool interaction - ECCE ’08(2008), 1. https://doi.org/10.1145/1473018.1473028

Digital Library

[20]

Andrew Gibiansky, Sercan Ömer Arik, Gregory Frederick Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep Voice 2: Multi-Speaker Neural Text-to-Speech. In NIPS.

[21]

Yunha Han. 2021. Wait, Let’s Think about Your Purchase Again: A Study on Interventions for Supporting Self-Controlled Online Purchases. (2021). To Appear in Proceedings of Web Conference 2021.

[22]

Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors 57, 3 (2015), 407–434. https://doi.org/10.1177/0018720814547570

[23]

Philipp Kirschthaler, Martin Porcheron, and Joel E Fischer. 2020. What can i say? effects of discoverability in vuis on task performance and user experience. In Proceedings of the 2nd Conference on Conversational User Interfaces. 1–9.

Digital Library

[24]

Alexander Kunze, Stephen J. Summerskill, Russell Marshall, and Ashleigh J. Filtness. 2019. Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics 62, 3 (2019), 345–360. https://doi.org/10.1080/00140139.2018.1547842

[25]

J. D. Lee and K. A. See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society 46, 1(2004), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

[26]

Rosemarijn Looije, Mark A Neerincx, and Fokie Cnossen. 2010. Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors. International Journal of Human-Computer Studies 68, 6 (2010), 386–397.

Digital Library

[27]

Ewa Luger and Abigail Sellen. 2016. ” Like Having a Really Bad PA” The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI conference on human factors in computing systems. 5286–5297.

Digital Library

[28]

D. Manzey, J Elin Bahner, and Anke-Dorothea Hueper. 2006. Misuse of Automated Aids in Process Control: Complacency, Automation Bias and Possible Training Interventions. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 3(2006), 220–224. https://doi.org/10.1177/154193120605000303

[29]

John M. McGuirl and Nadine B. Sarter. 2006. Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information. Human Factors 48, 4 (2006), 656–665. https://doi.org/10.1518/001872006779166334 arXiv:https://doi.org/10.1518/001872006779166334PMID: 17240714.

[30]

Bonnie M. Muir. 1987. Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies 27, 5-6 (1987), 527–539. https://doi.org/10.1016/S0020-7373(87)80013-5

Digital Library

[31]

M. Ng, K. P. L. Coopamootoo, E. Toreini, M. Aitken, K. Elliot, and A. van Moorsel. 2020. Simulating the Effects of Social Presence on Trust, Privacy Concerns Usage Intentions in Automated Bots for Finance. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS PW). 190–199. https://doi.org/10.1109/EuroSPW51379.2020.00034

[32]

John O’Donovan and Barry Smyth. 2005. Trust in recommender systems. In Proceedings of the 10th international conference on Intelligent user interfaces. 167–174.

Digital Library

[33]

Thomas C O’Guinn and Ronald J Faber. 1989. Compulsive buying: A phenomenological exploration. Journal of consumer research 16, 2 (1989), 147–157.

[34]

Richard Pak, Nicole Fink, Margaux Price, Brock Bass, and Lindsay Sturre. 2012. Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics 55, 9 (2012), 1059–1072.

[35]

Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30, 3 (2000), 286–297.

Digital Library

[36]

Claudio S Pinhanez. 2021. Expose Uncertainty, Instill Distrust, Avoid Explanations: Towards Ethical Guidelines for AI. arXiv preprint arXiv:2112.01281(2021).

[37]

Lingyun Qiu and Izak Benbasat. 2009. Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. Journal of management information systems 25, 4 (2009), 145–182.

Digital Library

[38]

Minjin Rheu, Ji Youn Shin, Wei Peng, and Jina Huh-Yoo. 2021. Systematic review: Trust-building factors and implications for conversational agent design. International Journal of Human–Computer Interaction 37, 1(2021), 81–96.

[39]

Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215.

[40]

Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A. Keim. 2016. The Role of Uncertainty, Awareness, and Trust in Visual Analytics. IEEE Transactions on Visualization and Computer Graphics 22, 1(2016), 240–249. https://doi.org/10.1109/TVCG.2015.2467591

Digital Library

[41]

Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). 41–51.

[42]

Ameneh Shamekhi, Q Vera Liao, Dakuo Wang, Rachel KE Bellamy, and Thomas Erickson. 2018. Face Value? Exploring the effects of embodiment for a group facilitation agent. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.

Digital Library

[43]

Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6(2020), 495–504.

[44]

Kimberly Stowers, Nicholas Kasdaglis, Michael Rupp, Jessie Chen, Daniel Barber, and Michael Barnes. 2017. Insights into Human-Agent Teaming: Intelligent Agent Transparency and Uncertainty. In Advances in Human Factors in Robots and Unmanned Systems, Pamela Savage-Knepshield and Jessie Chen (Eds.). Springer International Publishing, Cham, 149–160.

[45]

Éva Székely, Gustav Eje Henter, Jonas Beskow, and Joakim Gustafson. 2019. Spontaneous Conversational Speech Synthesis from Found Data. In INTERSPEECH. 4435–4439.

[46]

Madiha Tabassum, Tomasz Kosiński, Alisa Frik, Nathan Malkin, Primal Wijesekera, Serge Egelman, and Heather Richter Lipford. 2019. Investigating Users’ Preferences and Expectations for Always-Listening Voice Assistants. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1–23.

Digital Library

[47]

Ilaria Torre, Jeremy Goslin, Laurence White, and Debora Zanatto. 2018. Trust in artificial voices: A” congruency effect” of first impressions and behavioural experience. In Proceedings of the Technology, Mind, and Society. 1–6.

Digital Library

[48]

Johanne R Trippas. 2019. Spoken Conversational Search: Audio-only Interactive Information Retrieval. Ph.D. Dissertation. PhD thesis, RMIT, Melbourne.

[49]

Nigel Ward, Jonathan E Avila, and Aaron M Alarcon. 2021. Towards Continuous Estimation of Dissatisfaction in Spoken Dialog. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 13–20.

[50]

Nigel G Ward. 2019. Prosodic patterns in English conversation. Cambridge University Press.

[51]

Nigel G Ward, Ambika Kirkland, Marcin Włodarczak, and Eva Székely. [n.d.]. Two Pragmatic Functions of Breathy Voice in American English Conversation. ([n. d.]).

[52]

Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. 2020. “Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design. Journal on Multimodal User Interfaces(2020), 1–12.

[53]

Ryen W White and Ian Ruthven. 2006. A study of interface support mechanisms for interactive information retrieval. Journal of the American Society for Information Science and Technology 57, 7 (2006), 933–948.

Digital Library

[54]

Christine T. Wolf and Kathryn E. Ringland. 2020. Designing Accessible, Explainable AI (XAI) Experiences. SIGACCESS Access. Comput.125, Article 6 (March 2020), 1 pages. https://doi.org/10.1145/3386296.3386302

Digital Library

[55]

Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256(2016).

[56]

Sunghwan Yi and Hans Baumgartner. 2011. Coping with guilt and shame in the impulse buying context. Journal of Economic Psychology 32, 3 (2011), 458–467.

[57]

Debora Zanatto, Massimiliano Patacchiola, Jeremy Goslin, and Angelo Cangelosi. 2016. Priming anthropomorphism: Can the credibility of humanlike robots be transferred to non-humanlike robots?. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 543–544.

Cited By

Zargham NDubiel MDesai SMildner TBelz H(2024)Designing AI Personalities: Enhancing Human-Agent Interaction Through Thoughtful Persona DesignProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701608(490-494)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3701608
Truong LLee SSawhney N(2024)Enhancing Conversations in Migrant Counseling Services: Designing for Trustworthy Human-AI CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/36870348:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687034
Dubiel MDesai SZargham NSchmitt A(2024)Voicecraft: Designing Task-specific Voice Assistant PersonasProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3670000(1-3)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3670000
Show More Cited By

Index Terms

Conversational Agents Trust Calibration: A User-Centred Perspective to Design
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces
    2. Interaction techniques
      1. Auditory feedback
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods

Recommendations

The effects of personalizaion and familiarity on trust and adoption of recommendation agents

In the context of personalization technologies, such as Web-based product-brokering recommendation agents (RAs) in electronic commerce, existing technology acceptance theories need to be expanded to take into account not only the cognitive beliefs ...
A need for trust in conversational interface research
CUI '19: Proceedings of the 1st International Conference on Conversational User Interfaces

Across several branches of conversational interaction research including interactions with social robots, embodied agents, and conversational assistants, users have identified trust as a critical part of those interactions. Nevertheless, there is little ...
Conversational Agents: Acting on the Wave of Research and Development
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

In the last five years, work on software that interacts with people via typed or spoken natural language, called chatbots, intelligent assistants, social bots, virtual companions, non-human players, and so on, increased dramatically. Chatbots burst into ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CUI '22: Proceedings of the 4th Conference on Conversational User Interfaces

July 2022

289 pages

ISBN:9781450397391

DOI:10.1145/3543829

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2022

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

CUI 2022

CUI 2022: 4th Conference on Conversational User Interfaces

July 26 - 28, 2022

Glasgow, United Kingdom

Acceptance Rates

CUI '22 Paper Acceptance Rate 12 of 33 submissions, 36%;

Overall Acceptance Rate 34 of 100 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
1,136
Total Downloads

Downloads (Last 12 months)540
Downloads (Last 6 weeks)87

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zargham NDubiel MDesai SMildner TBelz H(2024)Designing AI Personalities: Enhancing Human-Agent Interaction Through Thoughtful Persona DesignProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701608(490-494)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3701608
Truong LLee SSawhney N(2024)Enhancing Conversations in Migrant Counseling Services: Designing for Trustworthy Human-AI CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/36870348:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687034
Dubiel MDesai SZargham NSchmitt A(2024)Voicecraft: Designing Task-specific Voice Assistant PersonasProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3670000(1-3)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3670000
Pias SHuang RWilliamson DKim MKapadia A(2024)The Impact of Perceived Tone, Age, and Gender on Voice Assistant Persuasiveness in the Context of Product RecommendationsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665545(1-15)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665545
Kovacevic NBoschung THolz CGross MWampfler R(2024)Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality InfusionProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665543(1-16)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665543
Cohn MPushkarna MOlanubi GMoran JPadgett DMengesha ZHeldreth C(2024)Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language ModelsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650818(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650818
Desai SWei CSin JDubiel MZargham NAhire SPorcheron MKuzminykh ALee MCandello HFischer JMunteanu CCowan B(2024)CUI@CHI 2024: Building Trust in CUIs—From Design to DeploymentExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636287(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636287
Mildner TCooney OMeck ABartl MSavino GDoyle PGaraialde DClark LSloan JWenig NMalaka RNiess J(2024)Listening to the Voices: Describing Ethical Caveats of Conversational User Interfaces According to Experts and Frequent UsersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642542(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642542
Radensky MSéguin JLim JOlson KGeiger R(2023)“I Think You Might Like This”: Exploring Effects of Confidence Signal Patterns on Trust in and Reliance on Conversational Recommender SystemsProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594043(792-804)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594043
Fischer J(2023)Generative AI Considered HarmfulProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3603756(1-5)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3603756
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten