research-article

"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents

Authors:

Abigail SellenAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 5286 - 5297

https://doi.org/10.1145/2858036.2858288

Published: 07 May 2016 Publication History

Abstract

The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.

References

[1]

ALICE http://www.alicebot.org/bios/richardwallace.html Artificial Intelligence Foundation

[2]

Jennifer Attride-Stirling. 2001. Thematic networks: an analytic tool for qualitative research. Qualitative Research, 1(3), 385-405

[3]

Margaret A Boden. 2007. Conversationalists and Confidants. In Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute. Retrieved from http://www.academia.edu/2669520/Artificial_Companions_in_Society_Perspectives_on_the_Present_and_Future

[4]

Richard, A Bolt. 1980. "Put-That-There": Voice and Gesture at the Graphics Interface, In the proceedings of SIGGRAGH '80 Proceedings, 14, 3 (July 1980), 262270.

Digital Library

[5]

Timothy Bickmore and Rosalind W Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction. 12, 2: 293--327.

Digital Library

[6]

Susan Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. In The Art of Human-Computer Interface Design. B.K. Laurel (Ed.), Reading, MA: Addison-Wesley.

[7]

Alan Bryman. 2004. Social Research Methods (2nd Ed). NY: Oxford University Press

[8]

Justine Cassell. 2001. Embodied Conversational Agent: Representation and Intelligence in User Interfaces. AI Magazine. 22, 4: 67--83.

Digital Library

[9]

Justine Cassell. Tim Bickmore, Lee Campbell, Hannes Vihjalmsson & Hao Yan. 2000. Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Embodied Conversational Agents. Justine Cassell. (ed) MIT Press: 29--63

Digital Library

[10]

Martin Davis. 2000. Engines of Logic: Mathematicians and the Origin of the Computer. W W Norton & Company

Digital Library

[11]

Anind Dey and Jennifer Mankoff. 2005. Designing mediation for context-aware applications. ACM Transactions on Computer-Human Interaction 12, 1: 53--80. doi 10.1145/1057237.1057241

Digital Library

[12]

Abbe Don, Susan Brennen., Brenda Laurel and Ben Shneiderman. 1992 Anthropomorphism: from Eliza to Terminator 2. Panel In Proc. CHI '92, Bauersfeld, P., Bennett, J., and Lynch, G. (eds.). ACM, 67--70

Digital Library

[13]

Jilian D'Onfro. 2015. Microsoft Created a Chatbot in China that has Millions of Loyal Followers who talk to it like in the Movie 'Her'. Business Insider UK. http://uk.businessinsider.com/microsoft-chatbotxiaoice-2015-8?r=US&IR=T

[14]

Mauro Dragone, Thomas Holz, Brian R. Duffy, Greogory M.P. O'Hare. 2005. Social Situated Agents in Virtual, Real and Mixed Reality Environments. In Intelligent Virtual Agents. Themis Panayiotopoulos, Jonathan Gratch, Ruth Aylett, Daniel Ballin, Patrick Olivier & Thomas Rist (Eds.) Proceedings of 5th International Working Conference, IVA 2005 Kos, Greece, September 12--14, 2005. Springer: 166--177.

Digital Library

[15]

James R. Glass, 1999. Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA

[16]

Erving Goffman. 1967. Interaction Ritual: Essays on Face-to-Face Behaviour. Pathenon

[17]

Aurthur Graesser, Haiying Li & Carol Forsyth. 2014. Learning by Communicating in Natural Language with Conversational Agents. Current Directions in Psychological Science. 23, 5: 374--380

[18]

Greg Guest, Arwen Bunce & Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 1: 59--82.

[19]

Joakim Gustafson, Johan Boye, Morgan Fredriksson, Lasse Johanneson, Jürgen Königsmann. 2005. Providing Computer Game Characters with Conversational Abilities. In Intelligent Virtual Agents. Themis Panayiotopoulos, Jonathan Gratch, Ruth Aylett, Daniel Ballin, Patrick Olivier & Thomas Rist (Eds.) Proceedings of 5th International Working Conference, IVA 2005 Kos, Greece, September 12--14, 2005. Springer: 37--51

Digital Library

[20]

Jürgen Habermas. 1998. On the Pragmatics of Communication. Cambridge: Polity Press.

[21]

Richard H. R. Harper. 2010. Texture: Human Expression in the Age of Communications Overload. MIT Press

[22]

Kerstin Heuwinkel. 2012. Framing the Invisible - The Social Background of Trust. In Your Virtual Butler: The Making-of. Robert Trappl (ed). Springer, 16--26.

[23]

Ido A. Iurgel & Manuel Ziegler. 2005. Ask & Answer: An Educational Game Where It Pays to Endear Your Capricious Virtual Companion. In Intelligent Virtual Agents. Themis Panayiotopoulos, Jonathan Gratch, Ruth Aylett, Daniel Ballin, Patrick Olivier & Thomas Rist (Eds.) Proceedings of 5th International Working Conference, IVA 2005 Kos, Greece, September 12--14, 2005. Springer: 15--24

Digital Library

[24]

Stephan Kopp, Lars Gesellensetter, Nicole, C. Krämer & Ipke Wachsmuth. 2005. A Conversational Agent as Museum Guide: Design and Evaluation of a RealWorld Application. Intelligent Virtual Agents. 3661: 329--343

Digital Library

[25]

Joseph Carl Robnett Licklider. Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics, HFE-1,(March, 1960), 4--11

[26]

Paul P. Maglio and Christopher S. Campbell. 2003. Attentive agents. Communications of the ACM 46, 3: 47--51

Digital Library

[27]

Cade Metz. 2015. Get a Peek at Using Facebook's New Assistant, 'M'. Wired. Retrieved from http://www.wired.com/2015/09/get-peek-someoneusing-facebooks-new-assistant-m/ (accessed 07.09.15)

[28]

Roger Moore. 2012. Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed). Springer, 119--133

Digital Library

[29]

Andreea I Niculescu, Kheng Hui Yeo, Luis F D'Haro, Seokhwan Kim, Ridong Jiang & Rafael E Banchs. 2014. Design and evaluation of a conversational agent for the touristic domain. In Proceedings of APSIPA'14: 1-10.

[30]

Don Norman. 2013. The Design of Everyday Things. Basic Books.

Digital Library

[31]

Kenton O'Hara, Richard Harper, Helena Mentis. Abigail Sellen and Alex Taylor. 2013. On the naturalness of touchless: putting the "interaction" back into NUI. ACM Transactions on Computer-Human Interaction. 20, 1

Digital Library

[32]

Sabine Payr. 2012. Virtual Butlers and Real People: Styles and Practices in Long Term Use of a Companion. In Your Virtual Butler: The Making-of. Robert Trappl (ed). Springer, 134--178

Digital Library

[33]

Jenny Preece, Yvonne Rogers & Helen Sharp. 2015. Interaction Design: Beyond Huma- Computer Interaction. Wiley & Sons Ltd

[34]

Stephen Pulman. Johan Boye, Marc Cavazza, Cameron Smith & Raúl Santos de la Cámara. 2010. How was your Day? In Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics, Stroudsburg, PA, USA: 37-

Digital Library

[35]

Byron Reeves & Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. NY: Cambridge University Press.

Digital Library

[36]

Deborah Richards. 2012. Agent-based museum and tour guides: applying the state of the art. In Proceedings of The 8th Australasian Conference on Interactive Entertainment: Playing the System (IE '12). ACM (2012)

Digital Library

[37]

Lina Maria Rojas-Barahona and Christophe Cerisara. 2014. Bayesian Inverse Reinforcement Learning for Modelling Conversational Agents in a Virtual Environment. Computational Linguistics and Intelligent Text Processing. 8403: 503--514

Digital Library

[38]

Nicole Shechtman, and Leonard M Horowitz. 2003 Media Inequality in Conversation: How People Behave Differently when Interacting with Computers and People. In Proceedings of the. SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, (2003), 281--288

Digital Library

[39]

Nathan Shedroff & Christopher Noessel. 2012. Make it so: Interaction Design Lessons from Science Fiction. Brooklyn, NY: Rosenfeld

[40]

Stergios Tegos, Stavros Demetriadis, Thrasyvoulos Tsiatsos. 2012. Using a Conversational Agent for Promoting Collaborative Language Learning. In Proceedings of Fourth International Conference on Intelligent Networking and Collaborative Systems: 162--165

Digital Library

[41]

Janienke Sturm, Else den Os, Lou Boves. 1999. Issues in Spoken Dialogue Systems: Experiences with the Dutch ARISE System. In the proceedings of ESCA Workshop on Interactive Dialogue in Multi-Modal Systems (UDS-99), Kolster Irsee, Germany.

[42]

Robert Trappl. 2012. From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed). Springer, 1--8

Digital Library

[43]

Giorgio Vassallo, Giovanni Pilato, Agnese Augello & Salvatore Gaglio. 2010. Phrase Coherence in Conceptual Spaces for Conversational Agents. In Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds). IEEE: 357--371

[44]

Astrid M. Von der Pütten, Nicloe C. Krämer, Jonathan Gratch & Sin-Hwa Kang. 2010. "It doesn't matter what you are!" Explaining Social Effects of Agents and Avatars. In Computers in Human Behaviour, 26: 16411650

Digital Library

[45]

Joseph Weizenbaum. 1976. Computer Power and Human Reason: From Judgement to Calculation. W. H. Freeman

Digital Library

[46]

Joseph Weizenbaum. 1965. Eliza: A Computer Program for the Study of Natural Language Communication between Man and Machine. In Communications of the ACM. 9,1:36--45

Digital Library

[47]

Yorick Wilks. 2010. Is a companion a distinctive kind of relationship with a machine?. In Proceedings of the 2010 Workshop on Companionable Dialogue Systems (CDS '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 13--18

Digital Library

[48]

Yorick Wilks (ed.). 2010. Close Engagements with Artificial Companions. Key social, psychological, ethical and design issues. Amsterdam: John Benjamins.

Cited By

Kenaphoom SSupsin JSuktam WLapchit S(2024)Digital Government and AISustainable Development, Humanities, and Social Sciences for Society 5.010.4018/979-8-3693-7989-9.ch007(125-142)Online publication date: 18-Oct-2024
https://doi.org/10.4018/979-8-3693-7989-9.ch007
(2024)Enhancing Team Collaboration With AIHolistic Approach to AI and Leadership10.4018/979-8-3693-2695-4.ch006(120-157)Online publication date: 28-Jun-2024
https://doi.org/10.4018/979-8-3693-2695-4.ch006
Doi TNishikawa Y(2024)Usability Analysis of Smart Speakers from a Learnability Perspective for Novel UsersApplied System Innovation10.3390/asi70300367:3(36)Online publication date: 25-Apr-2024
https://doi.org/10.3390/asi7030036
Show More Cited By

Index Terms

"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
1. Human-centered computing

Recommendations

ELIZA—a computer program for the study of natural language communication between man and machine
Chatbots for customer service: user experience and motivation
CUI '19: Proceedings of the 1st International Conference on Conversational User Interfaces

Companies are increasingly using chatbots to provide customer service. Despite this trend, little in-depth research has been conducted on user experience and user motivation for this important application area of conversational interfaces. To close this ...
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Artificial Intelligence (AI) plays an increasingly important role in improving HCI and user experience. Yet many challenges persist in designing and innovating valuable human-AI interactions. For example, AI systems can make unpredictable errors, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

672
Total Citations
View Citations
8,397
Total Downloads

Downloads (Last 12 months)984
Downloads (Last 6 weeks)144

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kenaphoom SSupsin JSuktam WLapchit S(2024)Digital Government and AISustainable Development, Humanities, and Social Sciences for Society 5.010.4018/979-8-3693-7989-9.ch007(125-142)Online publication date: 18-Oct-2024
https://doi.org/10.4018/979-8-3693-7989-9.ch007
(2024)Enhancing Team Collaboration With AIHolistic Approach to AI and Leadership10.4018/979-8-3693-2695-4.ch006(120-157)Online publication date: 28-Jun-2024
https://doi.org/10.4018/979-8-3693-2695-4.ch006
Doi TNishikawa Y(2024)Usability Analysis of Smart Speakers from a Learnability Perspective for Novel UsersApplied System Innovation10.3390/asi70300367:3(36)Online publication date: 25-Apr-2024
https://doi.org/10.3390/asi7030036
Izadi SForouzanfar M(2024)Error Correction and Adaptation in Conversational AI: A Review of Techniques and Applications in ChatbotsAI10.3390/ai50200415:2(803-841)Online publication date: 4-Jun-2024
https://doi.org/10.3390/ai5020041
Go KTokuda SNiiyama HOnishi TOgushi AMiyata A(2024)Evaluation of Prosodic Features Suitable for Conversational Agents Replying with a JokeJournal of Information Processing10.2197/ipsjjip.32.3532(35-40)Online publication date: 2024
https://doi.org/10.2197/ipsjjip.32.35
Yoo YJung JKang CKim HJun S(2024)Do Computers Have Gender Roles?: Investigating Users’ Gender Role Stereotyping of Anthropomorphized Voice AgentsArchives of Design Research10.15187/adr.2024.02.37.1.12337:1(123-136)Online publication date: 29-Feb-2024
https://doi.org/10.15187/adr.2024.02.37.1.123
Zhou ATsai WMen L(2024)Optimizing AI Social Chatbots for Relational Outcomes: The Effects of Profile Design, Communication Strategies, and Message FramingInternational Journal of Business Communication10.1177/23294884241229223Online publication date: 5-Feb-2024
https://doi.org/10.1177/23294884241229223
Rheu MDai YMeng JPeng W(2024)When a Chatbot Disappoints You: Expectancy Violation in Human-Chatbot Interaction in a Social Support ContextCommunication Research10.1177/00936502231221669Online publication date: 25-Jan-2024
https://doi.org/10.1177/00936502231221669
Seaborn KSawa YWatanabe M(2024)Coimagining the Future of Voice Assistants with Cultural SensitivityHuman Behavior and Emerging Technologies10.1155/2024/32387372024(1-21)Online publication date: 25-Mar-2024
https://doi.org/10.1155/2024/3238737
Aldeen MYoung JLiao SChang TCheng LCai HLuo XHu H(2024)End-Users Know Best: Identifying Undesired Behavior of Alexa Skills Through User Review AnalysisProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785178:3(1-28)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678517
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents