research-article

All Work and No Play?

Authors:

Muhammed Mas-ud Hussain,

Praveen Chandar,

Yasaman Khazaeni,

Marco Patricio Crasso,

Michael Muller,

N. Sadat Shami,

Werner GeyerAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 3, Pages 1 - 13

https://doi.org/10.1145/3173574.3173577

Published: 19 April 2018 Publication History

Abstract

Many conversational agents (CAs) are developed to answer users' questions in a specialized domain. In everyday use of CAs, user experience may extend beyond satisfying information needs to the enjoyment of conversations with CAs, some of which represent playful interactions. By studying a field deployment of a Human Resource chatbot, we report on users' interest areas in conversational interactions to inform the development of CAs. Through the lens of statistical modeling, we also highlight rich signals in conversational interactions for inferring user satisfaction with the instrumental usage and playful interactions with the agent. These signals can be utilized to develop agents that adapt functionality and interaction styles. By contrasting these signals, we shed light on the varying functions of conversational interactions. We discuss design implications for CAs, and directions for developing adaptive agents based on users' conversational behaviors.

Supplementary Material

MP4 File (pn1024.mp4)

Download
254.37 MB

References

[1]

John Langshaw Austin. 1975. How to do things with words. Oxford university press.

[2]

Niels Bernsen and Laila Dybkjær. 2004. Domain-oriented conversation with HC Andersen. Affective Dialogue Systems (2004), 142--153.

[3]

Yulong Bian, Chenglei Yang, Dongdong Guan, Sa Xiao, Fengqiang Gao, Chia Shen, and Xiangxu Meng. 2016. Effects of Pedagogical Agent's Personality and Emotional Feedback Strategy on Chinese Students' Learning Experiences and Performance: A Study Based on Virtual Tai Chi Training Studio. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 433--444.

Digital Library

[4]

Timothy Bickmore, Laura Pfeifer, and Daniel Schulman. 2011. Relational agents improve engagement and learning in science museum visitors. In International Workshop on Intelligent Virtual Agents. Springer, 55--67.

Digital Library

[5]

Timothy W Bickmore, Laura M Pfeifer, and Brian W Jack. 2009. Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1265--1274.

Digital Library

[6]

Dan Bohus and Alexander I Rudnicky. 2003. RavenClaw: Dialog management using hierarchical task decomposition and an expectation agenda. (2003).

[7]

Susan E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. (1990).

[8]

Hendrik Buschmeier and Stefan Kopp. 2011. Towards conversational agents that attend to and adapt to communicative user feedback. In Intelligent Virtual Agents. Springer, 169--182.

Digital Library

[9]

John M Carroll and John C Thomas. 1988. Fun. ACM SIGCHI Bulletin 19, 3 (1988), 21--24.

Digital Library

[10]

Justine Cassell. 2001. Embodied conversational agents: representation and intelligence in user interfaces. AI magazine 22, 4 (2001), 67.

Digital Library

[11]

Justine Cassell and Timothy Bickmore. 2003. Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. User modeling and user-adapted interaction 13, 1 (2003), 89--132.

Digital Library

[12]

Praveen Chandar, Yasaman Khazaeni, Matthew Davis, Michael Muller, Marco Crasso, Q Vera Liao, N Sadat Shami, and Werner Geyer. 2017. Leveraging Conversational Systems to Assists New Hires During Onboarding. In IFIP Conference on Human-Computer Interaction. Springer, 381--391.

[13]

Mark Coeckelbergh. 2011. You, robot: on the linguistic construction of artificial others. AI & society 26, 1 (2011), 61--69.

Digital Library

[14]

Mark G Core and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In AAAI fall symposium on communicative action in humans and machines, Vol. 56. Boston, MA.

[15]

Doris M Dehn and Susanne Van Mulken. 2000. The impact of animated interface agents: a review of empirical research. International journal of human-computer studies 52, 1 (2000), 1--22.

Digital Library

[16]

Nicholas Epley, Adam Waytz, and John T Cacioppo. 2007. On seeing human: a three-factor theory of anthropomorphism. Psychological review 114, 4 (2007), 864.

[17]

Dan Fletcher. 2010. The 50 Worst Inventions: Microsoft Bob. TIME. (27 May 2010).

[18]

Eric Gilbert. 2012. Phrases that signal workplace hierarchy. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1037--1046.

Digital Library

[19]

Joakim Gustafson and Linda Bell. 2000. Speech technology on trial: Experiences from the August system. Natural Language Engineering 6, 3--4 (2000), 273--286.

Digital Library

[20]

John Heritage and John Maxwell Atkinson. 1984. Structures of social action: Studies in conversation analysis. Cambridge University Press.

[21]

Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 506--516.

Digital Library

[22]

Diane Kelly and Jaime Teevan. 2003. Implicit feedback for inferring user preference: a bibliography. In ACM SIGIR Forum, Vol. 37. ACM, 18--28.

Digital Library

[23]

Alfred Kobsa and Wolfgang Wahlster. 1989. User models in dialog systems. Springer.

Digital Library

[24]

Stefan Kopp, Lars Gesellensetter, Nicole C Krämer, and Ipke Wachsmuth. 2005. A conversational agent as museum guide--design and evaluation of a real-world application. In International Workshop on Intelligent Virtual Agents. Springer, 329--343.

Digital Library

[25]

Min Kyung Lee, Sara Kiesler, and Jodi Forlizzi. 2010. Receptionist or information kiosk: How do people talk with a robot?. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 31--40.

Digital Library

[26]

Namseok Lee, Hochul Shin, and S Shyam Sundar. 2011. Utilitarian vs. hedonic robots: role of parasocial tendency and anthropomorphism in shaping user attitudes. In Human-Robot Interaction (HRI), 2011 6th ACM/IEEE International Conference on. IEEE, 183--184.

Digital Library

[27]

Q Vera Liao, Matthew Davis, Werner Geyer, Michael Muller, and N Sadat Shami. 2016. What Can You Do?: Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 264--275.

Digital Library

[28]

Ewa Luger and Abigail Sellen. 2016. Like Having a Really Bad PA: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5286--5297.

Digital Library

[29]

Chris Matyszczyk. 2012. Apple's Siri wrong 38 percent of the time in test. CNET. (30 June 2012).

[30]

Tanushree Mitra and Eric Gilbert. 2014. The language that gets people to give: Phrases that predict success on kickstarter. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 49--61.

Digital Library

[31]

Robert J. Moore, Rafah A. Hosn, and Ashima Arora. 2016. The Machinery of Natural Conversation and the Design of Conversational Machines. In American Sociological Association annual meeting.

[32]

Amy Ogan, Samantha Finkelstein, Elijah Mayfield, Claudia D'Adamo, Noboru Matsuda, and Justine Cassell. 2012a. Oh dear stacy!: social interaction, elaboration, and learning with teachable agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 39--48.

Digital Library

[33]

Amy Ogan, Samantha L Finkelstein, Erin Walker, Ryan Carlson, and Justine Cassell. 2012b. Rudeness and Rapport: Insults and Learning Gains in Peer Tutoring. In ITS. Springer, 11--21.

Digital Library

[34]

Christopher Peters, Stylianos Asteriadis, and Kostas Karpouzis. 2010. Investigating shared attention with a virtual agent using a gaze-based interface. Journal on Multimodal User Interfaces 3, 1 (2010), 119--130.

[35]

Martin Porcheron, Joel E Fischer, and Sarah Sharples. 2017. Do Animals Have A ccents?: Talking with Agents in Multi-Party Conversation. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 207--219.

Digital Library

[36]

Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval. ACM, 117--126.

Digital Library

[37]

Susan Robinson, Antonio Roque, and David R Traum. 2010. Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue. In LREC.

[38]

Susan Robinson, David R Traum, Midhun Ittycheriah, and Joe Henderer. 2008. What would you Ask a conversational Agent? Observations of Human-Agent Dialogues in a Museum Setting. In LREC.

[39]

Maha Salem, Friederike Eyssel, Katharina Rohlfing, Stefan Kopp, and Frank Joublin. 2013. To err is human (-like): Effects of robot gesture on perceived anthropomorphism and likability. International Journal of Social Robotics 5, 3 (2013), 313--323.

[40]

Jyotirmay Sanghvi, Ginevra Castellano, Iolanda Leite, André Pereira, Peter W McOwan, and Ana Paiva. 2011. Automatic analysis of affective postures and body motion to detect engagement with a game companion. In Human-Robot Interaction (HRI), 2011 6th ACM/IEEE International Conference on. IEEE, 305--311.

Digital Library

[41]

John R Searle. 1976. A classification of illocutionary acts. Language in society 5, 01 (1976), 1--23.

[42]

John R Searle, Ferenc Kiefer, and Manfred Bierwisch. 1980. Speech act theory and pragmatics. Vol. 10. Springer.

[43]

Alexander Serenko. 2008. A model of user adoption of interface agents for email notification. Interacting with Computers 20, 4--5 (2008), 461--472.

Digital Library

[44]

Nicole Shechtman and Leonard M Horowitz. 2003. Media inequality in conversation: how people behave differently when interacting with computers and people. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 281--288.

Digital Library

[45]

Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42--61.

Digital Library

[46]

Yang Song and Li-wei He. 2010. Optimal rare query suggestion with implicit user feedback. In Proceedings of the 19th international conference on World wide web. ACM, 901--910.

Digital Library

[47]

Luke Swartz. 2003. Why people hate the paperclip: Labels, appearance, behavior, and social responses to user interface agents. Ph.D. Dissertation. Stanford University Palo Alto, CA.

[48]

Daniel Szafir and Bilge Mutlu. 2012. Pay attention!: designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 11--20.

Digital Library

[49]

Jaime Teevan, Susan T Dumais, and Eric Horvitz. 2005. Personalizing search via automated analysis of interests and activities. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 449--456.

Digital Library

[50]

Robert Trappl. 2013. Your Virtual Butler. Springer.

Digital Library

[51]

Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 271--280.

Digital Library

[52]

William Yang Wang, Samantha Finkelstein, Amy Ogan, Alan W Black, and Justine Cassell. 2012. Love ya, jerkface: using sparse log-linear models to build positive (and impolite) relationships with teens. In Proceedings of the 13th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, 20--29.

Digital Library

[53]

Jane Webster. 1988. Making computer tasks at work more playful: Implications for systems analysts and designers. In Proceedings of the ACM SIGCPR conference on Management of information systems personnel. ACM, 78--87.

Digital Library

[54]

Jane Webster and Joseph J Martocchio. 1992. Microcomputer playfulness: Development of a measure with workplace implications. MIS quarterly (1992), 201--226.

Digital Library

[55]

Joseph Weizenbaum. 1966. ELIZA computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36--45.

Digital Library

[56]

Jun Xiao, John Stasko, and Richard Catrambone. 2004. An empirical study of the effect of agent competence on user performance and perception. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 1. IEEE Computer Society, 178--185.

Digital Library

[57]

Jun Xiao, John Stasko, and Richard Catrambone. 2007. The role of choice and customization on users' interaction with embodied conversational agents: effects on perception and performance. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 1293--1302.

Digital Library

[58]

ChengXiang Zhai and John Lafferty. 2006. A risk minimization framework for information retrieval. Information Processing & Management 42, 1 (2006), 31--55.

Digital Library

[59]

Ran Zhao, Tanmay Sinha, Alan W Black, and Justine Cassell. 2016. Automatic Recognition of Conversational Strategies in the Service of a Socially-Aware Dialog System.

Cited By

Go KTokuda SNiiyama HOnishi TOgushi AMiyata A(2024)Evaluation of Prosodic Features Suitable for Conversational Agents Replying with a JokeJournal of Information Processing10.2197/ipsjjip.32.3532(35-40)Online publication date: 2024
https://doi.org/10.2197/ipsjjip.32.35
Zhang RLi HChen ALiu ZLee Y(2024)AI Privacy in Context: A Comparative Study of Public and Institutional Discourse on Conversational AI Privacy in the US and Chinese Social MediaSocial Media + Society10.1177/2056305124129084510:4Online publication date: 28-Oct-2024
https://doi.org/10.1177/20563051241290845
Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Show More Cited By

Index Terms

All Work and No Play?
1. Human-centered computing

Recommendations

Evaluating and Informing the Design of Chatbots
DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not ...
Conversational Agents Replying with a Manzai-style Joke
OzCHI '21: Proceedings of the 33rd Australian Conference on Human-Computer Interaction

Automated conversational agents are becoming popular in various everyday contexts. In order to fulfill a more important role in human society, people would need to feel a sense of familiarity with such agents. To achieve this, we focus on humor, which ...
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and Design
Abstract
In this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

77
Total Citations
View Citations
3,048
Total Downloads

Downloads (Last 12 months)246
Downloads (Last 6 weeks)14

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Go KTokuda SNiiyama HOnishi TOgushi AMiyata A(2024)Evaluation of Prosodic Features Suitable for Conversational Agents Replying with a JokeJournal of Information Processing10.2197/ipsjjip.32.3532(35-40)Online publication date: 2024
https://doi.org/10.2197/ipsjjip.32.35
Zhang RLi HChen ALiu ZLee Y(2024)AI Privacy in Context: A Comparative Study of Public and Institutional Discourse on Conversational AI Privacy in the US and Chinese Social MediaSocial Media + Society10.1177/2056305124129084510:4Online publication date: 28-Oct-2024
https://doi.org/10.1177/20563051241290845
Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Liu YTing ILiberona DHsu M(2024)Personalized User Interface Design For Psychological Counseling Chatbots: A Pilot StudyProceedings of the 2024 11th Multidisciplinary International Social Networks Conference10.1145/3675669.3675683(23-28)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3675669.3675683
Gautam A(2024)Reconfiguring Participatory Design to Resist AI RealismProceedings of the Participatory Design Conference 2024: Exploratory Papers and Workshops - Volume 210.1145/3661455.3669867(31-36)Online publication date: 11-Aug-2024
https://dl.acm.org/doi/10.1145/3661455.3669867
Belghith YMahdavi Goloujeh AMagerko BLong DMcklin TRoberts J(2024)Testing, Socializing, Exploring: Characterizing Middle Schoolers’ Approaches to and Conceptions of ChatGPTProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642332(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642332
Hoque MMahfuz AKindi MHassan N(2024)Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and PerspectivesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642007(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642007
Pezenka IAunimo LJanous GDobrowsky D(2024)Emotionality in Task-Oriented Chatbots – The Effect of Emotion Expression on Chatbot PerceptionCommunication Studies10.1080/10510974.2024.236325975:6(825-843)Online publication date: 6-Jun-2024
https://doi.org/10.1080/10510974.2024.2363259
Chen KShao ABurapacheep JLi Y(2024)Conversational AI and equity through assessing GPT-3’s communication with diverse social groups on contentious topicsScientific Reports10.1038/s41598-024-51969-w14:1Online publication date: 18-Jan-2024
https://doi.org/10.1038/s41598-024-51969-w
Hauptman AFlathmann CMcNeese N(2024)Adapting to the human: A systematic review of a decade of human factors research on adaptive autonomyApplied Ergonomics10.1016/j.apergo.2024.104336120(104336)Online publication date: Oct-2024
https://doi.org/10.1016/j.apergo.2024.104336
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten