research-article

Public Access

A Systematic Cross-Corpus Analysis of Human Reactions to Robot Conversational Failures

Authors:

Dimosthenis Kontogiorgos,

Joakim Gustafson,

Mohammad SoleymaniAuthors Info & Claims

ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

Pages 112 - 120

https://doi.org/10.1145/3462244.3479887

Published: 18 October 2021 Publication History

All formats PDF

Abstract

In this paper, we analyze multimodal behavioral responses to robot failures across different tasks. Two multimodal datasets are examined in which humans interact with guided-task robots in task-oriented dialogues. In both datasets, the robots simulated failures of conversational breakdown and miscommunication typically observed in human-robot interactions. We closely examine human reactions to these failures looking at facial and acoustic features. Our analyses identify the significant behavioral features for automatic detection of such failures in interaction. We also examine human responses to different types of robot failures and if failures occurred early or late in the interaction cause variation in the responses. Our findings indicate that several nonverbal behaviors are consistently present in responses to robots’ failures, e.g., gaze and speech prosody, whereas, linguistic features appear to be task-dependent. We discuss how these findings may generalize to other tasks, and how autonomous robots may identify opportunities to detect and recover from failures in interactions with humans.

Supplementary Material

MP4 File (1357_Kontogiorgos.mp4)

In this video, we present our paper "A Systematic Cross-Corpus Analysis of Robot Conversational Failures" by Dimosthenis Kontogiorgos, Minh Tran, Joakim Gustafson and Mohammad Soleymani.

Download
19.26 MB

References

[1]

Sean Andrist, Dan Bohus, Ece Kamar, and Eric Horvitz. 2017. What went wrong and why? diagnosing situated interaction failures in the wild. In International Conference on Social Robotics. Springer, 293–303.

[2]

Deepali Aneja, Daniel McDuff, and Mary Czerwinski. 2020. Conversational Error Analysis in Human-Agent Interaction. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents. 1–8.

Digital Library

[3]

Ron Artstein, Jill Boberg, Alesia Gainer, Jonathan Gratch, Emmanuel Johnson, Anton Leuski, Gale Lucas, and David Traum. 2018. The Niki and Julie Corpus: collaborative multimodal dialogues between humans, robots, and virtual agents. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

[4]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1–10.

[5]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1–48. https://doi.org/10.18637/jss.v067.i01

[6]

Gustavo EAPA Batista, Ronaldo C Prati, and Maria Carolina Monard. 2004. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD explorations newsletter 6, 1 (2004), 20–29.

[7]

Dan Bohus. 2007. Error awareness and recovery in conversational spoken language interfaces. Technical Report. CARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF COMPUTER SCIENCE.

[8]

Dan Bohus and Alexander Rudnicky. 2005. Sorry and I Didn’t Catch That!-An Investigation of Non-understanding Errors and Recovery Strategies. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue. 128–143.

[9]

Judee K Burgoon and Jerold L Hale. 1988. Nonverbal expectancy violations: Model elaboration and application to immediacy behaviors. Communications Monographs 55, 1 (1988), 58–79.

[10]

Dito Eka Cahya, Rahul Ramakrishnan, and Manuel Giuliani. 2019. Static and Temporal Differences in Social Signals Between Error-Free and Erroneous Situations in Human-Robot Collaboration. In International Conference on Social Robotics. Springer, 189–199.

[11]

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794.

Digital Library

[12]

Herbert H Clark and Susan E Brennan. 1991. Grounding in communication.(1991).

[13]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171–4186.

[14]

Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia. 1459–1462.

Digital Library

[15]

Russell H Fazio and Michael A Olson. 2007. Attitudes: Foundations, functions, and consequences. The handbook of social psychology(2007), 123–145.

[16]

Rebecca Flook, Anas Shrinah, Luc Wijnen, Kerstin Eder, Chris Melhuish, and Séverin Lemaignan. 2019. On the impact of different types of errors on trust in human-robot interaction: Are laboratory-based HRI experiments trustworthy?Interaction Studies 20, 3 (2019), 455–486.

[17]

Raphaela Gehle, Karola Pitsch, Timo Dankert, and Sebastian Wrede. 2015. Effects of a robot’s unexpected reactions in robot-to-group interactions. (2015).

[18]

Manuel Giuliani, Nicole Mirnig, Gerald Stollnberger, Susanne Stadler, Roland Buchner, and Manfred Tscheligi. 2015. Systematic analysis of video data from different human–robot interaction studies: a categorization of social signals during error situations. Frontiers in psychology 6 (2015), 931.

[19]

Joakim Gustafson and Linda Bell. 2000. Speech technology on trial: Experiences from the August system. Natural Language Engineering 6, 3-4 (2000), 273–286.

Digital Library

[20]

Cory J Hayes, Maryam Moosaei, and Laurel D Riek. 2016. Exploring implicit human responses to robot mistakes in a learning from demonstration task. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 246–252.

Digital Library

[21]

Julia Hirschberg, Diane Litman, and Marc Swerts. 2004. Prosodic and other cues to speech recognition failures. Speech communication 43, 1-2 (2004), 155–175.

[22]

Shanee Honig and Tal Oron-Gilad. 2018. Understanding and resolving failures in human-robot interaction: Literature review and model development. Frontiers in psychology 9 (2018), 861.

[23]

Jeesun Kim and Chris Davis. 2016. The Consistency and Stability of Acoustic and Visual Cues for Different Prosodic Attitudes. In INTERSPEECH. 57–61.

[24]

Dimosthenis Kontogiorgos, Andre Pereira, and Joakim Gustafson. 2021. Grounding behaviours with conversational interfaces: effects of embodiment and failures. Journal on Multimodal User Interfaces(2021), 1–16.

[25]

Dimosthenis Kontogiorgos, Andre Pereira, Boran Sahindal, Sanne van Waveren, and Joakim Gustafson. 2020. Behavioural responses to robot conversational failures. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. 53–62.

Digital Library

[26]

Dimosthenis Kontogiorgos, Sanne van Waveren, Olle Wallberg, Andre Pereira, Iolanda Leite, and Joakim Gustafson. 2020. Embodiment Effects in Interactions with Failing Robots. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[27]

Jacqueline Marie Kory-Westlund. 2019. Relational ai: Creating long-term interpersonal interaction, rapport, and relationships with social robots. Ph.D. Dissertation. Massachusetts Institute of Technology.

[28]

Diane Litman, Marilyn Walker, and Michael S Kearns. 1999. Automatic detection of poor speech recognition at the dialogue level. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. 309–316.

Digital Library

[29]

Raveesh Meena, José Lopes, Gabriel Skantze, and Joakim Gustafson. 2015. Automatic detection of miscommunication in spoken dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 354–363.

[30]

Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. 72–78.

Digital Library

[31]

James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.

[32]

R Core Team. 2020. R: A Language and Environment for Statistical Computing. (2020). https://www.R-project.org/

[33]

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3973–3983.

[34]

Albert Rilliard, Donna Erickson, Takaaki Shochi, and João Antônio de Moraes. 2013. Social face to face communication-American English attitudinal prosody. In INTERSPEECH. 1648–1652.

[35]

Emanuel A Schegloff and Harvey Sacks. 1973. Opening up closings. Semiotica 8, 4 (1973), 289–327.

[36]

Chao Shi, Masahiro Shiomi, Christian Smith, Takayuki Kanda, and Hiroshi Ishiguro. 2013. A Model of Distributional Handing Interaction for a Mobile Robot. In Robotics: science and systems. 24–28.

[37]

Elaine Schaertl Short, Mai Lee Chang, and Andrea Thomaz. 2018. Detecting contingency for HRI in open-world environments. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. 425–433.

Digital Library

[38]

Gabriel Skantze. 2005. Exploring human error recovery strategies: Implications for spoken dialogue systems. Speech Communication 45, 3 (2005), 325–341.

[39]

Gabriel Skantze and Jens Edlund. 2004. Early error detection on word level. In COST278 and ISCA Tutorial and Research Workshop (ITRW) on Robustness Issues in Conversational Interaction.

[40]

David R Traum and Peter A Heeman. 1996. Utterance units in spoken dialogue. In Workshop on Dialogue Processing in Spoken Language Systems. Springer, 125–140.

Digital Library

[41]

Pauline Trung, Manuel Giuliani, Michael Miksch, Gerald Stollnberger, Susanne Stadler, Nicole Mirnig, and Manfred Tscheligi. 2017. Head and shoulders: automatic error detection in human-robot interaction. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 181–188.

Digital Library

[42]

Marilyn Walker, Jerry Wright, and Irene Langkilde. 2000. Using natural language processing and discourse features to identify understanding errors in a spoken dialogue system. In Proceedings of the 17th international conference on machine learning. 1111–1118.

Cited By

Stiber MGrollman DBroadbent EJu WSoh HWilliams T(2024)Flexible Robot Error Detection Using Natural Human Responses for Effective HRICompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638365(148-150)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638365
Stiber MSpitale MGunes HHuang CGrollman DBroadbent EJu WSoh HWilliams T(2024)Social Signal Modeling in Human-Robot InteractionCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638163(1358-1360)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638163
Axelsson MSpitale MGunes HGrollman DBroadbent EJu WSoh HWilliams T(2024)"Oh, Sorry, I Think I Interrupted You": Designing Repair Strategies for Robotic Longitudinal Well-being CoachingProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634948(13-22)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634948
Show More Cited By

Recommendations

Behavioural Responses to Robot Conversational Failures
HRI '20: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction

Humans and robots will increasingly collaborate in domestic environments which will cause users to encounter more failures in interactions. Robots should be able to infer conversational failures by detecting human users' behavioural and social signals. ...
Towards a Conversational Corpus for Human-Robot Conversations
HRI '17: Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction

Conversational corpora based on human-human dialogues have often been used for training of data-driven dialogue systems. However, human-human conversations might not be the optimal inputs for machine learning training aims used in HRI. This paper ...
Improving Human-Robot Conversational Groups

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

October 2021

876 pages

ISBN:9781450384810

DOI:10.1145/3462244

Editors:
Zakia Hammal
Carnegie Mellon University
,
Carlos Busso
University of Texas at Dallas
,
Catherine Pelachaud
CNRS - ISIR, Sorbonne University
,
Sharon Oviatt
Monash University
,
Albert Ali Salah
Utrecht University and Boğaziçi University
,
Guoying Zhao
University of Oulu

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

U.S. Army Research Office
Swedish Foundation for Strategic Research

Conference

ICMI '21

Sponsor:

SIGCHI

ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 18 - 22, 2021

QC, Montréal, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
617
Total Downloads

Downloads (Last 12 months)275
Downloads (Last 6 weeks)34

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Stiber MGrollman DBroadbent EJu WSoh HWilliams T(2024)Flexible Robot Error Detection Using Natural Human Responses for Effective HRICompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638365(148-150)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638365
Stiber MSpitale MGunes HHuang CGrollman DBroadbent EJu WSoh HWilliams T(2024)Social Signal Modeling in Human-Robot InteractionCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638163(1358-1360)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638163
Axelsson MSpitale MGunes HGrollman DBroadbent EJu WSoh HWilliams T(2024)"Oh, Sorry, I Think I Interrupted You": Designing Repair Strategies for Robotic Longitudinal Well-being CoachingProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634948(13-22)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634948
Candon KChen JKim YHsu ZTsoi NVázquez MAgmon NAn BRicci AYeoh W(2023)Nonverbal Human Signals Can Help Autonomous Agents Infer Human Preferences for Their BehaviorProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598652(307-316)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598652
Stiber MTaylor RHuang CCastellano GRiek LCakmak MLeite I(2023)On Using Social Signals to Enable Flexible Error-Aware HRIProceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568162.3576990(222-230)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3568162.3576990
Spitale MAxelsson MKara NGunes H(2023)Longitudinal Evolution of Coachees’ Behavioural Responses to Interaction Ruptures in Robotic Positive Psychology Coaching2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309386(315-322)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309386
Bremers AParreira MFang XFriedman NRamirez-Aristizabal APabst ASpasojevic MKuniavsky MJu W(2023)The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10342442(11443-11450)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10342442
Tsubokura KIribe YKitaoka N(2023)Analysis of the relationship between user response to dialog breakdown and personality traitsAdvanced Robotics10.1080/01691864.2023.227961038:4(246-255)Online publication date: 13-Nov-2023
https://doi.org/10.1080/01691864.2023.2279610
Matsumoto KSasayama MYoshida MKita KRen F(2022)Emotion Analysis and Dialogue Breakdown Detection in Dialogue of Chat Systems Based on Deep Neural NetworksElectronics10.3390/electronics1105069511:5(695)Online publication date: 24-Feb-2022
https://doi.org/10.3390/electronics11050695
Stiber MTaylor RHuang C(2022)Modeling Human Response to Robot Errors for Timely Error Detection2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS47612.2022.9981726(676-683)Online publication date: 23-Oct-2022
https://doi.org/10.1109/IROS47612.2022.9981726
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents