Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3319502.3374779acmconferencesArticle/Chapter ViewAbstractPublication PageshriConference Proceedingsconference-collections
research-article
Open access

Decision-Making for Bidirectional Communication in Sequential Human-Robot Collaborative Tasks

Published: 09 March 2020 Publication History
  • Get Citation Alerts
  • Abstract

    Communication is critical to collaboration; however, too much of it can degrade performance. Motivated by the need for effective use of a robot's communication modalities, in this work, we present a computational framework that decides if, when, and what to communicate during human-robot collaboration. The framework, titled CommPlan, consists of a model specification process and an execution-time POMDP planner. To address the challenge of collecting interaction data, the model specification process is hybrid : where part of the model is learned from data, while the remainder is manually specified. Given the model, the robot's decision-making is performed computationally during interaction and under partial observability of human's mental states. We implement CommPlan for a shared workspace task, in which the robot has multiple communication options and needs to reason within a short time. Through experiments with human participants, we confirm that CommPlan results in the effective use of communication capabilities and improves human-robot collaboration.

    Supplementary Material

    ZIP File (fp1063aux.zip)
    The supplemental material provides a video demonstration of CommPlan, our framework for generating a collaborative robot's action and communication policy, in the meal preparation task.
    MP4 File (p329-unhelkar.mp4)

    References

    [1]
    Henny Admoni and Brian Scassellati. 2017. Social Eye Gaze in Human-Robot Interaction: A Review. Journal of Human-Robot Interaction (JHRI), Vol. 6, 1 (2017), 25--63.
    [2]
    Sean Andrist, Micheline Ziadee, Halim Boukaram, Bilge Mutlu, and Majd Sakr. 2015. Effects of Culture on the Credibility of Robot Speech: A Comparison between English and Arabic. In International Conference on Human-Robot Interaction (HRI). ACM, 157--164.
    [3]
    Jacob Arkin, Rohan Paul, Daehyung Park, Subhro Roy, Nicholas Roy, and Thomas M. Howard. 2018. Real-Time Human-Robot Communication for Manipulation Tasks in Partially Observed Environments. In International Symposium on Experimental Robotics (ISER) .
    [4]
    Kent Bach and Robert M. Harnish. 1979. Linguistic Communication and Speech Acts. (1979).
    [5]
    Kim Baraka and Manuela M. Veloso. 2018. Mobile Service Robot State Revealing through Expressive Lights: Formalism, Design, and Evaluation. International Journal of Social Robotics (IJSR), Vol. 10, 1 (2018), 65--92.
    [6]
    Michael D. Basil. 2012. Multiple Resource Theory. Encyclopedia of the Sciences of Learning (2012), 2384--2385.
    [7]
    Craig Boutilier. 1996. Planning, Learning and Coordination in Multi-Agent Decision Processes. In Conference on Theoretical Aspects of Rationality and Knowledge. Morgan Kaufmann Publishers Inc., 195--210.
    [8]
    Frank Broz, Illah Nourbakhsh, and Reid Simmons. 2013. Planning for Human-Robot Interaction in Socially Situated Tasks. International Journal of Social Robotics (IJSR), Vol. 5, 2 (2013), 193--214.
    [9]
    Abhizna Butchibabu, Christopher Sparano-Huiban, Liz Sonenberg, and Julie Shah. 2016. Implicit Coordination Strategies for Effective Team Communication. Human Factors, Vol. 58, 4 (2016), 595--610.
    [10]
    Crystal Chao and Andrea Thomaz. 2016. Timed Petri Nets for Fluent Turn-Taking over Multimodal Interaction Resources in Human-Robot Collaboration. International Journal of Robotics Research (IJRR), Vol. 35, 11 (2016), 1330--1353.
    [11]
    Sonia Chernova and Andrea L. Thomaz. 2014. Robot Learning from Human Teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 8, 3 (2014), 1--121.
    [12]
    Herbert H. Clark, Susan E. Brennan, et almbox. 1991. Grounding in Communication. Perspectives on Socially Shared Cognition, Vol. 13, 1991 (1991), 127--149.
    [13]
    J. Michael Crant. 2000. Proactive Behavior in Organizations. Journal of Management, Vol. 26, 3 (2000), 435--462.
    [14]
    Sandra Devin and Rachid Alami. 2016. An Implemented Theory of Mind to Improve Human-Robot Shared Plans Execution. In International Conference on Human-Robot Interaction (HRI). ACM, 319--326.
    [15]
    Sandra Devin, Aurélie Clodic, and Rachid Alami. 2017. About Decisions during Human-Robot Shared Plan Achievement: Who should Act and How?. In International Conference on Social Robotics (ICSR). Springer, 453--463.
    [16]
    Barbara Di Eugenio, Pamela W. Jordan, Richmond H. Thomason, and Johanna D. Moore. 2000. The Agreement Process: An Empirical Investigation of Human-Human Computer-Mediated Collaborative Dialogs. International Journal of Human-Computer Studies, Vol. 53, 6 (2000), 1017--1076.
    [17]
    Myron A. Diftler, J.S. Mehling, Muhammad E. Abdallah, Nicolaus A. Radford, Lyndon B. Bridgwater, Adam M. Sanders, Roger Scott Askew, D. Marty Linn, John D. Yamokoski, F.A. Permenter, et almbox. 2011. Robonaut 2-the First Humanoid Robot in Space. In International Conference on Robotics and Automation (ICRA). IEEE, 2178--2183.
    [18]
    Anca D. Dragan, Kenton C.T. Lee, and Siddhartha S. Srinivasa. 2013. Legibility and Predictability of Robot Motion. In International Conference on Human-Robot Interaction (HRI). ACM, 301--308.
    [19]
    Michelangelo Fiore, Aurélie Clodic, and Rachid Alami. 2016. On Planning and Task Achievement Modalities for Human-Robot Collaboration. In Experimental Robotics. Springer, 293--306.
    [20]
    Kerstin Sophie Haring, David Silvera-Tawil, Yoshio Matsumoto, Mari Velonaki, and Katsumi Watanabe. 2014. Perception of an Android Robot in Japan and Australia: A Cross-Cultural Comparison. In International Conference on Social Robotics (ICSR). Springer, 166--175.
    [21]
    Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, and Jethro Tan. 2018. Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions. In International Conference on Robotics and Automation (ICRA). IEEE, 3774--3781.
    [22]
    Guy Hoffman. 2019. Evaluating Fluency in Human--Robot Collaboration. IEEE Transactions on Human-Machine Systems (THMS), Vol. 49, 3 (2019), 209--218.
    [23]
    Guy Hoffman and Cynthia Breazeal. 2007. Effects of Anticipatory Action on Human-Robot Teamwork Efficiency, Fluency, and Perception of Team. In International Conference on Human-Robot Interaction (HRI). ACM, 1--8.
    [24]
    Eric Horvitz and Johnson Apacible. 2003. Learning and Reasoning about Interruption. In International Conference on Multimodal Interfaces (ICMI). ACM, 20--27.
    [25]
    Mithun George Jacob, Yu-Ting Li, George A. Akingba, and Juan P. Wachs. 2013. Collaboration with a Robotic Scrub Nurse. Commun. ACM, Vol. 56, 5 (2013), 68--75.
    [26]
    Matthew Johnson, Catholijn Jonker, Birna Van Riemsdijk, Paul J. Feltovich, and Jeffrey M. Bradshaw. 2009. Joint Activity Testbed: Blocks World for Teams (BW4T). In International Workshop on Engineering Societies in the Agents World. Springer, 254--256.
    [27]
    Tobias Kaupp, Alexei Makarenko, and Hugh Durrant-Whyte. 2010. Human-Robot Communication for Collaborative Decision Making -- A Probabilistic Approach. Robotics and Autonomous Systems, Vol. 58, 5 (2010), 444 -- 456.
    [28]
    Ross A. Knepper, Christoforos I. Mavrogiannis, Julia Proft, and Claire Liang. 2017. Implicit Communication in a Joint Action. In International Conference on Human-Robot Interaction (HRI). ACM, 283--292.
    [29]
    Thibault Kruse, Patrizia Basili, Stefan Glasauer, and Alexandra Kirsch. 2012. Legible Robot Navigation in the Proximity of Moving Humans. In Advanced Robotics and its Social Impacts (ARSO). IEEE, 83--88.
    [30]
    Paul Lamere, Philip Kwok, Evandro Gouvêa, Bhiksha Raj, Rita Singh, William Walker, Manfred Warmuth, and Peter Wolf. 2003. The Carnegie Mellon University SPHINX-4 Speech Recognition System.
    [31]
    Przemyslaw A. Lasota and Julie A. Shah. 2017. A Multiple-Predictor Approach to Human Motion Prediction. In International Conference on Robotics and Automation (ICRA). IEEE, 2300--2307.
    [32]
    M Leonetti, L Iocchi, Anthony G Cohn, and Daniele Nardi. 2019. Adaptive Human-Aware Task Planning. In ICAPS Workshop on Planning and Robotics (PlanRob) .
    [33]
    Jing Li, Zhong Shen, Wen Yu Tian Xu, Walter Yu Hang Lam, Richard Tai Chiu Hsung, Edmond Ho Nang Pow, Kazuhiro Kosuge, and Zheng Wang. 2019. A Compact Dental Robotic System Using Soft Bracing Technique. IEEE Robotics and Automation Letters (RA-L), Vol. 4, 2 (2019), 1271--1278.
    [34]
    Sirui Li, Weixing Sun, and Tim Miller. 2015. Communication in Human-Agent Teams for Tasks with Joint Action. In Coordination, Organizations, Institutions, and Norms in Agent Systems (COIN). Springer, 224--241.
    [35]
    Jean MacMillan, Elliot E. Entin, and Daniel Serfaty. 2004. Communication Overhead: The Hidden Cost of Team Cognition. Team Cognition: Process and Performance at the Inter and Intra-Individual Level (2004).
    [36]
    Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to Parse Natural Language Commands to a Robot Control System. In International Symposium on Experimental Robotics (ISER). 403--415.
    [37]
    Nikolaos Mavridis. 2015. A Review of Verbal and Non-Verbal Human--Robot Interactive Communication. Robotics and Autonomous Systems, Vol. 63 (2015), 22--35.
    [38]
    Nikolaos Mavridis and Haiwei Dong. 2012. To Ask or To Sense? Planning to Integrate Speech and Sensori-motor Acts. In International Congress on Ultra Modern Telecommunications and Control Systems. IEEE, 227--233.
    [39]
    Nikolaos Mavridis, Marina-Selini Katsaiti, Silvia Naef, Abdullah Falasi, Abdulrahman Nuaimi, Hamdan Araifi, and Ahmed Kitbi. 2012. Opinions and Attitudes toward Humanoid Robots in the Middle East . AI & society, Vol. 27, 4 (2012), 517--534.
    [40]
    Stefanos Nikolaidis, Minae Kwon, Jodi Forlizzi, and Siddhartha Srinivasa. 2018. Planning with Verbal Communication for Human-Robot Collaboration. ACM Transactions on Human-Robot Interaction (THRI), Vol. 7, 3 (2018), 22.
    [41]
    Stefanos Nikolaidis and Julie Shah. 2013. Human-Robot Cross-Training: Computational Formulation, Modeling and Evaluation of a Human Team Training Strategy. In International Conference on Human-Robot Interaction (HRI). ACM, 33--40.
    [42]
    Frans A Oliehoek, Christopher Amato, et almbox. 2016. A Concise Introduction to Decentralized POMDPs. Vol. 1. Springer.
    [43]
    Steven Pinker, Martin A Nowak, and James J Lee. 2008. The Logic of Indirect Speech. Proceedings of the National Academy of Sciences, Vol. 105, 3 (2008), 833--838.
    [44]
    Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: An Open-source Robot Operating System. In ICRA Workshop on Open Source Software, Vol. 3. 5.
    [45]
    Aditi Ramachandran, Chien-Ming Huang, and Brian Scassellati. 2017. Give Me a Break!: Personalized Timing Strategies to Promote Learning in Robot-Child Tutoring. In International Conference on Human-Robot Interaction (HRI). ACM, 146--155.
    [46]
    E Ratner, D Hadfield-Mennell, and A Dragan. 2018. Simplifying Reward Design through Divide-and-Conquer. In Robotics: Science and Systems (RSS) .
    [47]
    Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. 2017. Active Preference-Based Learning of Reward Functions. In Robotics: Science and Systems (RSS) .
    [48]
    Kristin E. Schaefer, Edward R. Straub, Jessie Y.C. Chen, Joe Putney, and Arthur W. Evans III. 2017. Communicating Intent to Develop Shared Situation Awareness and Engender Trust in Human-Agent Teams. Cognitive Systems Research, Vol. 46 (2017), 26--39.
    [49]
    Antonio Sgorbissa, Irena Papadopoulos, Barbara Bruno, Christina Koulouglioti, and Carmine Recchiuto. 2018. Encoding Guidelines for a Culturally Competent Robot for Elderly Care. In International Conference on Intelligent Robots and Systems (IROS). IEEE, 1988--1995.
    [50]
    Ankit Shah, Pritish Kamath, Shen Li, and Julie A. Shah. 2018. Bayesian Inference of Temporal Task Specifications from Demonstrations. In Advances in Neural Information Processing Systems (NeurIPS). 3804--3813.
    [51]
    Julie Shah, James Wiken, Brian Williams, and Cynthia Breazeal. 2011. Improved Human-Robot Team Performance using Chaski, a Human-Inspired Plan Execution System. In International Conference on Human-Robot Interaction (HRI). ACM, 29--36.
    [52]
    Aaron St. Clair and Maja Mataric. 2015. How Robot Verbal Feedback can Improve Team Performance in Human-Robot Task Collaborations. In International Conference on Human-Robot Interaction (HRI). ACM, 213--220.
    [53]
    Daniel Szafir, Bilge Mutlu, and Terrence Fong. 2014. Communication of Intent in Assistive Free Flyers. In International Conference on Human-Robot Interaction (HRI). ACM, 358--365.
    [54]
    Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. 2011. Approaching the Symbol Grounding Problem with Probabilistic Graphical Models. AI magazine, Vol. 32, 4 (2011), 64--76.
    [55]
    Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, and Nicholas Roy. 2012. Toward Information Theoretic Human-Robot Dialog. In Robotics: Science and Systems (RSS) .
    [56]
    Andrea Thomaz, Guy Hoffman, Maya Cakmak, et almbox. 2016. Computational Human-Robot Interaction. Foundations and Trends® in Robotics, Vol. 4, 2--3 (2016), 105--223.
    [57]
    Michael Tomasello. 2010. Origins of Human Communication .MIT Press.
    [58]
    Vaibhav V. Unhelkar*, Shen Li*, and Julie A. Shah. 2019. Semi-Supervised Learning of Decision-Making Models for Human-Robot Collaboration. In Conference on Robot Learning (CoRL) . Asterisks denote equal contribution.
    [59]
    Vaibhav V. Unhelkar and Julie A. Shah. 2019. Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior. In AAAI Conference on Artificial Intelligence .
    [60]
    Vaibhav V. Unhelkar, X. Jessie Yang, and Julie A. Shah. 2017. Challenges for Communication Decision-Making in Sequential Human-Robot Collaborative Tasks. In Workshop on Mathematical Models, Algorithms, and Human-Robot Interaction at RSS .
    [61]
    Ning Wang, David V. Pynadath, and Susan G. Hill. 2016. Trust Calibration within a Human-Robot Team: Comparing Automatically Generated Explanations. In International Conference on Human-Robot Interaction (HRI). ACM, 109--116.
    [62]
    Changyun Wei, Koen V. Hindriks, and Catholijn M. Jonker. 2014. The Role of Communication in Coordination Protocols for Cooperative Robot Teams. In International Conference on Agents and Artificial Intelligence. 28--39.
    [63]
    David Whitney, Eric Rosen, James MacGlashan, Lawson L.S. Wong, and Stefanie Tellex. 2017. Reducing Errors in Object-Fetching Interactions through Social Feedback. In International Conference on Robotics and Automation (ICRA). IEEE.
    [64]
    Tom Williams, Matthew Bussing, Sebastian Cabrol, Elizabeth Boyle, and Nhan Tran. 2019. Mixed Reality Deictic Gesture for Multi-Modal Robot Communication. In International Conference on Human-Robot Interaction (HRI). ACM, 191--201.
    [65]
    Ping Xuan, Victor Lesser, and Shlomo Zilberstein. 2004. Modeling Cooperative Multiagent Problem Solving as Decentralized Decision Processes. Submitted for Publication to Autonomous Agents and Multi-Agent Systems (2004).
    [66]
    Nan Ye, Adhiraj Somani, David Hsu, and Wee Sun Lee. 2017. DESPOT: Online POMDP Planning with Regularization. Journal of Artificial Intelligence Research (JAIR), Vol. 58 (2017), 231--266.
    [67]
    Andrea Maria Zanchettin, Nicola Maria Ceriani, Paolo Rocco, Hao Ding, and Björn Matthias. 2015. Safety in Human-Robot Collaborative Manufacturing Environments: Metrics and Control. IEEE Transactions on Automation Science and Engineering (T-ASE), Vol. 13, 2 (2015), 882--893.
    [68]
    Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. 2008. Maximum Entropy Inverse Reinforcement Learning. In AAAI Conference on Artificial Intelligence, Vol. 8. 1433--1438.

    Cited By

    View all
    • (2024)Toward Competent Robot Apprentices: Enabling Proactive Troubleshooting in Collaborative RobotsMachines10.3390/machines1201007312:1(73)Online publication date: 18-Jan-2024
    • (2024)A Multi-Robot Architecture Framework for Effective Robot Teammates in Mixed-Initiative TeamsProceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction10.1145/3648536.3648545(74-82)Online publication date: 9-Mar-2024
    • (2024)Interactively Explaining Robot Policies to Humans in Integrated Virtual and Physical Training EnvironmentsCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640656(847-851)Online publication date: 11-Mar-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HRI '20: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction
    March 2020
    690 pages
    ISBN:9781450367462
    DOI:10.1145/3319502
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 March 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. human-in-the-loop learning
    2. human-robot communication
    3. planning under uncertainty

    Qualifiers

    • Research-article

    Conference

    HRI '20
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 268 of 1,124 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)560
    • Downloads (Last 6 weeks)45
    Reflects downloads up to 11 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Toward Competent Robot Apprentices: Enabling Proactive Troubleshooting in Collaborative RobotsMachines10.3390/machines1201007312:1(73)Online publication date: 18-Jan-2024
    • (2024)A Multi-Robot Architecture Framework for Effective Robot Teammates in Mixed-Initiative TeamsProceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction10.1145/3648536.3648545(74-82)Online publication date: 9-Mar-2024
    • (2024)Interactively Explaining Robot Policies to Humans in Integrated Virtual and Physical Training EnvironmentsCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640656(847-851)Online publication date: 11-Mar-2024
    • (2024)Exploring Preferences in Human-Robot Navigation Plan Proposal RepresentationCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640627(369-373)Online publication date: 11-Mar-2024
    • (2024)PoseTron: Enabling Close-Proximity Human-Robot Collaboration Through Multi-human Motion PredictionProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3635006(830-839)Online publication date: 11-Mar-2024
    • (2024)Shared Autonomy of a Robotic Manipulator for Grasping Under Human Intent Uncertainty Using POMDPsIEEE Transactions on Robotics10.1109/TRO.2023.333463140(332-350)Online publication date: 2024
    • (2023)Nonverbal Human Signals Can Help Autonomous Agents Infer Human Preferences for Their BehaviorProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598652(307-316)Online publication date: 30-May-2023
    • (2023)A Literature Review of Human–AI Synergy in Decision Making: From the Perspective of Affordance Actualization TheorySystems10.3390/systems1109044211:9(442)Online publication date: 25-Aug-2023
    • (2023)PrISM-TrackerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35695046:4(1-27)Online publication date: 11-Jan-2023
    • (2023)Towards Robot Learning from Spoken LanguageCompanion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568294.3580053(112-116)Online publication date: 13-Mar-2023
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media