research-article

Two people walk into a bar: dynamic multi-party social interaction with a robot agent

Authors:

Mary Ellen Foster,

Andre Gaschler,

Manuel Giuliani,

Maria Pateraki,

Ronald P.A. PetrickAuthors Info & Claims

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Pages 3 - 10

https://doi.org/10.1145/2388676.2388680

Published: 22 October 2012 Publication History

Abstract

We introduce a humanoid robot bartender that is capable of dealing with multiple customers in a dynamic, multi-party social setting. The robot system incorporates state-of-the-art components for computer vision, linguistic processing, state management, high-level reasoning, and robot control. In a user evaluation, 31 participants interacted with the bartender in a range of social situations. Most customers successfully obtained a drink from the bartender in all scenarios, and the factors that had the greatest impact on subjective satisfaction were task success and dialogue efficiency.

Supplementary Material

JPG File (fp131.jpg)

Download
16.30 KB

suppl.mov (fp131.mp4)

Supplemental video

Download
9.82 MB

References

[1]

A. Argyros and M. Lourakis. 3D tracking of skin-colored regions by a moving stereoscopic observer. Applied Optics, 43 (2): 366--378, Jan. 2004.

[2]

W. Bainbridge, J. Hart, E. Kim, and B. Scassellati. The benefits of interactions with physically present robots over video-displayed agents. International Journal of Social Robotics, 3: 41--52, 2011. 10.1007/s12369-010-0082-7.

[3]

H. Baltzakis and A. Argyros. Propagation of pixel hypotheses for multiple objects tracking. In Proceedings of ISVC 2009, Nov. 2009.

Digital Library

[4]

H. Baltzakis, M. Pateraki, and P. Trahanias. Visual tracking of hands, faces and facial features of multiple persons. Machine Vision and Applications, pages 1--17, 2012. 10.1007/s00138-012-0409-5.

[5]

C. Bartneck, D. Kulić, E. Croft, and S. Zoghbi. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1: 71--81, 2009. 10.1007/s12369-008-0001-3.

[6]

D. Bohus and E. Horvitz. Dialog in the open world: platform and applications. In Proceedings of ICMI-MLMI 2009, pages 31--38, Nov. 2009. 10.1145/1647314.1647323.

Digital Library

[7]

C. Breazeal. Socially intelligent robots. interactions, 12 (2): 19--22, 2005. 10.1145/1052438.1052455.

Digital Library

[8]

G. Castellano, I. Leite, A. Pereira, C. Martinho, A. Paiva, and P. W. McOwan. Affect recognition for interactive companions: challenges and design in real world scenarios. Journal on Multimodal User Interfaces, 3 (1): 89--98, 2010. 10.1007/s12193-009-0033-5.

[9]

K. Dautenhahn. Socially intelligent robots: dimensions of human-robot interaction. Philosophical Transactions of the Royal Society B: Biological Sciences, 362 (1480): 679--704, 2007. 10.1098/rstb.2006.2004.

[10]

R. E. Fikes and N. J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2: 189--208, 1971. 10.1016/0004-3702(71)90010-5.

[11]

M. Frampton and O. Lemon. Recent research advances in reinforcement learning in spoken dialogue systems. The Knowledge Engineering Review, 24 (4): 375--408, 2009. 10.1017/S0269888909990166.

Digital Library

[12]

S. S. Ge and M. J. Matarić. Preface. International Journal of Social Robotics, 1 (1): 1--2, 2009. 10.1007/s12369-008-0010-2.

[13]

M. Giuliani, M. E. Foster, A. Isard, C. Matheson, J. Oberlander, and A. Knoll. Situated reference in a hybrid human-robot interaction system. In Proceedings of INLG 2010, 2010.

Digital Library

[14]

T. Horf, R. Roller, and S. Wilske. MiCo: The robotic bartender for mini-cocktails. http://www.coli.uni-saarland.de/courses/lego-04/page.php?id=barkeeper, 2004.

[15]

A. Hunt and S. McGlashan. Speech recognition grammar specification version 1.0. W3C recommendation, W3C, Mar. 2004.

[16]

K. Huth. Wie man ein Bier bestellt. Master's thesis, Universität Bielefeld, 2011.

[17]

A. Isard and C. Matheson. Rhetorical structure for natural language generation in dialogue. In Proceedings of SemDial 2012, 2012.

[18]

A. Kapoor, W. Burleson, and R. W. Picard. Automatic prediction of frustration. International Journal of Human-Computer Studies, 65 (8): 724--736, 2007. 10.1016/j.ijhcs.2007.02.003.

Digital Library

[19]

S. Larsson and D. Traum. Information state and dialogue management in the TRINDI dialogue move engine toolkit. Natural Language Engineering, 6 (3&4): 323--340, 2000. 10.1017/S1351324900002539.

Digital Library

[20]

D. J. Litman and S. Pan. Designing and evaluating an adaptive spoken dialogue system. User Modeling and User-Adapted Interaction, 12 (2-3): 111--137, 2002.

Digital Library

[21]

W. C. Mann and S. A. Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8 (3): 243--281, 1988.

[22]

E. Márquez Segura, M. Kriegel, R. Aylett, A. Deshmukh, and H. Cramer. How do you like me in this: User embodiment preferences for companion agents. In Proceedings of IVA 2012, Sept. 2012.

Digital Library

[23]

T. Masuda and D. Misaki. Development of Japanese green tea serving robot "T-Bartender". In Proceedings of ICMA 2005, volume 2, pages 1069--1074, July 2005. 10.1109/ICMA.2005.1626700.

[24]

Y. Matsusaka, T. Tojo, and T. Kobayashi. Conversation robot participating in group conversation. IEICE Transactions on Information and Systems, 86 (1): 26--36, 2003.

[25]

B. Mutlu, T. Shiwa, T. Kanda, H. Ishiguro, and N. Hagita. Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In Proceedings of HRI 2009, pages 61--68, 2009. 10.1145/1514095.1514109.

Digital Library

[26]

M. Pateraki, H. Baltzakis, and T. P. Visual tracking of hands, faces and facial features as a basis for human-robot communication. In Proceedings of the IROS 2011 Workshop on Visual Tracking and Omnidirectional Vision, September 2011.

[27]

M. Pateraki, H. Baltzakis, and P. Trahanias. Using Dempster's rule of combination to robustly estimate pointed targets. In Proceedings of ICRA 2012, May 2012.

[28]

R. P. A. Petrick and F. Bacchus. A knowledge-based approach to planning with incomplete information and sensing. In Proceedings of AIPS-2002, pages 212--221, Apr. 2002.

[29]

R. P. A. Petrick and F. Bacchus. Extending the knowledge-based approach to planning with incomplete information and sensing. In Proceedings of KR-2004, pages 613--622, June 2004.

[30]

R. P. A. Petrick and M. E. Foster. What would you like to drink? recognising and planning with social states in a robot bartender domain. In Proceedings of CogRob 2012, 2012.

[31]

M. Rickert. Efficient Motion Planning for Intuitive Task Execution in Modular Manipulation Systems. Dissertation, Technische Universität München, 2011.

[32]

V. Rieser and O. Lemon. Reinforcement Learning for Adaptive Dialogue Systems: A Data-driven Methodology for Dialogue Management and Natural Language Generation. Springer, 2011. 10.1007/978-3-642-24942-6.

Digital Library

[33]

Robotics Library. URL http://roblib.sf.net/.

[34]

M. Sigalas, H. Baltzakis, and P. Trahanias. Visual tracking of independently moving body and arms. In Proceedings of IROS '09, Oct. 2009.

Digital Library

[35]

A. Vinciarelli, M. Pantic, and H. Bourlard. Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27 (12): 1743--1759, 2009. 10.1016/j.imavis.2008.11.007.

Digital Library

[36]

J. Wainer, D. J. Feil-Seifer, D. A. Shell, and M. J. Matarić. Embodiment and human-robot interaction: A task-based perspective. In Proceedings of IEEE RO-MAN 2007, pages 872--877, Aug. 2007. 10.1109/ROMAN.2007.4415207.

[37]

M. Walker, C. Kamm, and D. Litman. Towards developing general models of usability with PARADISE. Natural Language Engineering, 6 (3&4): 363--377, 2000. 10.1017/S1351324900002503.

Digital Library

[38]

M. A. Walker. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12: 387--416, 2000.

[39]

M. White. Efficient realization of coordinate structures in Combinatory Categorial Grammar. Research on Language and Computation, 4 (1): 39--75, 2006. 10.1007/s11168-006-9010-2.

Cited By

Lv DSun RZhu QZuo JQin S(2024)Sustainable Impact of Stance Attribution Design Cues for Robots on Human–Robot Relationships—Evidence from the ERSPSustainability10.3390/su1617725216:17(7252)Online publication date: 23-Aug-2024
https://doi.org/10.3390/su16177252
Reimann MKunneman FOertel CHindriks K(2024)A Survey on Dialogue Management in Human-robot InteractionACM Transactions on Human-Robot Interaction10.1145/364860513:2(1-22)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3648605
Lee CPraveena PMutlu B(2024)REX: Designing User-centered Repair and Explanations to Address Robot FailuresProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661559(2911-2925)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661559
Show More Cited By

Index Terms

Two people walk into a bar: dynamic multi-party social interaction with a robot agent
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. External interfaces for robotics

Recommendations

Comparing task-based and socially intelligent behaviour in a robot bartender
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

We address the question of whether service robots that interact with humans in public spaces must express socially appropriate behaviour. To do so, we implemented a robot bartender which is able to take drink orders from humans and serve drinks to them. ...
Multi-party interaction with a virtual character and a human-like robot
VRST '13: Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology

Research on interactive virtual characters and social robots focuses mainly on one-to-one interactions and multi-party interactions concept are rather less explored. As we are developing these characters to be helpful to us in our daily lives as guides, ...
Demonstration of a Robot Receptionist with Multi-party Situated Interaction
HRI '22: Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction

We present a demonstration of a Robot Receptionist: a situated interactive robot that can coordinate turn-taking and handle multi-party engagement and dialogue in dynamic environments, where users might enter or leave the scene at any time. We use a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

October 2012

636 pages

ISBN:9781450314671

DOI:10.1145/2388676

General Chairs:
Louis-Philippe Morency
University of Southern California, USA
,
Dan Bohus
Microsoft Research, USA
,
Hamid Aghajan
Stanford University, USA
,
Program Chairs:
Justine Cassell
Carnegie Mellon University, USA
,
Anton Nijholt
University of Twente, Netherlands
,
Julien Epps
The University of New South Wales, Australia

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '12

Sponsor:

SIGCHI

ICMI '12: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 22 - 26, 2012

California, Santa Monica, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
709
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lv DSun RZhu QZuo JQin S(2024)Sustainable Impact of Stance Attribution Design Cues for Robots on Human–Robot Relationships—Evidence from the ERSPSustainability10.3390/su1617725216:17(7252)Online publication date: 23-Aug-2024
https://doi.org/10.3390/su16177252
Reimann MKunneman FOertel CHindriks K(2024)A Survey on Dialogue Management in Human-robot InteractionACM Transactions on Human-Robot Interaction10.1145/364860513:2(1-22)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3648605
Lee CPraveena PMutlu B(2024)REX: Designing User-centered Repair and Explanations to Address Robot FailuresProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661559(2911-2925)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661559
Sorrentino AFiorini LCavallo F(2024)From the Definition to the Automatic Assessment of Engagement in Human–Robot Interaction: A Systematic ReviewInternational Journal of Social Robotics10.1007/s12369-024-01146-w16:7(1641-1663)Online publication date: 4-Jun-2024
https://doi.org/10.1007/s12369-024-01146-w
Robinson NTidd BCampbell DKulić DCorke P(2023)Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic ReviewACM Transactions on Human-Robot Interaction10.1145/357073112:1(1-66)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.1145/3570731
Jeffcock JHansen MRuiz Garate VCastellano GRiek LCakmak MLeite I(2023)Transformers and Human-robot Interaction for Delirium DetectionProceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568162.3576971(466-474)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3568162.3576971
Bassetti CBlanzieri EBorgo SMarangon S(2023)Towards socially-competent and culturally-adaptive artificial agentsInteraction Studies. Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies / Social Behaviour and Communication in Biological and Artificial SystemsInteraction Studies10.1075/is.22021.bas23:3(469-512)Online publication date: 21-Apr-2023
https://doi.org/10.1075/is.22021.bas
Jin ADeng QDeng Z(2022)S2M-Net: Speech Driven Three-party Conversational Motion Synthesis NetworksProceedings of the 15th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3561975.3562954(1-10)Online publication date: 3-Nov-2022
https://dl.acm.org/doi/10.1145/3561975.3562954
Mara MAppel MGnambs T(2022)Human-Like Robots and the Uncanny ValleyZeitschrift für Psychologie10.1027/2151-2604/a000486230:1(33-46)Online publication date: Jan-2022
https://doi.org/10.1027/2151-2604/a000486
Raza SVitale JTonkin MJohnston BBillingsley RHerse SWilliams M(2022)An in-the-wild study to find type of questions people ask to a social robot providing question-answering serviceIntelligent Service Robotics10.1007/s11370-022-00411-z15:3(411-426)Online publication date: 11-Feb-2022
https://doi.org/10.1007/s11370-022-00411-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents