research-article

Adaptive eye gaze patterns in interactions with human and artificial agents

Authors:

Paul Schermerhorn,

Matthias ScheutzAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 1, Issue 2

Article No.: 13, Pages 1 - 25

https://doi.org/10.1145/2070719.2070726

Published: 13 January 2012 Publication History

Abstract

Efficient collaborations between interacting agents, be they humans, virtual or embodied agents, require mutual recognition of the goal, appropriate sequencing and coordination of each agent's behavior with others, and making predictions from and about the likely behavior of others. Moment-by-moment eye gaze plays an important role in such interaction and collaboration. In light of this, we used a novel experimental paradigm to systematically investigate gaze patterns in both human-human and human-agent interactions. Participants in the study were asked to interact with either another human or an embodied agent in a joint attention task. Fine-grained multimodal behavioral data were recorded including eye movement data, speech, first-person view video, which were then analyzed to discover various behavioral patterns. Those patterns show that human participants are highly sensitive to momentary multimodal behaviors generated by the social partner (either another human or an artificial agent) and they rapidly adapt their gaze behaviors accordingly. Our results from this data-driven approach provide new findings for understanding micro-behaviors in human-human communication which will be critical for the design of artificial agents that can generate human-like gaze behaviors and engage in multimodal interactions with humans.

References

[1]

Allopenna, P., Magnuson, J., and Tanenhaus, M. 1998. Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. J. Mem. Lang. 38, 4, 419--439.

[2]

Argyle, M. and Dean, J. 1965. Eye contact, distance and affiliation. Sociometry 28, 2, 89--304.

[3]

Argyle, M. and Ingham, J. 1972. Gaze, mutual gaze, and proximity. Semiotica 6, 32--49.

[4]

Baldwin, D. 1993. Early referential understanding: Infants' ability to recognize referential acts for what they are. Dev. Psych. 29, 5, 832--843.

[5]

Ballard, D., Hayhoe, M., Pook, P., and Rao, R. 1997. Deictic codes for the embodiment of cognition. Behav. Brain Sci. 20, 4, 723--742.

[6]

Bee, N., Wagner, 1., Andre, E., Vogt, T., Charles, F., Pizzi, D., and Cavazza, M. 2010a. Discovering eye gaze behavior during human-agent conversation in an interactive storytelling application. In Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction.

Digital Library

[7]

Bee, N., Wagner, J., Andre, E., Vogt, T., Charles, F., Pizzi, D., and Cavazza, M. 2010b. Multimodal interaction with a virtual character in interactive storytelling. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems. 1535--1536.

Digital Library

[8]

Breazeal, C. and Scassellati, B. 2000. Infant-like social interactions between a robot and a human caregiver. Adapt. Behav. 8, 1, 49.

Digital Library

[9]

Brennan, S., Chen, X., Dickinson, C., Neider, M., and Zelinsky, G. 2008. Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition 106, 3, 1465--1477.

[10]

Brick, T., and Scheutz, M. 2007. Incremental natural language processing for HRI. In Proceedings of the 2nd ACM IEEE International Conference on Human-Robot Interaction.

Digital Library

[11]

Butterworth, G. 200 I. Joint visual attention in infancy. In Blackwell Handbook of Infant Development 213--240.

[12]

Clark, H. and Krych, M. 2004. Speaking while monitoring addressees for understanding. J. Mem. Lang. 50, 1, 62--81.

[13]

Gergle, D. and Clark, A. T. 2011. See what I'm saying&quest; Using dyadic mobile eye tracking to study collaborative reference. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 435--444.

Digital Library

[14]

Griffin, Z. 2004. Why look&quest; Reasons for eye movements related to language production. In The Interface of Language, Vision, and Action: Eye Movements and the Visual World. 213--247.

[15]

Griffin, Z. and Bock, K. 2000. What the eyes say about speaking. Psych. Sci. 11, 4, 274.

[16]

Gu, E. and Badler, N.I. 2006. Visual Attention and Eye Gaze During Multiparty Conversations with Distractions. In Proceedings of the 6th International Conference on Intelligent Virtual Agents. 193--204.

Digital Library

[17]

Hanna, J. and Brennan, S. 2007. Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation. J. Mem. Lang. 57, 4, 596--615.

[18]

Hasegawa, D., Cassell, J., and Araki, K. 2010. The role of embodiment and perspective in direction-giving systems. In Proceedings of the AAAI Fall Workshop on Dialog with Robots.

[19]

Hayhoe, M. and Ballard, D. 2005. Eye movements in natural behavior. Trends Cogn. Sci. 9, 4, 188--194.

[20]

Hayhoe, M. M., Shrivastava, A., Mruczek, R., and Pelz, J. B. 2003. Visual memory and motor planning in a natural task. J. Vis. 3, 1, 49--63.

[21]

Katzenmaier, K., Stiefelhagen, R., and Schultz, T. 2004. Identifying the addressee in human-human-robot interactions based on head pose and speech. In Proceedings of the 6th ACM International Conference on Multimodal Interfaces. 144--151.

Digital Library

[22]

Kipp, M. and Gebhard, P. 2008. IGaze: Studying reactive gaze behavior in semi-immersive human-avatar interactions, In Proceedings of the 8th International Conference on Intelligent Virtual Agents. 191--199.

Digital Library

[23]

Kramer, J. and Scheutz, M. 2007. ADE: A framework for robust complex robotic architectures. In Proceedings of the International Conference on Intelligent Robots and Systems.

[24]

Land, M., Mennie, N., and Rusted, J. 1999. The roles of vision and eye movements in the control of activities of daily living. Perception 28, 11, 1311--1328.

[25]

MacDorman, K. and Ishiguro, H. 2006. The uncanny advantage of using androids in cognitive and social science research. Interact. Stud. 7, 3, 297--337.

[26]

Meyer, A., Sleiderink, A., and Levelt, W. l998. Viewing and naming objects: Eye movements during noun phrase production. Cognition, 66, 2, 25--33.

[27]

Morency, L.-P., Christoudias, C., and Darrell, T. 2006. Recognizing gaze aversion gestures in embodied conversational discourse. In Proceedings of the 8th International Conference on Multimodal Interfaces.

Digital Library

[28]

Mutlu, B., Forlizzi, J., and Hodgins, J. 2006. A storytelling robot: modeling and evaluation of human-like gaze behavior. In Proceedings of the 6th IEEE-RAS International Conference on Humanoid Robots. 518--523.

[29]

Nakano, Y., Reinstein, G., Stocky, T., and Cassell, J. 2003. Towards a model of face-to-face grounding. In Proceedings of the 41st Meeting of the Association for Computational Linguistics. 553--561.

Digital Library

[30]

Pelz, J., Hayhoe, M., and Loeber, R. 2001. The coordination of eye, head, and hand movements in a natural task. Exper. Brain Resear. 139, 3, 266--277.

[31]

Qu, S. and Chai, J. 2010. Context-based word acquisition for situated dialogue in a virtual world. J. Artif. Intel. Resear. 37, 1, 247--278.

Digital Library

[32]

Rayner, K. 1998. Eye movements in reading and information processing: 20 years of research. Psych. Bull. 124, 372--422.

[33]

Rehm, M. and Andre, E. 2005. Where do they look&quest; Gaze behaviors of multiple users interacting with an embodied conversational agent. In Proceedings of the 5th International Conference on Intelligent Virtual Agents.

[34]

Scheutz, M., Schermerhorn, P., Kramer, J., and Anderson, D. 2007. First steps toward natural human-like HRI. Auton. Robots, 22, 4, 411--423.

Digital Library

[35]

Shintel, H. and Keysar, B. 2009. Less is more: A minimalist account of joint action in communication. Topics Cogn. Sci. 1, 2, 260--273.

[36]

Shockley, K., Santana, M., and Fowler, C. 2003. Mutual interpersonal postural constraints are involved in cooperative conversation. J. Exper. Psych. 29, 2, 326--332.

[37]

Staudte, M. and Crocker, M. 2009. Visual attention in spoken human-robot interaction. In Proceedings of the 4th ACM/IEEE Conference on Human-Robot Interaction.

Digital Library

[38]

Tanenhaus, M., Spivey-Knowlton, M., Eberhard, K., and Sedivy, J. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268, 5217, 1632--1634.

[39]

Trafton, J., Cassimatis, N., Bugajska, M., Brock, D., Mintz, F., and Schultz, A. 2005.Enabling effective human-robot interaction using perspective-taking in robots. IEEE Trans Syst. Man Cyber. 35, 4,460--470.

Digital Library

[40]

Vertegaal, R. 2003. Attentive user interfaces. Comm. ACM 46, 3, 3133.

Digital Library

[41]

Yamazaki, A., Yamazaki, K., Kuno, Y., Burdelski, M., Kawashima, M., and Kuzuoka, H. 2008. Precision timing in human-robot interaction: coordination of head movement and utterance. In Proceedings of the Conference on Human Factors in Computing Systems.

Digital Library

[42]

Yu, C. and Ballard, D. 2004. A multi modal learning interface for grounding spoken language in sensory perceptions. ACM Trans. Appl. Percept. 1, 1, 57--80.

Digital Library

[43]

Yu, C., Scheutz, M., and Schermerhorn, P. 2010a. Investigating multimodal real-time patterns of joint attention in an hri word learning task. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction.

Digital Library

[44]

Yu, C., Smith, L., Shen, H., Pereira, A., and Smith, T. 2009a. Active information selection: Visual attention through the hands. IEEE Trans. Auton. Ment. Devel. 2, 141--151.

Digital Library

[45]

Yu, C., Smith, T., Hidaka, S., Scheutz, M., and Smith, L. 2010b. A data-driven paradigm to understand multimodal communication in human-human and human-robot interaction. In Advances in Intelligent Data Analysis, vol. 9, 232--244.

Digital Library

[46]

Yu, C., Zhong, Y., Smith, T., Park, I., and Huang, W. 2009b. Visual data mining of multimedia data for social and behavioral studies. Inf. Visual. 8, 1, 56--70.

Digital Library

[47]

Zhang, H., Fricker, D., Smith, T., and Yu, C. 2010. Real-time adaptive behaviors in multimodal human-avatar interactions. In Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction.

Digital Library

Cited By

Menendez EMartínez SDíaz-de-María FBalaguer C(2024)Integrating Egocentric and Robotic Vision for Object Identification Using Siamese Networks and Superquadric Estimations in Partial Occlusion ScenariosBiomimetics10.3390/biomimetics90201009:2(100)Online publication date: 8-Feb-2024
https://doi.org/10.3390/biomimetics9020100
Sivridag FMani NGrollman DBroadbent EJu WSoh HWilliams T(2024)Children's Word Learning from Socially Contingent Robots Under Active vs. Passive Learning ConditionsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634931(669-677)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634931
Hoorn JHuang I(2024)The media inequality, uncanny mountain, and the singularity is far from nearInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103142181:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103142
Show More Cited By

Index Terms

Adaptive eye gaze patterns in interactions with human and artificial agents

Recommendations

See You See Me: The Role of Eye Contact in Multimodal Human-Robot Interaction
Special Issue on New Directions in Eye Gaze for Interactive Intelligent Systems (Part 2 of 2), Regular Articles and Special Issue on Highlights of IUI 2015 (Part 1 of 2)

We focus on a fundamental looking behavior in human-robot interactions—gazing at each other's face. Eye contact and mutual gaze between two social partners are critical in smooth human-human interactions. Therefore, investigating at what moments and in ...
Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes
CHI '01: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

In multi-agent, multi-user environments, users as well as agents should have a means of establishing who is talking to whom. In this paper, we present an experiment aimed at evaluating whether gaze directional cues of users could be used for this ...
Conversational gaze mechanisms for humanlike robots

During conversations, speakers employ a number of verbal and nonverbal mechanisms to establish who participates in the conversation, when, and in what capacity. Gaze cues and mechanisms are particularly instrumental in establishing the participant roles ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 1, Issue 2

January 2012

157 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/2070719

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 January 2012

Accepted: 01 October 2011

Revised: 01 July 2011

Received: 01 January 2011

Published in TIIS Volume 1, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
806
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Menendez EMartínez SDíaz-de-María FBalaguer C(2024)Integrating Egocentric and Robotic Vision for Object Identification Using Siamese Networks and Superquadric Estimations in Partial Occlusion ScenariosBiomimetics10.3390/biomimetics90201009:2(100)Online publication date: 8-Feb-2024
https://doi.org/10.3390/biomimetics9020100
Sivridag FMani NGrollman DBroadbent EJu WSoh HWilliams T(2024)Children's Word Learning from Socially Contingent Robots Under Active vs. Passive Learning ConditionsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634931(669-677)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634931
Hoorn JHuang I(2024)The media inequality, uncanny mountain, and the singularity is far from nearInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103142181:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103142
Mishra COffrede TFuchs SMooshammer CSkantze G(2023)Does a robot’s gaze aversion affect human gaze aversion?Frontiers in Robotics and AI10.3389/frobt.2023.112762610Online publication date: 23-Jun-2023
https://doi.org/10.3389/frobt.2023.1127626
Koller MWeiss AHirschmanner MVincze M(2023)Robotic gaze and human views: A systematic exploration of robotic gaze aversion and its effects on human behaviors and attitudesFrontiers in Robotics and AI10.3389/frobt.2023.106271410Online publication date: 10-Apr-2023
https://doi.org/10.3389/frobt.2023.1062714
Nunnemann EKreysa HKnoeferle P(2023)The effects of referential gaze in spoken language comprehension: Human speaker vs. virtual agent listener gazeFrontiers in Communication10.3389/fcomm.2023.10291578Online publication date: 18-Apr-2023
https://doi.org/10.3389/fcomm.2023.1029157
Chang YGandi NShin KMun YDriggs-Campbell KKim J(2023)Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids)10.1109/Humanoids57100.2023.10375186(1-7)Online publication date: 12-Dec-2023
https://doi.org/10.1109/Humanoids57100.2023.10375186
Parreira MGillet SVázquez MLeite ISakamoto DWeiss AHiatt LShiomi M(2022)Design Implications for Effective Robot Gaze Behaviors in Multiparty InteractionsProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523918(976-980)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523918
Briggs GChita-Tegmark MKrause EBridewell WBello PScheutz MSakamoto DWeiss AHiatt LShiomi M(2022)A Novel Architectural Method for Producing Dynamic Gaze Behavior in Human-Robot InteractionsProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523812(383-392)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523812
ZHANG H(2022)交流语言认知理论 <sup>*</sup>Advances in Psychological Science10.3724/SP.J.1042.2018.0101926:6(1019-1019)Online publication date: 13-Jul-2022
https://doi.org/10.3724/SP.J.1042.2018.01019
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents