article

2006 Special issue: A probabilistic model of gaze imitation and shared attention

Authors:

Matthew W. Hoffman,

David B. Grimes,

Rajesh P. N. RaoAuthors Info & Claims

Neural Networks, Volume 19, Issue 3

Pages 299 - 310

https://doi.org/10.1016/j.neunet.2006.02.008

Published: 01 April 2006 Publication History

Abstract

An important component of language acquisition and cognitive learning is gaze imitation. Infants as young as one year of age can follow the gaze of an adult to determine the object the adult is focusing on. The ability to follow gaze is a precursor to shared attention, wherein two or more agents simultaneously focus their attention on a single object in the environment. Shared attention is a necessary skill for many complex, natural forms of learning, including learning based on imitation. This paper presents a probabilistic model of gaze imitation and shared attention that is inspired by Meltzoff and Moore's AIM model for imitation in infants. Our model combines a probabilistic algorithm for estimating gaze vectors with bottom-up saliency maps of visual scenes to produce maximum a posteriori (MAP) estimates of objects being looked at by an observed instructor. We test our model using a robotic system involving a pan-tilt camera head and show that combining saliency maps with gaze estimates leads to greater accuracy than using gaze alone. We additionally show that the system can learn instructor-specific probability distributions over objects, leading to increasing gaze accuracy over successive interactions with the instructor. Our results provide further support for probabilistic models of imitation and suggest new ways of implementing robotic systems that can interact with humans over an extended period of time.

References

[1]

Alissandrakis, A., Nehaniv, C. L., & Dautenhahn, K. (2000). Learning how to do things with imitation. In Proceeding 'learning how to do things' (AAAI fall symposium series), (pp. 1-8).]]

[2]

Alissandrakis, A., Nehaniv, C. L., & Dautenhahn, K. (2003). Solving the correspondence problem between dissimilarly embodied robotic arms using the ALICE imitation mechanism. In Proceedings of the second international symposium on imitation in animals & artifacts (pp. 79-92).]]

[3]

Billard, A., Epars, Y., Calinon, S., Cheng, G., & Schaal S. (2004). Discovering optimal imitation strategies. Robotics and Autonomous SystemsM, 47, 2-3.]]

[4]

Billard, A., & Mataric, M. J. (2000). A biologically inspired robotic model for learning by imitation. In C. Sierra, M. Gini, & J. S. Rosenschein (Eds.), Proceedings of the fourth international conference on autonomous agents (pp. 373-380). Barcelona, Catalonia, Spain: ACM Press.]]

Digital Library

[5]

Blakemore, S. J., Frith, C. D., & Wolpert, D. M. (2001). The cerebellum is involved in predicting the sensory consequences of action. Neuroreport, 12, 1879-1884.]]

[6]

Blakemore, S. J., Goodbody, S. J., & Wolpert, D. M. (1998). Predicting the consequences of our own actions: The role of sensorimotor context estimation. Journal of Neuroscience, 18(18), 7511-7518.]]

[7]

Booth, M. C. A., & Rolls, E. T. (1998). View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex. Cerebral Cortex, 8(6), 510-523.]]

[8]

Breazeal, C. (1999). Imitation as social exchange between humans and robots. In Proceedings of the artificial intelligence and the simulation of behaviour (pp. 96-104).]]

[9]

Breazeal, C., Buchsbaum, D., Gray, J., Gatenby, D., & Blumberg, B. (2005). Learning from and about others: Towards using imitation to bootstrap the social understanding of others by robots. Artificial Life, 11, 31-62.]]

Digital Library

[10]

Breazeal, C., & Scassellati, B. (2001). Challenges in building robots that imitate people. In K. Dautenhahn, & C. Nehaniv (Eds.), Imitation in animals and artifacts. Cambridge, MA: MIT Press.]]

Digital Library

[11]

Breazeal, C., & Velasquez, J. (1998). Toward teaching a robot 'infant' using emotive communication acts. In Proceedings of the 1998 simulation of adaptive behavior, workshop on socially situated intelligence (pp. 25-40).]]

[12]

Brooks, R., & Meltzoff, A. (2002). The importance of eyes: How infants interpret adult looking behavior. Developmental Psychology, 38, 958-966.]]

[13]

Buccino, G., Binofski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, V., et al. (2001). Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study. European Journal of Neuroscience, 13, 400-404.]]

[14]

Byrne, R. W., & Russon, A. E. (1998). Learning by imitation: A hierarchical approach. Behavioral and Brain Sciences, 21, 667-721.]]

[15]

Calinon, S., & Billard, A. (2005). Recognition and reproduction of gestures using a probabilistic framework combining PCA, ICA and HMM. In Proceedings of the international conference on machine learning (ICML).]]

Digital Library

[16]

Calinon, S., Guenter, F., & Billard, A. (2005). Goal-directed imitation in a humanoid robot. In Proceedings of the IEEE international conference on robotics and automation (ICRA)]]

[17]

Calinon, S., Guenter, F., & Billard, A. (2006). On learning the statistical representation of a task and generalizing it to various contexts. In Proceedings of the IEEE international conference on robotics and automation (ICRA).]]

[18]

Carpenter, R. H. S. (1988). Movements of the eyes (2nd ed.). London: Pion Ltd.]]

[19]

Comaniciu, D., Ramesh, V., & Meer, P. (2000). Real-time tracking of non-rigid objects using mean shift. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 142-151).]]

[20]

Demiris, Y., & Khadhouri, B. (2005). Hierarchical, attentive multiple models for execution and recognition (HAMMER). In Proceedings of the ICRA workshop on robot programming by demonstration.]]

[21]

Demiris, J., Rougeaux, S., Hayes, G., Berthouze, L., & Kuniyoshi, Y. (1997). Deferred imitation of human head movements by an active stereo vision head. In Proceedings of the sixth IEEE international workshop on robot human communication.]]

[22]

Dempster, A.P., Laird, N.M., & Rubin, D. (1977). Maximum-likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39.]]

[23]

di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolati, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176-180.]]

[24]

Fasel, I., Deak, G. O., Triesch, J., & Movellan, J. R. (2002). Combining embodied models and empirical research for understanding the development of shared attention. In Proceeding of ICDL 2.]]

Digital Library

[25]

Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2002). A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3-4), 142-166.]]

[26]

Haruno, M., Wolpert, D., & Kawato, M. (2000). MOSAIC model for sensorimotor learning and control. Neural Computation, 13, 2201-2222.]]

Digital Library

[27]

Iacoboni, M. (2005). Understanding others: Imitation, language, empathy. In S. Hurley, & N. Chater (Eds.), Perspectives on imitation: From mirror neurons to memes. Mechanisms of imitation and imitation in animals: Vol. 1. Cambrdige, MA: MIT Press.]]

[28]

Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11), 1254-1259.]]

Digital Library

[29]

Jansen, B., & Belpaeme, T. (2005). Goal-directed imitation through repeated trial-and-error interactions between agents. In Proceedings of the workshop on social mechanisms of robot programming by demonstration.]]

[30]

Johnson, M., & Demiris, Y. (2005). Hierarchies of coupled inverse and forward models for abstraction in robot planning, recognition and imitation. In: Proceedings of the AISB symposium on imitation in animals and artifacts (pp. 69-76).]]

[31]

Johnson-Frey, S. H., Maloof, F. R., Newman-Norlund, R., Farter, C., Inati, S., & Grafton, S. T. (2003). Actions or hand-objects interactions? Human inferior frontal cortex and action observation. Neuron, 39, 1053-1058.]]

[32]

Jordan, M. I., & Rumelhart, D. E. (1992). Forward models: Supervised learning with a distal teacher. Cognitive Science, 16, 307-354.]]

[33]

Krebs, J. R., & Kacelnik, A. (1991). Decision making. In J. R. Krebs, & N. B. Davies (Eds.), Behavioural ecology. 3rd ed. (pp. 105-137). Oxford: Blackwell Scientific Publishers.]]

[34]

Lempers, J. D. (1979). Young children's production and comprehension of nonverbal deictic behaviors. Journal of Genetic Psychology, 135, 93-102.]]

[35]

Lungarella, M., & Metta, G. (2003). Beyond gazing, pointing, and reaching: A survey of developmental robotics. In EPIROB '03, pp. 81-89.]]

[36]

Meltzoff, A. N., & Moore, M. K. (1977). Imitation of facial and manual gestures by human neonates. Science, 198, 75-78.]]

[37]

Meltzoff, A. N., & Moore, M. K. (1997). Explaining facial imitation: A theoretical model. Early Development and Parenting, 6, 179-192.]]

[38]

Meltzoff, A. N. (2005). Imitation and other minds: The 'like me' hypothesis. In S. Hurley, & N. Chater (Eds.), Perspectives on imitation: From cognitive neuroscience to social science (pp. 55-77). Cambridge, MA: MIT Press.]]

[39]

Miall, R. C. (2003). Connecting mirror neurons and forward models. Neuroreport, 14, 2135-2137.]]

[40]

Moore, C., Angelopoulos, M., & Bennett, P. (1997). The role of movement in the development of joint visual attention. Infant Behavior and Development, 20(1), 83-92.]]

[41]

Nagai, Y., Hosoda, K., Morita, A., & Asada, M. (2003). Emergence of joint attention based on visual attention and self learning. In Proceedings of the second international symposium on adaptive motion of animals and machines.]]

[42]

Nehaniv, C. L., & Dautenhahn, K. (2000). Of hummingbirds and helicopters: An algebraic framework for interdisciplinary studies of imitation and its applications. In J. Demiris, & A. Birk (Eds.), Interdisciplinary approaches to robot learning. Singapore: World Scientific Press.]]

[43]

Price, B. (2003). Accelerating reinforcement learning with imitation. PhD thesis, University of British Columbia.]]

Digital Library

[44]

Rao, R. P. N., & Meltzoff, A. N. (2003). Imitation learning in infants and robots: Towards probabilistic computational models. In Proceedings of the artificial intelligence and the simulation of behaviour.]]

[45]

Rao, R. P. N., Shon, A. P., & Meltzoff, A. N. (2004). A Bayesian model of imitation in infants and robots. In K. Dautenhahn, & C. Nehaniv (Eds.), Imitation and Social Learning in Robots, Humans, and Animals: Behavioural, Social and Communicative Dimensions. Cambridge: Cambridge University Press.]]

[46]

Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review of Neuroscience, 27, 169-192.]]

[47]

Rizzolatti, G., Fogassi, L., & Gallese, V. (2000). Mirror neurons: Intentionality detectors? International Journal of Psychology, 35, 205.]]

[48]

Scassellati, B. (1999). Imitation and mechanisms of joint attention: A developmental structure for building social skills on a humanoid robot. Lecture Notes in Computer Science, 1562, 176-195.]]

[49]

Viola, P., & Jones, M. (2001). Robust real-time object detection. International Journal of Computer Vision.]]

Digital Library

[50]

Visalberghy, E., & Fragaszy, D. (1990). Do monkeys ape? In Language and intelligence in monkeys and apes: Comparative developmental perspectives , Cambridge University Press (pp. 247-273).]]

[51]

Wu, Y., Toyama, K., & Huang, T. (2000). Wide-range, person-and illumination-insensitive head orientation estimation. In AFGR00 (pp. 183-188).]]

Digital Library

[52]

Yamane, S., Kaji, S., & Kawano, K. (1988). What facial features activate face neurons in the inferotemporal cortex of the monkey? Experimental Brain Research, 73(1), 209-214.]]

Cited By

Zhang QHuang YChernyshov GLi JPai YKunze K(2022)GazeSync: Eye Movement Transfer Using an Optical Eye Tracker and Monochrome Liquid Crystal DisplaysCompanion Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490100.3516469(54-57)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490100.3516469
Stefanov KSalvi GKontogiorgos DKjellström HBeskow J(2019)Modeling of Human Visual Attention in Multiparty Open-World DialoguesACM Transactions on Human-Robot Interaction10.1145/33232318:2(1-21)Online publication date: 3-Jun-2019
https://dl.acm.org/doi/10.1145/3323231
Admoni HScassellati B(2017)Social eye gaze in human-robot interactionJournal of Human-Robot Interaction10.5898/JHRI.6.1.Admoni6:1(25-63)Online publication date: 26-May-2017
https://dl.acm.org/doi/10.5898/JHRI.6.1.Admoni
Show More Cited By

Index Terms

Recommendations

Weak gaze awareness in video-mediated communication
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

We present a video mediated communication system that conveys gaze information to a remote location. Unlike existing video mediated communication system, this system does not send visual information directly, only gaze position and face direction. The ...
MAGIC pointing for eyewear computers
ISWC '15: Proceedings of the 2015 ACM International Symposium on Wearable Computers

In this paper, we propose a combination of head and eye movements for touchlessly controlling the "mouse pointer" on eyewear devices, exploiting the speed of eye pointing and accuracy of head pointing. The method is a wearable computer-targeted ...
2010 Special Issue: How do children learn to follow gaze, share joint attention, imitate their teachers, and use tools during social interactions?

How does an infant learn through visual experience to imitate actions of adult teachers, despite the fact that the infant and adult view one another and the world from different perspectives? To accomplish this, an infant needs to learn how to share ...

Comments

Information & Contributors

Information

Published In

cover image Neural Networks

Neural Networks Volume 19, Issue 3

2006 Special issue: The brain mechanisms of imitation learning

April 2006

89 pages

ISSN:0893-6080

Issue’s Table of Contents

Copyright © © 2006.

Publisher

Elsevier Science Ltd.

United Kingdom

Publication History

Published: 01 April 2006

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang QHuang YChernyshov GLi JPai YKunze K(2022)GazeSync: Eye Movement Transfer Using an Optical Eye Tracker and Monochrome Liquid Crystal DisplaysCompanion Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490100.3516469(54-57)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490100.3516469
Stefanov KSalvi GKontogiorgos DKjellström HBeskow J(2019)Modeling of Human Visual Attention in Multiparty Open-World DialoguesACM Transactions on Human-Robot Interaction10.1145/33232318:2(1-21)Online publication date: 3-Jun-2019
https://dl.acm.org/doi/10.1145/3323231
Admoni HScassellati B(2017)Social eye gaze in human-robot interactionJournal of Human-Robot Interaction10.5898/JHRI.6.1.Admoni6:1(25-63)Online publication date: 26-May-2017
https://dl.acm.org/doi/10.5898/JHRI.6.1.Admoni
Mavridis N(2015)A review of verbal and non-verbal human-robot interactive communicationRobotics and Autonomous Systems10.1016/j.robot.2014.09.03163:P1(22-35)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.1016/j.robot.2014.09.031
Doriya RPrasad JNandi G(2010)Efficacy of gesture for communication among humanoid robots by fuzzy inference methodInternational Journal of Computational Vision and Robotics10.1504/IJCVR.2010.0381901:4(349-362)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1504/IJCVR.2010.038190
Mohammad YNishida T(2010)Controlling gaze with an embodied interactive control architectureApplied Intelligence10.1007/s10489-009-0180-032:2(148-163)Online publication date: 1-Apr-2010
https://dl.acm.org/doi/10.1007/s10489-009-0180-0
Dornaika FRaducanu B(2009)Three-dimensional face pose detection and tracking using monocular videosIEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics10.1109/TSMCB.2008.200956639:4(935-944)Online publication date: 1-Aug-2009
https://dl.acm.org/doi/10.1109/TSMCB.2008.2009566
Jasso HTriesch J(2008)Learning to Attend -- From Bottom-Up to Top-DownAttention in Cognitive Systems. Theories and Systems from an Interdisciplinary Viewpoint10.1007/978-3-540-77343-6_7(106-122)Online publication date: 3-Dec-2008
https://dl.acm.org/doi/10.1007/978-3-540-77343-6_7
Triesch JJasso HDeák G(2007)Emergence of Mirror Neurons in a Model of Gaze FollowingAdaptive Behavior - Animals, Animats, Software Agents, Robots, Adaptive Systems10.1177/105971230707865415:2(149-165)Online publication date: 1-Jun-2007
https://dl.acm.org/doi/10.1177/1059712307078654

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents