Abstract
Our state of mind is based on experiences and what other people tell us. This may result in conflicting information, uncertainty, and alternative facts. We present a robot that models relativity of knowledge and perception within social interaction following principles of the theory of mind. We utilized vision and speech capabilities on a Pepper robot to build an interaction model that stores the interpretations of perceptions and conversations in combination with provenance on its sources. The robot learns directly from what people tell it, possibly in relation to its perception. We demonstrate how the robot’s communication is driven by hunger to acquire more knowledge from and on people and objects, to resolve uncertainties and conflicts, and to share awareness of the perceived environment. Likewise, the robot can make reference to the world and its knowledge about the world and the encounters with people that yielded this knowledge.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Where possible, we follow the PROV-O model: https://www.w3.org/TR/prov-o/.
- 2.
There are now two perspectives from Lenka on the same claim (she changed his mind), expressed in two different utterances.
- 3.
The robot continuously detects objects, but these are only stored in memory when they are referenced by humans in the communication.
References
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
Amos, B., Ludwiczuk, B., Satyanarayanan, M.: Openface: a general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, CMU School of Computer Science (2016)
T.W. Project Authors: WebRTC. Online publication (2011). https://webrtc.org/
Baron-Cohen, S.: Mindblindness: An Essay on Autism and Theory of Mind. MIT Press, Cambridge (1997)
Bratman, M.: Intention, plans, and practical reason (1987)
Card, S.K.: The Psychology of Human-Computer Interaction. CRC Press, Boca Raton (2017)
Epley, N., Waytz, A., Cacioppo, J.T.: On seeing human: a three-factor theory of anthropomorphism. Psychol. Rev. 114(4), 864 (2007)
Fokkens, A., Vossen, P., Rospocher, M., Hoekstra, R., van Hage, W.: Grasp: grounded representation and source perspective. In: Proceedings of KnowRSH, RANLP-2017 Workshop, Varna, Bulgaria (2017)
Google: Cloud speech-to-text - speech recognition. Online publication (2018). https://cloud.google.com/speech-to-text/
Hiatt, L.M., Harrison, A.M., Trafton, J.G.: Accommodating human variability in human-robot teams through theory of mind. In: IJCAI Proceedings-International Joint Conference on Artificial Intelligence, vol. 22, p. 2066 (2011)
Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: 2000 Proceedings of Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53. IEEE (2000)
Leslie, A.M.: Pretense and representation: the origins of “theory of mind”. Psychol. Rev. 94(4), 412 (1987)
Mavridis, N.: A review of verbal and non-verbal human-robot interactive communication. Robot. Auton. Syst. 63, 22–35 (2015)
Mirnig, N., Stollnberger, G., Miksch, M., Stadler, S., Giuliani, M., Tscheligi, M.: To err is robot: how humans assess and act toward an erroneous social robot. Front. Robot. AI 4, 21 (2017)
Ono, T., Imai, M., Nakatsu, R.: Reading a robot’s mind: a model of utterance understanding based on the theory of mind mechanism. Adv. Robot. 14(4), 311–326 (2000)
Partan, S.R., Marler, P.: Issues in the classification of multimodal communication signals. Am. Nat. 166(2), 231–245 (2005)
Premack, D., Woodruff, G.: Does the chimpanzee have a theory of mind? Behav. Brain Sci. 4, 515–526 (1978)
Scassellati, B.: Theory of mind for a humanoid robot. Auton. Robot. 12(1), 13–24 (2002)
Scassellati, B.M.: Foundations for a theory of mind for a humanoid robot. Ph.D. thesis, Massachusetts Institute of Technology (2001)
Serban, I.V., Sordoni, A., Bengio, Y., Courville, A.C., Pineau, J.: Building end-to-end dialogue systems using generative hierarchical neural network models. In: AAAI, vol. 16, pp. 3776–3784 (2016)
She, L., Chai, J.: Interactive learning of grounded verb semantics towards human-robot communication. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 1634–1644 (2017)
Szegedy, C., et al.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015). http://arxiv.org/abs/1409.4842
Van Hage, W.R., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (SEM). Web Semant.: Sci. Serv. Agents World Wide Web 9(2), 128–136 (2011)
Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)
Vossen, P., et al.: Newsreader: using knowledge resources in a cross-lingual reading machine to generate more knowledge from massive streams of news. Knowl.-Based Syst. (2016). http://www.sciencedirect.com/science/article/pii/S0950705116302271
Wahlster, W.: SmartKom: Foundations of Multimodal Dialogue Systems, vol. 12. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-36678-4
Acknowledgement
This research was funded by the VU University Amsterdam and the Netherlands Organization for Scientific Research via the Spinoza grant awarded to Piek Vossen. We also thank Bob van der Graft for his support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix: Dialogues
Appendix: Dialogues
In the dialogues, L preceding an utterance stands for Leolani, other letters preceding utterances stand for various people. Perceptions of the robot of people and objects are marked using square brackets, e.g. [Sees a new face].
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Vossen, P., Baez, S., Bajc̆etić, L., Kraaijeveld, B. (2018). Leolani: A Reference Machine with a Theory of Mind for Social Communication. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech, and Dialogue. TSD 2018. Lecture Notes in Computer Science(), vol 11107. Springer, Cham. https://doi.org/10.1007/978-3-030-00794-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-00794-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00793-5
Online ISBN: 978-3-030-00794-2
eBook Packages: Computer ScienceComputer Science (R0)