Procedia Computer Science
Volume 71, 2015, Pages 112–118
2015 Annual International Conference on Biologically Inspired
Cognitive Architectures
Anthropomorphic artificial social agent
with simulated emotions and its implementation.
Vlada Kugurakova, Maxim Talanov, Nadir Manakhov, and Denis Ivanov
Kazan Federal University, Kazan, Tatarstan, Russian Federation
vlada.kugurakova@gmail.com
Abstract
In this paper we describe an emotional human-machine interface as an anthropomorphic social agent able to exhibit simulated emotions and react to emotional stimuli. We propose a
neurobiologically inspired agent implementation that is based on mechanics of chemical and
physiological processes within human brain. Implementation of model features simulation of
neuromodulators such as dopamine, serotonin, and noradrenaline. Demonstration of emotions is
achieved via combining aforementioned neuromodulators in different proportions. The Lovheim
cube of emotions is used for this purpose. Topic of ”uncanny valley” phenomenon and its effect
on human-machine interactions is also mentioned. In conclusion of this paper we have proposed
realistic computation model allowing us to visualize agents mimics in sync with his speech, and
have made a working prototype of aforementioned model.
Keywords: intelligent agents, visualization, emotional artificial intelligence, neuromodulators, visual
speech synthesis, expressive and controllable speech synthesis
1
Introduction
Today it becomes more and more apparent that IT-industry and humanity as whole needs
more versatile and easy to use and understand human-machine interfaces. Those interfaces
must feature context awareness and an ability to maintain a conversation, as well as a defined
emotional reaction to interlocutor’s actions and phrases. We could use them for designing
systems capable of learning during conversation and reflection in real-life environment. Use
cases of emotional human-machine interfaces include but not limited to the simulation of an
intelligent dialog respondent and the creation of human-friendly technogenic environment and
the ergatic systems which enhance safety and capabilities of human. [3]
Among the examples of that new human-machine interface concepts that are already being
used, there are intelligent personal assistants which recognize voice commands and chat bots
that use user messages databases to generate a response. However, it is not yet possible to
attribute them to emotional artificial intelligence due to absence of complex inner structure
that determines agents response and internal emotional state.
112
Selection and peer-review under responsibility of the Scientific Programme Committee of BICA 2015
c The Authors. Published by Elsevier B.V.
doi:10.1016/j.procs.2015.12.217
Anthropomorphic artificial social agent . . .
Kugurakova, Talanov, Manakhov and Ivanov
An agent implementing neuropsychological emotional model currently does not exist, even
though emotions are essential part of human life – going from decision making to enabling
self-preservation instinct. There are, however, many ways to convey emotion without altering
agents inner state. In this paper, our cross-disciplinary goal was to create an anthropomorphic
agent that has internal emotional state and reacts to emotional stimuli, that express sympathy
or aggression towards it’s interlocutor.
2
Description of Computation Model
We propose a neurobiologically inspired agent implementation that is based on mechanics of
neurophysiological processes within the human brain. Long-term research in this field provided
an option for us to define formal model that determines agents internal state. [11, 12, 10]
Key differences of our model stem from its underlying architecture, which incorporates in
itself both psychological and physiological approaches to explanation of emotions. On physiological side, design and development of the emotional model is carried out via simulation of
neuromodulators: dopamine, serotonin, and noradrenaline [9, 8]. On the psychological side, our
main concern was bridging neurobiological and psychological explanations of emotion appraisal
theory and doing so with respect to neural-dynamic reward-punishment learning models. This
topic was covered extensively in another paper submitted to BICA by one of our co-authors.
Different emotional states are achieved via combining aforementioned neuromodulators in
different proportions. We use Tomkins affects theory [19, 18, 17, 16] for this, which is further
extended by Lovheim’s cube of emotions [6]. Silvan Tomkins describes innate emotions that
are exhibited by toddlers since birth as affects: anger, embarrassment, sorrow, fear, interest,
happiness, surprise, disgust.
Emotional computation model would also benefit from integration of complex social emotions like: empathy, envy, love, aggression, awe, respect, etc in it. Another key improvement
to agent’s model is a set of various cognitive functions: knowledge and learning, intentions and
plans, and long-term goals [2, 14, 15].
Our solution to the aforementioned problem is construction of computation model of an
essential set of functions, including basic emotions and other cognitive functions (memory,
decision making, perception, understanding, judgment making and even language), and we are
already making advances in this direction.
3
Implementation
We begun our work in this field with addressing relevant existing research and starting to
work on what would be groundbreaking neuropsychological model for an emotional artificial
intelligence. Agent implementation and computational model development are processing in
the same time.
Most tricky part of agent implementation has to be the way we can tie together underlying fundamental logic and agent visualization completed with emotional display and humanmachine interaction. An agent designed to interact with people in real time obviously needs a
conversation system, which means that agent has to both be able to decode human speech and
correctly get the message, and to form a response to this message after processing it. Those
capabilities, of course, are nothing new to scientific community, with commercial products such
as Apple Siri and Microsoft Cortana being notable examples. However, emotional artificial intelligence is able to the whole process in a new way. Emotions can be added to human-machine
113
Anthropomorphic artificial social agent . . .
Kugurakova, Talanov, Manakhov and Ivanov
Figure 1: First prototype displaying various emotional masks
interactions in a multiple ways, from basic mimics and voice tone change to alteration of agents
thought process itself and to agents response. The scope of our project includes both of those
approaches.
As a result, we constructed realistic computation model which allows us to visualize agent’s
mimics in sync with his speech. Text To Speech technology was used to transform agents
messages in the voiced speech. Microsoft Kinect controller allowed us to perform photogrammetry of human mimics during pronunciation of different sounds, syllables and during demonstration of various emotions. Afterwards we were able to create basic emotional states of our
agent. Mimics recognition software Faceshift was used to carry out this part of work.
In the end, first prototype of virtual anthropomorphic agent finally was ready. The prototype
was capable of voicing complicated text in two different languages while displaying appropriate
human-like mimics. Correct mixture of mimic models of emotion and of speech pronunciation
was achieved with Blendshape technology. Those masks were put on the 3D-model of a
human head to create an anthropomorphic social agent. Unity Engine was an underlying
framework for visualizing resulting agent, and Microsoft Speech Library was utilized for
speech recognition and speech synthesis.
We should also mention an important problem specific to design of anthropomorphic humanmachine interfaces – ” uncanny valley ” phenomenon. Modern implementations of social agents
do not meet the requirements that people have to their virtual interlocutors. As a result, human
and machine cannot reach understanding, since human can’t trust agent or even finds it to be
creepy. Various ways of solving this problem were described, for example, in works of Japanese
psychologists. [21] We are currently conductiong further research of the topic, namely on how
people respond to the human-machine interfaces of varying closeness to human. This research
would be soon presented to scientific community.
4
Overview of Existing Technologies
Research in the field of intelligent social agents is long. Each year, new software is developed
and old software receives much needed updates. Among most prominent of ones that are related
to emotional social agents there are:
114
Anthropomorphic artificial social agent . . .
Kugurakova, Talanov, Manakhov and Ivanov
• Speech recognition
• Speech analysis
• Simulation of human speech
• Simulation of human mimics and movements
• Common sense based reasoning
• Conversation support
• Conversation context awareness
For example, chatterbot Eugene Goostman [13] had passed Turing test [5] in open competitions that was held in 2014.
Speech translation systems such as Skype Translator are currently actively being developed, too. Many intelligent agents software had become commercially viable and are being
used as intelligent personal assistants, like iOS Siri and Amazon Alexa. They primarily utilize
speech recognition and speech synthesis and Q&A systems.
Companies that are concerned with creation of anthropomorphic androids are also exist.
Aiko, ASIMO, Einstein Robot, Repliee R-1, and Ibn-Sinna are particularly remarkable.
Following are some solutions for Q&A systems to consider:
• IBM Watson – supercomputer that is capable of understanding questions in natural language and search for answers in data base;
• MicrosoftTTS, GoogleTTS, Ivona, Acapella are used for text-to-speech translation;
• FaceFX, CrazyTalk, MagPie synchronize lip motion with speech.
One example of the implementation of anthropomorphic social agents is a project Zoe of
the University of Cambridge [20], performed in collaboration with the laboratory of Toshiba,
the first prototype which was established in March 2013 and continues to develop now. It is a
realistic virtual social agent with a three-dimensional model of a woman’s face with a speech
synthesizer and 6 emotional states.
The fundamental problem of human cognition, thinking, consciousness, self-consciousness
(ego) is not solved yet, either. In other words, it remains a mystery how the elementary actions
of neurons, cortical columns, Brodmann areas, lobes of the brain are formed phenomena such
as motivation, imagination, creative thinking, sleeping, etc. Because of this, studies concerning human brain are on their rise, including ”Connectome” [4], ”Blue Brain”, ”Human Brain
Project” as remarkable examples.
5
Future Work
General artificial intelligence development cannot be executed exclusively through predefined
algorithms. Many aspects of human brain still remain mysterious to both scientists and engineers. Creation of machines that develop their very own algorithms in attempt to solve problems
they encounter can be a viable solution. In essence, the result of this would be a machine that
learns to learn. Danko Nicolich [1] suggested that this can be achieved through creation AI
kindergarten. In AI-Kindergarten machines are not left alone to figure out on their own the
necessary algorithms, but they are heavily guided through human feedback.
After advancing this far in our research, we determined our next steps.
115
Anthropomorphic artificial social agent . . .
5.1
Kugurakova, Talanov, Manakhov and Ivanov
Emotional Response Based on Innate Emotional Affects
There are inborn emotional reactions that any human exhibit since its very birth, such as crying,
smiling and disgust. In the terms of Marvin Minsky, ”Critics and ways of thinking” [7], when an
element of analytical framework (critic) is triggered, there is some emotional reaction, such as
fear, which is closely associated with physiological arousal and causes innate behavior. On the
basis of this triplet (analysis-response-behavior), with age in humans occurs gamma strategy to
deal with emotional reactions. Description of a formal model of these mechanisms in terms of
computing systems will become an adequate solution to the display of emotions and emotional
behavior of virtual agents.
5.2
Artificial Intelligence Kindergarten
Our main concerns in research of AI kindergarten are:
• Development of agents that are capable of learning
• Development of logical reasoning
• Development of machine self-reflection
5.3
Knowledge Base Queries Based on Natural Language Translation
Those are steps in which we are planning to handle those topic:
• Generation of multiple representations of knowledge base
• Natural language based machine learning
• Mechanisms of conversation maintaining and context awareness
5.4
Mathematical Model of Emotion Based Computing
It is possible to create a mathematical model of emotional decision making, which will be
integrated into agent as part of his inner architecture. The model will be firmly based on
ideas proposed by Marvin Minski in his ”Model of Six”, Tomkins psychological model and
neurobiological studies of Lovheim. Integration of neuromodulators effects on human perception
and reasoning and brain rhythms are also topics of discussion.
5.5
Common Sense Based Reasoning
Implementing human-like logical reasoning mechanism requires using approach based on common sense, as described in the literature by Marvin Minsky.
5.6
Generation of Psychological Attitude and Emotional Response
It is absolutely necessary that agent would recognize the person it is speaking to and act
accordingly. As such, we can define an essential minimum of cognitive functions that every
emotional intelligent agent should feature:
• Interlocutor recognition should work both in conditions of excessive and incomplete data,
with varying levels of lighting and angle of image rotation;
116
Anthropomorphic artificial social agent . . .
Kugurakova, Talanov, Manakhov and Ivanov
• Calculating behavior matrices with motivation, preparation and previous experience as
key parameters;
• Conversation maintenance and context awareness;
• Recognition of interlocutors mimics and intonation;
• Generation of psychological attitude to interlocutors;
• Generation of emotional response powered by sentiment analysis.
5.7
Computed Experiments and Validation
Final test that intelligent agent must pass is computational experiments such as well-known
Turing test and research of ”uncanny valley” phenomenon during conversations between agent
and human.
6
Conclusion
There are many intelligent agents and artificial intelligence technologies today, but none of them
can be identified as truly emotional based. We aim to fill this gap with our project.
So far we have described computation model which ties together human emotions and neuromodulator based computing and implemented the prototype of an intelligent emotional social
agent, which is able to exhibit various emotions as well as talking. We also determined further researches we will need to conduct in order to finish the said agent and bring its artificial
emotions to perfection.
7
Acknowledgment
This work was funded by the subsidy of the Russian Government to support the Program
of competitive growth of Kazan Federal University among world class academic centers and
universities.
References
[1] Nikoli D. Machines that dream: A brief introduction into developing artificial general intelligence
through ai-kindergarten. [online], 2015. http://www.danko-nikolic.com/wp-content/uploads/
2015/06/Machines-that-dream-Nikolic.pdf.
[2] Bratman M. E. Intention, Plans, and Practical Reason. CSLI Publications, 1999. ISBN: 1-57586192-5.
[3] Sergeev S. F. Methodological aspects of sophisticated technological environments design. 7-th
Annual Conference ”Aviation and osmonautics-2008”, 2008.
[4] NIH Blueprint for Neuroscience Research. The human connectome project). [online], 2013. http:
//neuroscienceblueprint.nih.gov/.
[5] Oppy G. and Dowe D. The Turing Test. Stanford Encyclopedia of Philosophy, 2011.
[6] Lovheim H. A new three-dimensional model for emotions andmonoamine neurotransmitters. Medical Hypotheses, pages 341–448, 2012.
[7] Minsky M. The Emotion Machine and The Society of Mind,NY. Simon & Schuster, 2006. ISBN:
0743276647.
117
Anthropomorphic artificial social agent . . .
Kugurakova, Talanov, Manakhov and Ivanov
[8] Talanov M. and Toschev A. Computational emotional thinking and virtual neurotransmitters.
IJSE, pages 1–8, 2014.
[9] Talanov M., Vallverdu J., Distefano S., Mazzara M., and D Radhakrishnan. Neuromodulating
cognitive architecture: Towards biomimetic emotional ai. AINA, pages 587–592, 2015.
[10] Plutchik R. Emotion: Theory, research, and experience: Vol. 1. Theories of emotion 1. Academic,
1980.
[11] Plutchik R. Emotions and Life: Perspectives from Psychology, Biology, and Evolution. American
Psychological Association, 2002.
[12] Plutchik R., Conte R., and Hope. ECircumplex Models of Personality and Emotions, Washington.
American Psychological Association, 1997.
[13] Aaronson S. My conversation with ”eugene goostman,” the chatbot that’s all over the news for
allegedly passing the turing test. The Blog of Scott Aaronson, 2014.
[14] Rao S. and Georgeff M. P. Modeling rational agents within a bdi-architecture. In Proceedings
of the 2nd International Conference on Principles of Knowledge Representation and Reasoning,
pages 473–484, 1991.
[15] Rao S. and Georgeff M. P. Bdi-agents: From theory to practice. In Proceedings of the First
International Conference on Multiagent Systems (ICMAS’95), 2012.
[16] Tomkins S. Affect imagery consciousness volume I the positive affects. Springer Publishing Company, 1962.
[17] Tomkins S. Affect imagery consciousness volume II the negative affects. Springer Publishing
Company, 1963.
[18] Tomkins S. The quest for primary motives: biography and autobiography of an idea. J Pers Soc
Psychol, 1981.
[19] Tomkins S. Affect imagery consciousness volume III the negative affects anger and fear. Springer
Publishing Company, 1991.
[20] Wan V., Anderson R., Blokland A., Braunschweiler N., Chen L., Kolluru B., Latorre J., Maia R.,
Stenger B., Yanagisawa K., Stylianou Y., Akamine M., Gales M.J.F., and Cipolla R. Photo-realistic
expressive text to talking head synthesis. Source of the Document Proceedings of the Annual
Conference of the International Speech Communication Association, INTERSPEECH, pages 2667–
5669, 2013.
[21] Yamada Y., Kawabe T., and Ihaya K. Categorization difficulty is associated with negative evaluation in the uncanny valley phenomenon. Japanese Psychological Research, pages 20–32, 2013.
118