A corpus of social interaction between Nao and elderly people

Mohamed A. Sehili1, Fan Yang1,2, Violaine Leynaert4, Laurence Devillers 1,3

1
Department of Human Communication, LIMSI-CNRS, Orsay, France
2
Department of Computer Sciences, University Paris 11, Orsay, France
3
University Paris-Sorbonne, Paris, France
4
Approche, Propara, Montpellier, France
E-mail: {sehili, fan.yang, devil}@limsi.fr

Abstract

This paper presents a corpus featuring social interaction between elderly people in a retirement home and the humanoid robot Nao.
This data collection is part of the French project ROMEO2 that follows the ROMEO project. The goal of the project is to develop a
humanoid robot that can act as a comprehensive assistant for persons suffering from loss of autonomy. In this perspective, the robot
is able to assist a person in their daily tasks when they are alone. The aim of this study is to design an affective interactive system
driven by interactional, emotional and personality markers. In this paper we present the data collection protocol and the interaction
scenarios designed for this purpose. We will then describe the collected corpus (27 subjects, average age: 85) and discuss the results
obtained from the analysis of two questionnaires (a satisfaction questionnaire, and the Big-five questionnaire).

Keywords: Human-Robot interaction, emotions recognition, elderly people interaction corpus

1. Introduction Since ELIZA's success [Weizenbaum, 1966] most

chatterbots have been emulating the same principles to
To effectively understand and model the behavioral overcome the Turing test. The Turing test is used as
patterns of very old people in presence of a robot, criterion of intelligence of a computer program and it
relevant data is needed. This kind of data is however not assesses the ability of the program to embody a human
easy to obtain and to share due to many factors. agent in a real-time conversation with a real human and
Databases recorded with young people might be easier to mislead them so that they cannot realize that they have
create but they do not meet the requirements of studies actually been talking to a machine. The basic idea of
like the one presented in this paper. ELIZA is the recognition of key words or phrases in the
input of the human subject and the effective use of these
This study takes place within the ROMEO2 research
words (or phrases) within preprepared or predefined
project (http://projetromeo.com/) which follows its
replies in order to push the conversation forward in a
precursor ROMEO [Delaborde & Devillers, 2010,
way that seems meaningful to the human. When an input
Buendia & Devillers, 2013] and whose goals are to
contains the words mother or son, for example, the
design a humanoid robot that can be used as an assistant
program's respond is typically Tell me more about your
for persons with loss of autonomy. The targeted
family [Weizenbaum, 1966].
population are elderly people living alone. In this work,
Thus, our human robot dialog has been designed in the
we present the first steps toward the design of an
same spirit of ELIZA. The main challenges are to give
interaction-driven system. We will present the data
the conversation a fairly good level of meaningfulness
collection protocol used to record conversations between
and make the elderly subject stick to the dialog as long as
Nao and 27 subjects, the dialogue strategy and an
possible. However, unlike ELIZA and all chatterbots in
analysis of two questionnaires used with each subject
general, as the robot is in the same room as the person,
after each interaction .
and hence visible to them, we have focused on the fact
that it should be viewed as an intelligent machine, not a robot-friends or I need to charge my batteries.
human.
2.2 Conversational scenarios
The conversation is split up into many independent
The goals of this work are:
scenarios that must be run in a specific order. Figure 1
Get a first feedback of elderly people
depicts an example of a social interaction between an
Validate and enhance the envisaged scenarios
elderly person and the robot Nao.
Getting a corpus for future research in
interaction
The scenarios of social interactions were:
Greetings
This paper will present the data collection protocol and
Reminder events: take medicine
used scenarios in Section 2, the collected data (27
Social interaction: call a relative
people, average age: 85) in Section 3, the results based
Cognitive stimulation: song recognition game
on the analysis of the two questionnaires (Satisfaction,
Big-five) in Section 4 and our first annotations of the
commitment level of the subjects (laughs, smiles, gazes,
etc.) (Section 5). Conclusion and perspectives will be
reported in Section 6.

2. Data collection protocol

2.1 Targeted type of data

To effectively carry out a study on elderly people robot
social interaction, some relevant data is needed. In fact,
this kind of data is rather rare and can neither be
collected in a laboratory nor from TV shows or phone Figure 1: Example of a social interaction between an
conversations [Castelano et al., 2010][Devillers, Schuller elderly people and the robot Nao
et al., 2012]. Furthermore, due to ethical and linguistic
issues, this type of content cannot be easily shared with In the first scenario (Introduction - greetings) Nao
other researchers. Relevant corpora must, from our introduces itself and announces its capacities (that it can
perspective, depict elderly subjects having spontaneous speak, sing and move) to spark the person's curiosity and
conversations with the robot. make them want to talk. It then asks the subject a few
personal questions that include name, age, how long they
To fulfill these requirements, our strategy was to seek the have been in this accommodation, if the person has a
desired population in an old people's retirement home, to family and so on. In the second scenario, Nao tries to
design a few interesting conversational scenarios that draw the person into more common social conversation
would encourage people to cooperate with the robot and subjects such as today's weather and their favorite
to use a Wizard of Oz (WoZ) scheme to control the robot games. In this scenario it asks them about the last meal
so that its behavior adapts seamlessly and quickly to they have had and which medication they should take. In
most situations. The retirement home is a French EHPAD the third scenario, Nao tries to talk about family and
(a public accommodation for non-autonomous old children for encouraging the person to call parents. In the
people) in Montpellier. We also have focused on the fact last scenario, whose main goal is to cognitively stimulate
that the robot should be viewed as an intelligent machine, the person, Nao tries to identify the subjects that might
not a human. Thus, many sentences are deliberately used be interesting for people such as movies, cooking, and
by Nao to emphasize this such as I have just come out TV programs. It then plays about thirty seconds of a few
of my box, I have just left the factory, I have many famous old French songs and asks them if they have
recognized the song's title or the performer's name. timestamped actions and can be used to rebuild the
dialog tree. Furthermore, it can be used to extract some
Using a Wizard of Oz, Nao is obviously remotely useful information such repeated utterances and the time
controlled by a human who observes the course of the spent by the person to react to an action of Nao etc.
conversions and reacts accordingly. The content of each
scenario is predefined and Nao (that is, the human Beside Nao's video camera and 4 microphones, we have
wizard) follow a conversation tree to perform the next used an HD webcam to capture facial expression (a
action (uttering a text, playing a song or performing a white screen was set up behind the person), a standard
gesture). Furthermore, thanks to Nao's text-to-speech HD camera to record the whole interaction from a profile
facility, the wizard can dynamically type and send new perspective and a lavaliere microphone to get an isolated
text such as the person's name during the conversation. high quality audio track.

In case of tricky situations when the person insists or not The number of subjects is 27 (3 men and 24 women),
following the conversation tree, the wizard uses generic recorded over two sessions (14 subjects in November
sentences (46 generic sentences such as it is true, 2013 and 13 in January 2014 respectively) making up
yes). The average number of phrases per session was around 9 hours of signal. The same hardware has been
82 sentences. The different number of sentences of the used for both sessions though each session has taken
WOZ is 265 (including the generic sentences) with lot of place in a different room. We also used two
empathic sentences such as I like you name. questionnaires for each subject.

2.3 Wizard of OZ
This study has been conducted with people who are not
The main goal of the WoZ is to take advantage of Nao's
under tutorship. They all agreed to participate to the
communication abilities and to build a social interaction
study and signed an authorization to use and reproduce
between the robot and elderly people. Therefore, the tool
the collected images and voice. To meet the researchers,
we used consists in a software with a GUI and is globally
each person was individually hosted in a room within the
designed to send the text utterances to Nao, perform
retirement home and was made aware of the ability to
gestures and play sounds (e.g. old songs). For the sake of
stop the experiment at any time.
spontaneity and quickness in Nao's reactions, almost all
speech utterances are encoded beforehand. Moreover, the 4. Questionnaires
human wizard can dynamically update a few snippets of After each interaction, two questionnaires have also been
text (e.g. the name of the person) or add a free text to used: a first satisfaction questionnaire meant to evaluate
keep an appropriately high level of conversation and the quality of the interaction with the robot and then a
match the subject's current theme if they do not stick to short version of the well known Big-five questionnaire.
the scenario. To make the use of free text as low as
possible, many generic utterances (e.g. Yes, No, I 4.1 Personality questionnaire
see, Can you hear me, I'm sorry, etc.) were made A very brief measure of the Big-Five personality domains
available for the wizard. Each scene in a scenario is built based upon the Ten-Item Personality Inventory (TIPI)
as dialogue tree. At each node the wizard has, according [Gosling et al., 2003] has been used. The questions relied
to the subject's reaction, to choose the next node of on the own perception of oneself in a variety of
dialogue to visit. situations. The subject is given a set of statements and
replies by indicating the strength of his agreement with
3. Corpus description each statement on a scale from 1 to 7 ( 1 denotes a strong
For this data collection we have mainly been focusing on disagreement, 7 denotes a strong agreement, and the
two modalities: audio and video. A log file is also other values represent intermediate judgments). For each
available for each conversation. It contains all Nao's subject, we computed a value for each of the five
dimensions which are Emotional Stability, Extroversion, (Q11) Doyou consider the robot as a machine
Openness, Conscientiousness and Agreeableness or as a friend or a companion (human)? The
answer was 52% for a machine and 48 % for a
4.2 Satisfaction questionnaire
friend and/or companion.
As for the satisfaction questionnaire, closed-ended
questions have been used. The subjects were also asked 5. Analysis of the questionnaires
to supply answers using a 7-scale evaluation scheme. In For a better understanding and interpretation of the
the following we report the satisfaction questions and the collected answers, a score of correlation is calculated
respective average scores for the 27 persons between between the personality traits and a few of the
parentheses: satisfaction questions. Correlations are calculated
(Q1) Did Nao understand you well? (5.2) between the satisfaction questions as well.
(Q2) Did it show any empathy? (6.3)
(Q3) Was it nice to you? (6.2) We used the Pearson product-moment correlation
(Q4) Was it polite? (6.4) coefficient with a permutation test (using the R
language). The most interesting correlation was with
For the open questions, we give a list of example answers Emotional Stability (see Table 1).
below. For convenience, the answers have then been
encoded into numerical values using different strategies.
Table1 shows the correlation between the emotional
For example, for Q6 we use 1 for human names and 0
stability of a subject and a number of questions that
for other names. Numerical values are used to calculate
reveal how the subject perceives the robot.
correlations between the satisfaction answers and
personality traits:

Question Corr. P-value

(Q5) What would be the best adjective to
describe the robot? (right, comic, nice, very Human/Non-human name (Q6) -0.31 0.1
nice, surprising, friendly, funny, sweet, Would you like to own a robot? -0.63 0.0003
pleasant) (Q9)
(Q6) Which name would you give to the robot?
Do you consider the robot as a 0.43 0.02
Some of the proposed names ( only 4 persons machine or as a friend or a
were not able to give a name): Pierre, Michel, companion (human)? (Q11)
Alfred, rigolo (comic), Zizou, Toto, Nano,
Nicolin, Jo, gentil (nice), patachou, a name of Table 1: Correlation between the Emotional Stability
an extraterrestrial, Mikey. personality trait and a few of the satisfaction questions.
(Q7) Would you like it to address you as tu
(using the familiar form) or as vous (using
It should be noted that a p-value under 0.05 indicates a
the formal form)? 55% of the subjects prefer
high significance level of the reliability of the correlation
the familiar form and 45% say that they have
between the two variables. The negative correlation
no preference. None prefers the familiar form.
between the Q6 and the emotional stability suggests that
(Q8) Would you agree to redo the test with the
subjects with a high emotional stability tend to give a
robot? 81.5 % of the subjetcs agree.
non-human name to the robot. Although no useful
(Q9) Would you like to own a robot? Only
conclusion can be learned from this correlation, due to a
26 % of the subjects agree.
high p-value, we can interestingly observe that subjects
(Q10) Would you prefer a robot that looks like
with a high emotional stability view the robot as
a robot or a human? 55% of the subjects prefer
machine, not as a human (Q11, third row of the table).
a human-like robot.
There is also a strong correlation between the emotional did they imitate it. Furthermore, many non-verbal cues
stability and the fact that the subject does not want to such laughter, smile, surprise have been annotated.
own a robot (row two). Annotations have been carried out by two experts on the
data collected in the first session (14 persons). As a first
Among the correlations found between the satisfaction
result, we report the presence of laughs and smiles from
questions, we mention three pairs of questions (Table 2).
all subjects through the conversation. A complete
From the first row in Table 2 one can find and obvious annotation of the whole corpus is being carried out. A
link between the understanding level of the robot and a more detailed annotation should also be done afterwards.
tendency to accept to redo the test. From the second row, It will include a more comprehensive annotation of audio
we can conclude that the more the robot is viewed as a and image cues.
machine, the less people want to have one. As for the
third row, we can learn that if people give a human name 7. Conclusion and future work
to the robot, they tend to agree when it comes to owning In this paper we present a data corpus of a social
a robot. interaction between a humanoid robot and elderly
Pair of questions Corr. P-value people. This work is part of the Romeo 2 project. The
Did Nao understand you well? 0.38 0.04 corpus contains 27 conversations with an average
(Q1) Would you agree to duration of 20 minutes.
redo the test (Q8)
To the best of our knowledge This kind of data is
Would you prefer a robot that -0.62 0.0004 relatively rare as it is a very challenging task to record
looks like a robot or a human?
people over 80 years old. Furthermore, it could be very
(Q10) Would you like to own
a robot? (Q9) difficult to share this kind of data.
The recorded subjects were furthermore asked to answer
Human/Non-human name (Q6) 0.41 0.03
Would like to have own a a set of questions from two different questionnaires.
robot? (Q9) They collected answers are used in this paper as first
evaluation data.
Table 2: Correlation between a couple of satisfaction As a first result, we can conclude from the answers that
questions. the majority of the subjects shown an interest in the
experiment. This could be backed by the presence of
6. Annotations laughs and smiles as observed in the first annotations.
Given the content of the corpus, there are many strategies
to annotate its content. Each strategy may apply to Moreover, many interesting correlation could be shown,
different levels of information. Annotations can apply to be that between the elements belonging to the same
both audio and video streams. For audio streams for questionnaire or to two different questionnaires. One of
instance, we can focus on non-verbal cues such as the major conclusions is that the way an elderly person
laughter, or cues that suggest that the person does not interacts with the robot depends on their personality.
understand what the robot is saying. For video streams, Therefore we intend in future work to dynamically build
attention can be given to visual cues on the face. Such a user profile and adapt the robot's behaviour
detailed annotations require both time and human effort. accordingly.

In this work we have followed a behavioral annotation The analysis of social interaction between elderly people
scheme. Thought has been given to the commitment and a robot allowed us to get a first feedback of the
level of the subject during conversation. In this regard, concerned people and to validate and enhance our
we have been interested on how much a subject looked at interaction scenarios. This corpus will be used as an
the robot, how well did they understand it and how much evaluation data for further experiments.
 8. Thanks

Thanks are due to all the participants to these

experiments (especially to the EHPAD of Montpellier)
and to the partners of the ROMEO2 project for the
interest they expressed for this data collection.

9. References
A.Buendia, L. Devillers (2013). From informative
cooperative dialogues to long-term social relation with
a robot. Towards a Natural Interaction with Robots,
Knowbots and Smartphones, IWSDS 2013, Springer.
G. Castellano, I. Leite, A. Pereira, C. Martinho, A. Paiva,
and P.W. McOwan (2010). Affect recognition for
interactive companions: challenges and design in real
world scenarios. Multimodal User Interfaces, 2010.
A.. Delaborde, L. Devillers. (2010) Use of Nonverbal
Speech Cues in Social Interaction between Human and
Robot: Emotional and Interactional Markers, in
AFFINE '10: Proceedings of the International
Workshop on Affective-Aware Virtual Agents and
Social Robots, ACM, October 2010.
L. Devillers, B., Schuller, A., Batliner, P., Rosso, E.,
Douglas-Cowie, R., Cowie, and C. Pelachaud, editors
(2012). Proceedings of the 4th International Workshop
on EMOTION SENTIMENT & SOCIAL SIGNALS
2012 (ES 2012) Corpora for Research on Emotion,
Sentiment & Social Signals, Istanbul, Turkey. ELRA,
ELRA. held in conjunction with LREC 2012.
S. D. Gosling, S. D., P. J. Rentfrow, P. J., W. B Swann,
W. B., Jr. (2003). Ten-Item Personality Inventory
(TIPI). A Very Brief Measure of the Big Five
Personality Domains. Journal of Research in
Personality, 37, 504-528
J. Weizenbaum, (1966), "ELIZAA Computer Program
For the Study of Natural Language Communication
Between Man And Machine", Communications of the
ACM 9 (1): 3645