Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition. arXiv:2103.09072v1 [cs.RO] 16 Mar 2021 Jonas Gonzalez-Billandon*1 , Giulia Belgiovine*2 , Alessandra Sciutti1 , Giulio Sandini2 , Francesco Rea2 ©2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Abstract— The ability to recognize human partners is an in the partner’s affective state [7]. important social skill to build personalized and long-term human-robot interactions, especially in scenarios like education, care-giving, and rehabilitation. Faces and voices constitute two important sources of information to enable artificial systems to reliably recognize individuals. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. However, when those networks are applied to different and unprecedented scenarios not included in the training set, they can suffer a drop in performance. For example, with robotic platforms in ever-changing and realistic environments, where always new sensory evidence is acquired, the performance of those models degrades. One solution is to make robots learn from their first-hand sensory data with selfsupervision. This allows coping with the inherent variability of the data gathered in realistic and interactive contexts. To this aim, we propose a cognitive architecture integrating low-level perceptual processes with a spatial working memory mechanism. The architecture autonomously organizes the robot’s sensory experience into a structured dataset suitable for human recognition. Our results demonstrate the effectiveness of our architecture and show that it is a promising solution in the quest of making robots more autonomous in their learning process. giulia.belgiovine@iit.it It becomes therefore increasingly necessary for robots to be able to reliably recognize the partners they have interacted with. Advances in artificial intelligence (AI), and more specifically machine learning (ML), provide today potentially powerful tools to perform human recognition. Recently, several recognition algorithms using visual information and based on deep learning networks (e.g. face recognition algorithms) have been proposed achieving stateof-the-art results and establishing themselves as the preferred techniques to address such task [8]. However, non-visual cues such as voice represent also important information for person recognition, especially when the person is not in the visual field of view or when vision is limited. Deep learning techniques have proved to be a robust tool also to address speaker recognition, achieving state-of-the-art results [9]. Nonetheless, several issues arise when applying AI methodologies to visual and non-visual information gathered in HRI frameworks. The problems are mostly caused by the scarcity of available data in these settings. Also, the diverse nature of the data stream, along with the inherent variability related to the robotic platform or the interactive context, could complicate the use of established AI solutions. Deep Learning algorithms work well as long as the data distribution they were trained on is similar to the distribution they will be used on. For robots evolving in real-world scenarios, there is often a mismatch between the training and the target data distribution, mainly due to the use of different sensors (e.g., microphones and cameras) and to the noise in the environment or generated by the robots [10]. Considering the example of face recognition, with the advent of the COVID-19 pandemic, classical face recognition approaches hardly work with the previous solutions, because during interactions the faces are covered by masks. To cope with this issue, it would be necessary to collect new ad hoc datasets and to re-train the model, e.g. by using domain adaptation techniques. However, this represents a strong limitation, as the collection and creation of these datasets are time-consuming as they frequently require human annotation. 1 Cognitive Architecture for Collaborative Technologies lab, Italian Institute of technology, Genova, Italy 2 Robotics Brain and Cognitive Science lab, Italian Institute of technology, Genova, Italy In order to overcome this limitation and to increase the amount of relevant data that a robot could leverage, I. INTRODUCTION The purpose of today’s robotics is to support humans in diverse contexts which range from functional collaboration, as in manufacturing, to personal relationships, as in caregiving, education, and rehabilitation scenarios. Especially in contexts like the latter, human-robot interaction (HRI) needs to reach the next level of complexity, with robots capable not only to efficiently interact but also to account for the diversity among individuals [1]. To this end, it is necessary to personalize interactions based on the memory of shared experiences and let the robot adapt its behavior considering the history of previous interactions with humans [2], [3]. Indeed, personalization and adaptation are keys to successful long-term HRI [4]. These skills can improve users’ engagement, making it last even after the novelty effect wears off [5]. They can also help to establish a rapport and trust between the robot and users, for example by recalling shared memories [6] or by detecting changes *Co-authors tion of that this contributed equally in the realisapaper jonas.gonzalez@iit.it, it is crucial to exploit the robot’s embodiment in the real world. The advantage lays in perceiving directly the world around it and continuously updating the sensorial data from the environment. The open challenge for the robotics community is therefore to design frameworks aimed at autonomously and effectively managing the data captured by the robot during recurrent interactions with humans (i.e., ”its experience”) so that it can use this information to learn in an autonomous and self-supervised way. The robot’s ability to organize its own sensorial experience and to retrieve it in future contexts is of utmost importance for personalized, long-term HRI scenarios, where the amount of information grows continuously [11]. This is especially advantageous for interactions involving multiple people, behaving dynamically. To this purpose, we propose a solution based on a cognitive architecture that exploits the robot’s low-level perceptual mechanisms (e.g., sound localization and face detection) and working memory that endows the robot with the ability to autonomously organize its experience while interacting with humans. Cognitive robotics focuses, among many capabilities, on designing and building cognitive models for robots that have the ability to learn from experience and from others. In particular, the ability to retrieve experiences from memory as the context requires, and flexibly reuse this knowledge to select appropriate actions in the pursuit of inner goals [12] is fundamental for cognitive robotics. In this process, memory plays a key role, as it helps the agent to encode, store, and retrieve knowledge. In particular, working memory refers to the capacity to maintain temporarily a limited amount of information, which can then be used to support various work-dependent skills, such as learning, reasoning, planning, and decision-making processes. In other words, working memory is not just a retention mechanism, but it is also a framework that preserves transient goalrelevant pieces of information in the service of complex cognitive processes [13]. The interest in exploring the role of different types of memory for human-robot interaction, with either embodied or virtual agents, is progressively gaining the attention of the community [14], [15]. However, its implementation in the robotic field for autonomous interactions with humans is still limited to few interesting researches [16], [17], [18]. In this work, we propose to use working memory and perceptual system together, to let the robot interact autonomously with multiple partners and cluster multi-modal data, by collecting them into a structured dataset suitable for the tasks of speaker and face recognition. After introducing our architecture, we present its implementation in a gamelike interaction scenario involving the humanoid iCub and three participants simultaneously and test it in a user study on a total of 12 participants. Finally, we report the results of testing different models for face and speaker recognition on our dataset, to analyze the pros and cons when implementing different methodological solutions on our limited and noisy dataset. II. COMPUTATIONAL ARCHITECTURE We present in this section the different components of the architecture (Fig. 1) used to learn in a self-supervised way to recognize persons during a natural human-robot interaction. All the different components are stand-alone modules and are integrated with the YARP middleware [19] which provides a modular design of the cognitive architecture. Fig. 1. Representation of the computational architecture: The raw data from the iCub sensors are passed towards perceptual mechanisms which allow the extraction of information such as faces and voice location. The proprioceptive information of the robot along with the spatial memory system allow saving this information into a structured dataset. A. Perceptual Modules 1) Sound Localization System (SLS): We used a sound localization system (SLS) to discern the incoming direction of auditory signals and thus clustering voices coming from sources differently placed in the space, with the assumption that the scene changes only in specific time instances. Once the relative position of the people is known in the space, the robot could infer the identity of the speaker by mapping the direction from which the voice comes to the information stored in the spatial memory. In this specific case, the direction of the sound has been discretized in three regions of the 180 degrees range of motion relative to the robot: left, right, and in front of it (see Fig. 2, top panel). 2) Hot-word detector: To make the participant interact naturally and for a prolonged period without human intervention in the different phases of the game, we used Snowboy, an artificial intelligence software toolkit developed by KITT.AI for hot word detection1 . We used the software to detect the words yes and no to automatically ask the players if the answer provided by another player was right or not. This allowed us to fully automatize the interaction without the need to rely on experimenter intervention or a Wizard-of-Oz approach. 1 https://github.com/Kitt-AI/snowboy 3) Face Detector: To identify the participants interacting with the robot we use a deep learning-based face detection system developed for the iCub robot [20]. The face detector was trained in a self-supervised way and it is inspired by how infants learn to detect faces using audiovisual attentional mechanisms to guide their learning. 4) Multiple Object Tracker (MOT): The face detection module allows for the localization of faces from the robot camera but cannot perform face identification. That is, from a sequence of frames, it cannot associate a currently detected face with a previously seen one. However, for the robot it is strictly necessary to keep track of an object’s position in time and to distinguish it from other objects. To perform data association, we used a multi-object tracker algorithm based on the combination of a Kalman Filter and Hungarian algorithm. The combination of the modules is used for the tracking components and allows an accuracy comparable to state-ofthe-art online trackers, but with real-time performance, which is crucial for robotic application [21]. We used the MOT module to associate for each participant interacting with the robot a unique tracker T , which is maintained during the whole task (see Fig 3). D. Data collector Module The combination of the perceptual module with the working memory allows the robot to store and update the position of the persons in the space throughout the interaction. This information is used to autonomously organize the sensory data (i.e., faces and voices) into a structured dataset: Dataset : {Pi : [(Ii , x1 , y1 , x2 , y2 ); ...; (In , x1 , y1 , x2 , y2 )] ∪ [audioi ; ...; audion ]]} For each participant Pi the module stores automatically the images and the associated bounding box of the face (Ii , x1 , y1 , x2 , y2 ) retrieved through the face detector, and correctly gives a label using the MOT and the memory module. Moreover, when the robot hears a person speaking, the system stores the detected audio associating it to the corresponding person Pi , thanks to the combination of the SLS module, which identifies the sound direction, and the memory module that maps the proprioceptive information of the robot to the identity of the memorized players. III. MATERIALS AND METHODS B. Action Module To perform the tracking of the faces and orient the head of the robot toward the localized sound source given by the SLS module output, we used the iKinGazeCtrl module: a controller for the iCub gaze capable of steering the neck and the eyes independently [22]. To execute the different arms motion of the robot we used pre-programmed sequences of full-body movements in joint space to have perfectly replicable movement comparable across all the participants. The Game supervisor module managed the transition between specific states of the state machine, by triggering different robot behaviors (see Fig. 4). A. Participants The experiment was conducted with the support of 12 healthy participants that voluntarily agreed to participate to the data collection (8 males, 4 females, 27.25±1.48 years of C. Spatial/Working Memory Module While interacting with others, the robot has the advantage to be embodied in the world and thus it can use alternative supervisory mechanisms to organize its incoming signals. For example, when a robot faces different speakers it can use the proprioceptive information of its head orientation to cluster the face and voice of the person. We used a spatial memory module representing working memory to allow iCub to map a limited number of persons interacting with it to their spatial positions according to an egocentric frame of reference. We implemented the spatial working memory using a Dictionary data structure: D : Keys → V alues ∪ ǫ (1) Where the keys are the azimuth values of the robot head, the values is a list of instances of trackers T and ǫ is the empty set. In our implementation, we discretized the azimuth range [-90, 90] into three bins: {left; center; right} each 60 degrees wide (see Fig 3). Fig. 2. Top panel: Scheme of the experimental setup during the game phase. Bottom panel: Participants and the iCub robot while playing the history taboo game. The third participant was present to the right of iCub but is not visible in this image. Fig. 3. Demonstration of the Spatial Memory (top) and the MOT tracking module (bottom): (A) At the beginning of the interaction the three participants are memorized on the left position by the memory module. (B) After the positioning phase, the memory module is updated by the MOT module, and the players are successfully represented in their corresponding spatial positions according to the robot frame of reference. age), divided in 4 groups of 3 participants each (1 female and 2 males). B. Experimental Design Participants were asked to play a modified version of the taboo game (where the words to be guessed, differently from the traditional taboo game, were historical figures) with iCub, a humanoid robot with a child appearance [23]. The objective of the game was to guess the identity of historical figures after one of the participants described them without pronouncing some keywords important for the identification (the ”taboo words”). Also, participants scored proportionally to the number of cards correctly guessed. The robot’s role was to guide participants through the different stages of the game, without actively competing by describing cards or guessing (see Fig 4). It was explained to the participants that since iCub is a child robot, it could not explain the characters in human history; however, it could learn by participating as a referee and supervisor of the game. The choice of this experimental design stems from the need to design an engaging interaction that, at the same time, could well represent a real-world scenario where a robot supervises and modulates interaction in a group of multiple users, such as robot teachers in a class of students or robot therapists in a rehabilitation session. C. Protocol and State-Machine Upon their arrival, participants were first brought to the office setting, where they were assigned a color-player label among blue, green, and red, that was later necessary during the game phase. They were also instructed that once in the experiment room, they should call iCub attention with their voice to make the interaction start. Indeed, using the SLS module, iCub turned its gaze toward the direction of participants’ voice and once it detected faces through the Face Detector module (Fig. 3A, bottom panel) it started the interaction by welcoming them and updating the Spatial Memory module with the position of the detected faces (Fig. 3A, top panel). Fig. 4. State machine followed by the robot during the game. In the next phase Player Positioning (Fig. 4), iCub invited one color-player at a time (in random order) to approach the table placed in front of it and to take the deck of cards of the corresponding color, containing the historical figures with the associated taboo words. Then, it asked the participant to take other positions in the room by pointing with its arm on its right, front, or left. In this phase, iCub associated the color-player with the tracked face, which was stored and labeled to be organized in the database through the Data Collector module (Fig. 3A, bottom panel). As the participants moved towards the position, iCub followed them with its gaze by using the MOT module. When the robot detected participants stopping, it updated their position in the space through the Spatial/Working Memory module. In this way, each color-player was mapped to a position relative to the robot reference frame (Fig. 3B, top panel). Then, the Players Presentation phase began and iCub addressed one player at a time to ask them their name and to invite them to present themselves in a pre-set time of 20 seconds. This phase strongly relied on the role of memory and had multiple purposes: i) to automatically retrieve the person’s name for the creation of the database and the recognition phase, replacing the player-color label with it, and ii) to prepare a framework already suitable for future one-shot learning scenarios, where one needs to have initial reference samples to implement on-the-fly personrecognition models. At this point, the actual history taboo game started (Start Game phase). On a screen placed behind the robot, a colored circle appeared giving the signal to the player with the corresponding color to prepare to describe the card. After few seconds, a 30 seconds timer started, during which the player had to describe the card to the other players. During this Card Description phase, we intentionally deprived the iCub of a priori knowledge of the players’ turns. This replicates common interactive contexts in realworld scenarios where the robot has to orient its attention in the scene relying exclusively on its perceptual mechanisms. In the game, it turned its gaze toward the direction inferred by the SLS module, it associated the detected voice and face to the color-player label by querying its spatial memory, and it stored them in memory with the corresponding label using the data collector module. At the end of the timer, iCub interrupted the player by saying: ”Time’s up. I wonder if anyone has figured out what this is all about?”. In the Answer phase, the player who wanted to answer, at the iCub signal had to grab the buzzer placed in the middle of the table and call iCub attention with his/her voice. Once iCub turned its attention to one of the players (with the same mechanisms described in the previous phase), it invited the person to give the answer, and then it turned again to the previous player to verify whether it was correct or not. To make the interaction completely autonomous, we exploited the Hot-word detector module to detect the words ”yes” and ”no”, as explained in section II. Also, during the interaction, iCub showed different facial expressions depending on the context through LEDs representing its eyebrows and mouth. The game was repeated until each player had described a total of three cards. Each session of interaction lasted about 30 minutes. The state-machine flow and robot behaviors were completely autonomous and triggered by events that relied solely on the robot’s perception (e.g., sound or face detection) and game dynamics (e.g., timers). The experimenter monitored the correct progress of the experiment from a different room, ready to intervene only in case of problems. IV. DATA ANALYSIS A. Database generation During each interaction, the architecture stored autonomously the different data points of faces and voices by associating them with the corresponding player by using the color labels coding. In a post-processing phase, the Data Collector module organized the visual and auditory data by creating a structured dataset as described in previous sections. The dataset is suited to perform speaker and face recognition through supervised-learning. In fact, the dataset was populated by creating labels corresponding to the name of each player, extracted from the audio files saved in the Players Presentation phase, by using the Google-Speech recognition API. For each of these labels (12 in total), two different structured datasets were created: Dataset f aces and Dataset voices, described in detail below. In other words, the Data Collector module reorganizes the robot’s temporary experience into a structured knowledge about people who interacted with it, creating a memory made up of unique names, faces, and voices. Through the experiment, we collected a total of 1320 seconds of audio files (110 seconds on average for each participant), and 1200 samples of face images (93 on average for each participant, see Table I). The audio recorded presented an SNR equal to 9 Db due to the high ego-noise of the robot produced by the fans in its head (q realistic and undesired characteristic of typical robotic applications). We pre-processed each face according to the face alignment procedure for the face recognition network [24]. The face alignment procedure consists of centering the faces bounding boxes and resizing them to a fixed dimension of 180x180 pixels (see Fig 5). B. Triplet-Loss network When addressing the task of face or speaker recognition one solution is to rely on triplet-loss trained deep networks. The triplet loss approach comes from the name of the loss function used to train a deep network to produce euclidean embeddings with the goal to minimize the distance between an anchor and a positive sample and maximize the distance between the anchor and a negative sample of a different identity [25]. Moreover, as the triplet loss enforces negative samples to be distant in the embedding space, the network can be used to identify previously unseen persons by using similarity or distance measures. We investigated the performance of pre-trained triplet-loss networks on our collected datasets. For speaker recognition, we used a Resnet network trained on the LibriSpeech dataset [26]. We used the network to create a database of embeddings (Demb−voices ) for each person in the Dataset voices. For the faces, we used the Facenet network [24], trained on the large-scale face dataset VGGFace2 [27], to produce the database of face embeddings (Demb−f aces ) from the Dataset f aces. To perform the identification, we used a KNN algorithm with k = 1 to quickly find the nearest elements from the embeddings database. We then used the returned distances and a threshold t (empirically chosen) to classify the person as the nearest embedding class (distance > t) or as an unknown person (distance < t). C. Multi classification We also investigated the performance of a multiclassification approach on our datasets. For the speaker recognition, we trained a deep network based on stacked convolutional layers with gammatonegrams features as input. Gammatonegrams decompose a signal by passing it through a bank of gammatone filters equally spaced on the ERB scale and were designed to model the human auditory system. We opted for gammatonegrams as they have proven to be robust to noisy environments [28] and thus suited to deal with robot data. We pre-processed the Dataset voices by taking chunks of audio of 1 second with a hop-length of 250ms and we discarded chunks having less than 80% of voice data using the Google voice activity detector. Finally, we generated the gammatonegrams on the chunks using a bank of 128 filters and stacked horizontally the two channels, giving a final feature size of 128x192. Instead, for the face recognition model we opted for a transfer learning approach and fine-tuned the Facenet network to perform multi-classification task. D. Test Dataset To test the proposed models for voice and face recognition, we asked participants to come back after few days in order to acquire additional data via the robot’s sensors. At this stage, there was no actual interaction between the participants and the robot, as our goal was to demonstrate the robustness of our approaches using a dataset acquired under different conditions of the environment (lighting, etc..) and of participants (see as example Fig. 5). The approach better approximates the application contexts in which the model will be deployed. We also collected face and voice data of 4 people who did not participate in the experiment, to test the accuracy of the triplet-loss models in recognizing unseen samples. For both the faces and the voices, we applied the same pre-processing described in sections IV-C, IV-B. The final test-set resulted in 6 audio samples (of 1 second) and 10 faces per person (see Table I). TABLE I D ETAILS OF THE COLLECTED DATA DURING THE GAME INTERACTIONS USING OUR PROPOSED ARCHITECTURE Dataset type Avg data points per persons Avg size Faces train set Faces test set 93 10 32x57 34x58 Voices train set Voices test set 110s 6s - API we successfully extracted the different names of the participants gathered during the Player Presentation phase with an accuracy of 83.3%. Indeed, out of the twelve participants who played the game, one player was not correctly recognized (Parco instead of Marco) and one player could not be correctly processed by the software and was therefore automatically labeled as ”unknown”. V. RESULTS Fig. 5. Samples from the collected dataset: For each person, the right photo is a sample from the training dataset (collected during the game interaction) and the left one from the test dataset. A. Architecture and data collection Our architecture proved to be efficient in autonomously labeling and clustering visual and auditory data during a dynamic interaction with multiple people. On a total of 36 trials, the SLS module showed an accuracy of 97%. The robot failed in predicting sound direction - and consequently the player label associated to that direction by the memory module - only once across all trials. Regarding the automatic creation of the datasets, using the Google speech recognition Fig. 6. Face Recognition classification results. Top: Triple-loss model positive accuracy. Bottom: Re-trained model with the data collected during the interaction using transfer learning. B. Triplet-loss networks To asses the performances of the triplet-loss networks for both the faces and voices, we computed the positive accuracy and negative accuracy: P ositive accuracy = T P/(T P + F N ) N egative accuracy = T N/(T N + F P ) We tested both triplet-loss networks on the gathered testset to compute the positive accuracy and further tested on unseen participants to compute the negative accuracy (see IV-D). We used the Euclidian distance and fine-tuned the threshold t to balance between the positive accuracy and negative accuracy. For the face recognition, using a t = 0.4 the triplet-loss network achieved a positive accuracy of 55% (see Fig 6) and a negative accuracy of 73%. Instead, for the voices the best compromise found between the positive accuracy and the negative accuracy was 30.3% and 37.5% respectively using t = 1.7 . Fig. 7. Speaker recognition classification results. Top: Triplet-loss model test positive accuracy. Bottom: New model based on gammatonegrams test accuracy C. Multi-classification networks We re-trained the Facenet network for a multiclassification task on the 12 persons present in our dataset. The re-trained network achieved on the test set (see IV-D) an accuracy of 72% (Fig 6 bottom) against a chance level of 8%. For the audio, we re-trained the model to classify the 12 persons plus an extra class, background, which we obtained by recording the ego-noise of the robot. The final model reached a total accuracy of 65% on the voice test set. VI. DISCUSSIONS In this work, we proposed an architecture to enable robots to organize autonomously multimodal data gathered while interacting with a group of human partners. The objective was to efficiently synthesize the knowledge necessary to replicate a person recognition system and build natural HRI interactions based on the history of users’ experiences. To achieve this, we built an architecture based on basic perceptual mechanisms: the localization of sounds in space, the detection of faces and the tracking of moving objects. These are primary cognitive processes that can be exploited by any embodied agent in real-life contexts. These basic perceptual mechanisms were integrated with a working spatial memory system. The spatial memory allowed the robot to associate each user to a specific spatial location relative to the robot’s egocentric frame of reference and consequently to infer the user’s identity from the sensorial data coming from the specific environmental region. We demonstrated that the proposed architecture worked efficiently in a prolonged game-like interaction between the robot and multiple persons. Using our architecture, we successfully created a dataset of people who interacted with the robot without the need for human intervention. The robot was not able to correctly recognize the name of participants in few situations, which can be solved in the future by enriching the interaction with advanced skills (e.g., by letting the robot ask for the user’s name again when the output of the software is not reliable). We used the obtained dataset to develop a face and speaker recognition system exploring two different approaches: pretrained triplet-loss models and re-trained multi-classification models. The processing of data and training of the models relied on post hoc analysis for a preliminary evaluation of the recognition methods, but in the future, they could be implemented in a fully automated pipeline process using DevOps tools. The obtained results demonstrated a higher accuracy for the re-trained models, which was to be expected in light of the data collected by the robot. Indeed, the low-resolution faces images, the presence of the mask, and the low SNR of the audio explained the poor performance of the tripletloss networks which were trained on very different sets. For example, the LibSpeech contains clean speech of people without noise and in the VGGDataset faces with masks were not present. This demonstrates the advantage of using the data directly collected by the robot during its interaction with the world and of developing frameworks able to manage and manipulate the on-line robot experience. Thanks to this, the robot can learn through a self-supervised strategy while interacting with humans and with the world. Indeed, the effectiveness of such approaches is likely to improve as the number of interactions, and thus of the available data, increases. Moreover, with more data available it could be possible to train a triplet-loss network from scratch and fully take advantage of this approach. Considering the limitations of our dataset collected during a real interaction, as we expected, the obtained results are hardly comparable to the state-of-the-art face/speech recognition models. Nonetheless, this was not our primary goal. Rather our intent was to demonstrate the robustness of our approach to the variability of natural interactions with humans. Such variability may derive from several, often unpredictable sources such as the different conditions of the environment, the inherent noisy nature of data, or the dynamic changes of the interactive context. Consider for example the limitations of trying to implement a facial recognition system during the pandemic, where the faces covered by masks are difficult to distinguish. In these scenarios, it is important that the robot exploits its direct experience to learn how to cope with these situations and adapt accordingly, for example by relying more on the voice data. To conclude, our proposed architecture represents a novel contribution to the development of robots capable of autonomously and efficiently managing their sensory experience to build a multimodal memory of the people they have interacted with. This memory could be improved so that a robot could autonomously learn to manage the amount of data collected, by synthesizing it and by selectively removing the non-necessary information, similarly to what humans do. The proposed framework proved to be accurate and functional to manage a fully autonomous interaction between the robot and a group of people. Moreover, our approach facilitates building personalized and adaptive HRI. Indeed, having a history of the facial, auditory, and contextual features of users in a long-term memory could allow the robot to relate the current experience to past experiences and for example to monitor changes in the partner’s inner and affective state, health, or mood - adapting its behavior accordingly. ACKNOWLEDGMENT A.S. is supported by a Starting Grant from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme. G.A. No 804388, wHiSPER. R EFERENCES [1] A. Tapus, C. Ţăpuş, and M. J. Matarić, “User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy,” Intelligent Service Robotics, vol. 1, no. 2, pp. 169–183, 2008. [2] G. Belgiovine, F. Rea, P. Barros, J. Zenzeri, and A. Sciutti, “Sensing the partner: Toward effective robot tutoring in motor skill learning,” in International Conference on Social Robotics. Springer, 2020, pp. 296–307. [3] A. Vignolo, A. Sciutti, and J. Michael, “Using robot adaptivity to support learning in child-robot interaction,” in International Conference on Social Robotics. Springer, 2020, pp. 428–439. [4] M. Ahmad, O. Mubin, and J. Orlando, “A systematic review of adaptivity in human-robot interaction,” Multimodal Technologies and Interaction, vol. 1, no. 3, p. 14, 2017. [5] D. Szafir and B. Mutlu, “Pay attention! designing adaptive agents that monitor and improve user engagement,” in Proceedings of the SIGCHI conference on human factors in computing systems, 2012, pp. 11–20. [6] J. Saez-Pons, D. S. Syrdal, and K. Dautenhahn, “What has happened today? memory visualisation of a robot companion to assist user’s memory,” Journal of Assistive Technologies, 2015. [7] G. Gordon, S. Spaulding, J. K. Westlund, J. J. Lee, L. Plummer, M. Martinez, M. Das, and C. Breazeal, “Affective Personalization of a Social Robot Tutor for Children’s Second Language Skills,” Tech. Rep. 1, mar 2016. [Online]. Available: www.aaai.org [8] G. Guo and N. Zhang, “A survey on deep learning based face recognition,” Computer Vision and Image Understanding, vol. 189, p. 102805, dec 2019. [9] D. Sztahó, G. Szaszák, and A. Beke, “Deep learning methods in speaker recognition: a review,” Tech. Rep. [10] N. Sünderhauf, O. Brock, W. Scheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford, and P. Corke, “The limits and potentials of deep learning for robotics,” The International Journal of Robotics Research, vol. 37, no. 5, pp. 405–420, 2018. [11] G. Sandini, V. Mohan, A. Sciutti, and P. Morasso, “Social cognition for human-robot symbiosis—challenges and building blocks,” Frontiers in neurorobotics, vol. 12, p. 34, 2018. [12] D. Vernon, S. Thill, and T. Ziemke, “The Role of Intention in Cognitive Robotics,” Tech. Rep. [13] A. Baddeley, “Working memory: theories, models, and controversies,” Annual review of psychology, vol. 63, pp. 1–29, 2012. [14] M. Persiani, A. M. Franchi, and G. Gini, “A working memory model improves cognitive control in agents and robots,” Cognitive Systems Research, vol. 51, pp. 1–13, 2018. [15] L. Martin, K. Jaime, F. Ramos, and F. Robles, “Declarative working memory: A bio-inspired cognitive architecture proposal,” Cognitive Systems Research, vol. 66, pp. 30–45, 2021. [16] D. Stachowicz and G.-J. M. Kruijff, “Episodic-like memory for cognitive robots,” IEEE Transactions on Autonomous Mental Development, vol. 4, no. 1, pp. 1–16, 2011. [17] J. Rothfuss, F. Ferreira, E. E. Aksoy, Y. Zhou, and T. Asfour, “Deep episodic memory: Encoding, recalling, and predicting episodic experiences for robot action execution,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 4007–4014, 2018. [18] Z. Kasap and N. Magnenat-Thalmann, “Towards episodic memorybased long-term affective interaction with a human-like robot,” in 19th International Symposium in Robot and Human Interactive Communication. IEEE, 2010, pp. 452–457. [19] G. Metta, P. Fitzpatrick, and L. Natale, “YARP: Yet another robot platform,” 2006. [20] J. Gonzalez-billandon, A. Sciutti, M. Tata, G. Sandini, and F. Rea, “Audiovisual cognitive architecture for autonomous learning of face localisation by a Humanoid Robot.” 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 5979– 5985. [21] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “SIMPLE ONLINE AND REALTIME TRACKING,” Tech. Rep., 2017. [Online]. Available: https://github.com/abewley/sort [22] A. Roncone, U. Pattacini, G. Metta, and L. Natale, “A cartesian 6dof gaze controller for humanoid robots.” in Robotics: science and systems, vol. 2016, 2016. [23] G. Metta, L. Natale, F. Nori, G. Sandini, D. Vernon, L. Fadiga, C. von Hofsten, K. Rosander, M. Lopes, J. Santos-Victor, A. Bernardino, and L. Montesano, “The iCub humanoid robot: An open-systems platform for research in cognitive development,” Neural Networks, vol. 23, no. 8-9, pp. 1125–1134, oct 2010. [24] F. Schroff and J. Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” Tech. Rep. [25] K. Q. Weinberger and L. K. Saul, “Distance Metric Learning for Large Margin Nearest Neighbor Classification,” Tech. Rep., 2009. [26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210. [27] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “VGGFace2: A dataset for recognising faces across pose and age,” in Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc., jun 2018, pp. 67–74. [28] H. K. Maganti and M. Matassoni, “Auditory processing inspired robust feature enhancement for speech recognition,” in International Joint Conference on Biomedical Engineering Systems and Technologies. Springer, 2011, pp. 205–218.