SenToy: an affective sympathetic interface

Int. J. Human-Computer Studies 59 (2003) 227–235 SenToy: an affective sympathetic interface Ana Paivaa,*, Marco Costaa, Ricardo Chavesa, Moise! s Piedadea, . b, Da! rio Mour*aoa, Daniel Sobrala, Kristina Ho. ok Gerd Anderssonb, Adrian Bullockb a IST-Technical University of Lisbon and INESC-ID, Rua Alves Redol, 9, 1000-029 Lisboa, Portugal b Swedish Institute of Computer Science (SICS), Box 1263, SE-164 29 Kista, Sweden Received 23 December 2002; accepted 20 January 2003 Abstract We describe the design and implementation of SenToy: a tangible doll with sensors that allows a user to influence the emotions of a synthetic character in a game. SenToy is an input device that allows the user to perform gestures or movements that the sensors inside the doll pick up. The gestures are interpreted according to a scheme found through two different user studies: one Wizard of Oz study and one study with a fully functioning SenToy. Different gestures express one of the following emotions: anger, fear, surprise, sadness, gloating and happiness. Depending upon the expressed emotion, the synthetic character in the game will, in turn, perform different actions (trading, duelling, etc.). The evaluation of SenToy acting as the interface to the computer game FantasyA has shown that the users were able to express the desired emotions to influence the synthetic characters, and that overall players liked the doll as an interface. r 2003 Elsevier Science Ltd. All rights reserved. Keywords: Affective Interactions; Tangible interfaces 1. Introduction During the last few years, the research area of synthetic life-like characters is rapidly changing and characters are becoming more and more believable. In parallel, the interaction between users and those synthetic characters is becoming more *Corresponding author. . gerd@sics.se (G. Andersson), E-mail addresses: ana.paiva@inesc.pt (A. Paiva), kia@sics.se (K. Ho. ok), adrian@sics.se (A. Bullock). 1071-5819/03/$ - see front matter r 2003 Elsevier Science Ltd. All rights reserved. doi:10.1016/S1071-5819(03)00048-X 228 A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 natural and more inspired in the way we communicate. The Safira project was an EU funded project aiming at building novel affective interfaces. In Safira we have developed SenToy, a tangible interface that allows the user to influence the emotions of his synthetic character (avatar) in a 3-D game. Sentoy is an input device to a role playing game (FantasyA) where players must exhibit a particular set of emotions and perform a set of actions as a way to progress in the game (see Paiva et al., 2002). By expressing an emotion, such as for example ‘‘anger’’, the user influences the emotional state of his controlled character, which in turn will act accordingly in the game (for example by attacking more and more aggressively his opponent). This will affect progress in the game. At each moment, the user must find the appropriate emotion to control so that he can win the game. The aim of SenToy is to ‘‘pull the user into the game’’ through the use of a physical, touchable affective interface. The first step in the development of SenToy was to find out if users were able to express certain emotions through a toy-like interface. Our main questions were: can a user control the emotional state of a synthetic character using a physical interface, such as a doll? What kind of gestures would be the best ones to express the set of emotional states we need for the game FantasyA? What type of doll is the most appropriate for this kind of interface? To answer these research questions we conducted a small Wizard of Oz study, Dahlb.ack et al. (1993). The results of this study, Andersson et al. (2002), showed that users were able to express certain emotions through the movements of the doll. Based on the results of this early study we developed SenToy, which is a wireless doll that captures six emotions (happiness, sadness, surprise, fear, gloat, anger) from the user’s gestures. SenToy was subsequently integrated with a computer game (FantasyA) and was evaluated with 30 subjects. There was no direct coupling between the actions performed by the user with SenToy and the behaviour of the synthetic character. Rather the synthetic character performed emotional animations in reaction to the emotion expressed by the SenToy. The results show that players really liked SenToy as an interface to FantasyA and were able to influence their characters in the game with SenToy. 2. Bootstrapping the design of SenToy: wizard-of-OZ study The design of SenToy started with a study to determine how users could express emotions through the handling of a physical doll. Users were placed in front of a ‘‘controllable’’ synthetic character (Papous) and were asked to control the character’s emotions through a plush toy. Fig. 1 shows the three toys that were used. They were told that the plush toy had sensors in its arms and that they controlled Papous, while in reality it was a human ‘‘Wizard behind the scenes’’ who controlled Papous. When the Wizard recognized a movement pattern according to a researched scheme (see Andersson et al., 2002 for details), she made Papous express that emotion. We found that different sets of movements were used more often to express the emotions through the fake SenToy, see Table 1. A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 229 Fig. 1. The dolls tested. Table 1 Results from the initial WoZ study Emotion Most common action Second most common action Anger Fear Disgust Boxing with its arms Hands in front of the eyes Arm in front of face as if wiping something away Dancing/jumping, continuous movement Bending down its trunk Arms in the air, frozen position Shake the doll Turn the doll away from the camera ‘‘Vomiting’’ Happiness Sadness Surprise Arms in the air, waving them back and forth Hands in front of the eyes Lifting the doll upwards into a frozen position We also got interesting feedback on how the doll should be designed to best fit with the purpose. Three dolls were tested: a teddy bear, a puppet ‘‘Pippi Longstocking’’ and a SuperModel Barbie Doll Kenneth (see Fig. 1). Users preferred the soft and cuddly teddy bear rather than the hard plastic Kenneth. They also preferred a doll with neutral facial expressions. The doll’s size should be about the size of the teddy bear. Based on these results, we developed the first prototype of the SenToy, and integrated it into the computer game FantasyA. 3. SenToy implementation The SenToy is an explicit sensorial interface equipped with three sets of sensors. The first and most important is the set of accelerometers, which measure the acceleration that the SenToy is subjected to. The movements made by the user with SenToy, like for example bouncing the doll up and down as a dance-happiness, cause 230 A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 variations of the acceleration value given by the accelerometer on the X-axis. These variations are picked up and processed to determine the emotion associated with the gesture. The second type of sensor is analog and these are used to determine the position of SenToy’s limbs. The third set of sensors are digital, and are used to indicate whether the hands of the doll are placed over the eyes or not. Since the emotions/actions cannot be obtained directly from the rather complex data received from the SenToy sensors, a signal-processing module (Stimuli Acquisition module) was required. This module was build to capture the patterns of each of the six chosen emotions. A significant part of the processing applied in this module is to process and interpret data produced in the accelerometers. These signals are sent to the PC via radio and consist on the amount of acceleration that the SenToy is subjected to in the three spatial axes given by a set of orthogonal accelerometers and the position of the limbs. The emotions and actions are inferred by the characteristics of these signals, mainly with the information given by the accelerometers, through which one can determine the SenToy’s attitude (angle) and motion characteristics such as, the direction of the movement and its intensity. As an example, the emotion Sad is detected when the SenToy is bent forward (determined by the Sentoy’s attitude), and the emotion Angry is identified when the doll is shaken (originating a fast and intense variation in the X-axis). The position of the limbs complements the information of the accelerometers. For example, the emotion Angry is only detected when the SenToy is shaken with the arms up. Concerning the look and feel of SenToy, we decided to make a soft and cuddly toy rather than a hard plastic one, following the user’s preference in the first Wizard of Oz study. The SenToy’s exterior was created using a soft fabric, filled with latex and covered with a skin made with lycra-like material (see Fig. 2). A number of factors were considered when choosing the material for the doll and the way it should be constructed. The doll should be produced in such as way so that it can be easily opened, providing access inside for tasks such as changing batteries and routine maintenance. Important areas on the doll should be able to be marked with a softer material, guiding users in the way they should feel the doll. It should have no defined facial expression, character or personality, a result from the earlier Wizard of Oz study. This neutral appearance should not constrain its manipulation to express emotion, the user’s ability to identify with the doll and the adequacy of the doll to influence different synthetic characters. Its minimalist appearance should allow for the same toy to be used to control many different types of characters. Given that FantasyA is like an adventure game with emotions, the image of the toy lies between a toy and an electronic gadget. Its colour and shape reflect this idea. 4. Evaluating the realized SenToy In the most recent study we have been able to use the working prototype SenToy doll to test whether people understand and naturally use the actions and gestures identified in the previous WoZ study, and also to gauge the success of the realized A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 231 Fig. 2. SenToy. SenToy doll. We performed a study with 30 subjects that was divided into two parts: first we tested the emotion gestures chosen in the WoZ study (4 additional subjects who helped in initial game testing and configuration only undertook this first part of the study), and second, SenToy was used to play a duel in the FantasyA game. Note that disgust was one of the emotions in the first WoZ study, but this was subsequently replaced with gloating. Our subjects ranged in age from 9=o 20.6=o 38 years (9=o 21.8=o 45 for the first part) and can be divided into three main categories: Young Children, High School Students and Adults. 5. Gesturing with SenToy Subjects were asked to express emotions, one by one, without receiving any prior instructions. For each emotion they were given about 30 s to perform a gesture for it. The system gave feedback through a textual display that showed which emotion was currently recognized from their actions with SenToy. We subsequently reviewed the video footage of each participant, noting when they successfully performed the gesture for the requested emotion (see Fig. 3). In many cases the participants were close, sometime extremely, to performing the gesture correctly, but the movements 232 A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 35 30 25 20 15 10 5 0 Happy Sad Happy Sad Gloat Anger Fear Surprise 35 30 25 20 15 10 5 0 Gloat Anger Fear Surprise no idea other alternate correct but not recognised correct Fig. 3. How subjects fared in expression emotions. were not exactly what the sensors were expecting. Alternate gestures, which had been identified in the original WoZ study but were not yet implemented in the prototype SenToy, were also noted. 6. Results As we can see happiness, sadness and anger worked really well. Gloating was hard since most pointed with the right arm, as intended, but did not simultaneously dance A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 233 with the doll (as in the happiness expression). Surprise was most difficult from a sensor point of view. Subjects did pull the doll backwards, but not fast or far enough for the sensor to pick up the acceleration. The questionnaire replies collected in this second evaluation confirm the results. Happiness, sadness and anger are perceived as easiest to express. Even though only half of the subjects managed to express sadness, the other half either bent the doll’s head forward or did not apply a large enough lean angle. We did not see a range of alternate gestures as with some of the other emotions (e.g. fear) and so feel comfortable listing it with the obviously more successful happiness and anger. Given the implementation of the alternate gestures used (arm waving for happy and head movement for sad) it is fair to say that these three emotions can be naturally and easily expressed using the doll. Further work should concentrate on fine-tuning the amplitudes and angles of the movements required, and not on identifying new or alternate gestures. The second group contains gloating and fear. Our subjects were less successful at expressing these emotions when asked, and while alternate gestures did emerge there was less of a consensus amongst these. It is clear that pointing should be involved in the gloating gesture, but further work is required to refine exactly what this gesture should be. This is not totally unexpected, as gloating was not part of the previous Wizard of Oz study, and this is the first time it is being evaluated as a gesture. Fear produced a large range of possible gestures, and the task now is to improve the sensor alignment and to decide upon an alternate gesture and support that well. 7. Playing FantasyA For the second part of the study users were given instructions on how to perform the gestures and then subsequently started to play the FantasyA game. The only emotion that was entirely avoided was surprise. This can be due to the fact that this emotion did not have a natural place in the game, but we suspect that it was also because it was quite hard to perform. Gloating was used frequently with great success. 8. Results During the game most emotional expressions were very physical and encouraged players to act out the emotion. Some users, especially the children, were really keen on having the doll and would pull it from the other player or interfere and try to help the other player in expressing some particular emotion. In the interviews, two children commented that they would have liked to have a doll each and be able to play against each other. Players often got very involved with the doll and the game. Often this was expressed through bigger movements with SenToy and through voice. Sometimes these movements would become so big so that the player would have to lean to one side to be able to see what was happening on the screen since the SenToy would be blocking their view. Smaller movements could be associated with less 234 A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 12 10 8 6 4 2 0 No, it was hopeless Yes, very much Kids High School Adults Fig. 4. Answers to the question: Did you like the doll? involvement with the game, though sometimes this came when subjects felt more and more assured that they knew how to express a particular emotion. Players sometimes expressed the emotion not only through moving the doll, but also, to some extent, through moving their own bodies. We regard this as a positive sign that we are on the right track in creating a tangible and sympathetic interface for affective interactions and we believe that this could probably be explored even more in subsequent designs. After the game about 80% seemed to like the doll, see Fig. 4. The kids were in general more enthusiastic about the doll than the adults. 9. Appearance of SenToy Many players cuddled the doll throughout the interview (after the game). They liked the size, compared it to a child, and seemed to enjoy its weight and design. In the comments field of the questionnaire, one player wrote: ‘‘A few days after having played, I still like the doll very much. I really appreciated his direct contact to give commands, even if in that case, the commands were not that obvious and their result a bit fuzzy.’’ (adult player) A. Paiva et al. / Int. J. Human-Computer Studies 59 (2003) 227–235 235 One of the kids remarked that he would probably like to use the SenToy for a whole month before getting bored. Considering that he was 12 years old, this is a very good result! 10. Robustness of SenToy SenToy, despite its prototype status, was able to survive 2 weeks of use with some really rough handling from time to time. Its robustness can probably be improved, but in general the clothing seems to work fine, the skeleton inside the doll did not break, and most sensors survived the shaking and bending. Towards the end of the trial phase we had some problems with sensor calibration, and the accelerometers sometimes stopped working which in turn made the emotion surprise easier to perform. The arm sensor for gloating sometimes failed near end the end of the trial, but in general people managed. 11. Conclusions In summary, we can say that SenToy was a great success. Users were able to express the emotions that allowed them to get engaged in the game. It was robust enough for the experiment, it encouraged a more tactile interaction, players did identify with it and the size, shape and behaviours were close enough to what they should be. A few improvements can be made (like adjusting the signal-processing module to capture some of the gestures not picked up and that are easily identifiable). SenToy works well as an input device for generating a range of gestures that effectively express three of the basic emotions we examined. Acknowledgements Thanks to all our partners in the Safira project for their comments and criticisms on the SenToy interface. References . K., Mour*ao, D., Paiva, A., Costa, M., 2002. Using a wizard of Oz study to inform Andersson, G., Ho. ok, the design of SenToy. Designing Interactive Systems (DIS). ACM Press, New York. . Dahlb.ack, N., Jonsson, A., Ahrenberg, L., 1993. Wizard of oz studies—why and how. In: Proceedings ACM International Workshop on Intelligent User Interfaces. ACM Press, New York. . K., Mour*ao, D., Martinho, C., Costa, M., 2002. SenToy in FantasyA: Paiva, A., Andersson, G., Ho. ok, designing an affective sympathetic interface to a computer game. Personal and Ubiquitous Computing Journal (Ub Comp) 6 (5–6), 2002.