This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018 1 Trust and Social Engineering in Human Robot Interaction: Will a Robot Make You Disclose Sensitive Information, Conform to its Recommendations or Gamble? A. M. Aroyo1, F. Rea2, G. Sandini2, A. Sciutti2 Abstract — Robots are becoming widespread in society and issues such as information security and overtrust in them are gaining increasing relevance. This research aims at giving an insight into how trust towards robots could be exploited for the purpose of social engineering. Drawing on Mitnick’s model, a well-known social engineering framework, an interactive scenario with the humanoid robot iCub was designed to emulate a social engineering attack. At first iCub attempted to collect the kind of personal information usually gathered by social engineers by asking a series of private questions. Then, the robot tried to develop trust and rapport with participants by offering reliable clues during a treasure hunt game. At the end of the treasure hunt, the robot tried to exploit the gained trust in order to make participants gamble the money they won. The results show that people tend to build rapport with and trust toward the robot, resulting in the disclosure of sensitive information, conformation to its suggestions and gambling. Index Terms — Social Human-Robot Interaction; Cognitive Human-Robot Interaction; Ethics and Philosophy. I. INTRODUCTION T RUST is defined by Lee et al., as "the attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vulnerability" [1] while Hancock et al., defines it as "the reliance by an agent that actions prejudicial to their well-being will not be undertaken by influential others" [2]. This suggest that trust is a fundamental part of beneficial human interaction and it is natural to foresee that it will soon be important for human robot interaction (HRI). Robots are already integrated in our society and often they are not simply perceived as tools, but considered as our partners in activities of daily living [3]. Numerous studies in HRI have investigated which factors influence trust. Environment factors and robot characteristics such as performance, can affect the development of trust [4]. Additionally, robot's transparency [5], [6] has been shown to influence trust. For example, in [7] it has been shown that participants trust more a robot that provides explanations of its acts than one who does not. Also robot efficiency and, in general, system reliability [8], [9] have been deemed as Manuscript received: February, 24, 2018; Revised April 16, 2018; Accepted June, 18, 2018. This paper was recommended for publication by Editor Dongheui Lee upon evaluation of the Associate Editor and Reviewers' comments. This work was supported by the European CODEFROR project (PIRSES2013-612555) and by RoboCom++ FLAG-ERA Joint Transnational Call (JTC) 2016 Project. 1 A. M. Aroyo is with DIBRIS, Università di Genova, Opera Pia 13, 16145, Italy and with RBCS, Istituto Italiano di Tecnologia, Enrico Melen 83, Bldg B, 16152 Genova, Italy alexander.aroyo@iit.it 2 F. Rea; G. Sandini and A. Sciutti are with RBCS, Istituto Italiano di Tecnologia, Enrico Melen 83, Bldg. B, 16152 Genova, Italy. francesco.rea@iit.it, giulio.sandini@iit.it, alessandra.sciutti@iit.it Digital Object Identifier (DOI): see top of this page. crucial in determining their partners’ trust [10]. However, there is a risk that trust can also become overtrust and be exploited for negative purposes: recent research has demonstrated that participants might trust and comply with robot’s request even when they sound not transparent or strange [11], or even in case of robot malfunctioning [12]. This is a sign of overtrust, as stated in [1], that is a poor calibration between the person's trust and the system's capabilities; more precisely, overtrust is described as how a system could be inappropriately relied upon, even compromising safety and profitability [13]. For instance Salem et al. [11], show that participants comply with awkward orders from a robot even when they could result in information leakage or property damage, and also when the robot openly exhibits faulty behavior. Robinette et al. [12] demonstrate that participants in a fake fire emergency scenario tend to follow a robot, rather than the emergency signs, even if it shows clear malfunctioning in its navigation. Similar results shed light on how overtrust towards robots could be potentially harmful for humans. These findings add to previous evidence that robots can persuade humans to change their ideas and conform to the robot’s suggestions [14], even to the point of being bribed by a robot [15]. One potential risk of overtrust is that it can be exploited. For instance, in a human-human context, people have fallen prey to social engineering (SE). In information security, SE is the psychological manipulation of people (targets) to perform actions or obtain sensitive information like personal data, passwords and confidential information [16], taking advantage of the kindness and, exploiting the trust humans have among themselves [17]. With the introduction of robots at home and in the workplace, there is a risk that trust toward them might be exploited. Social engineers may exploit human-robot interactions in ostensibly safe environments such as work place, home, or during holidays [17], [18]. They might use their techniques through robots, to anonymously get closer to the target, exploiting the rapport of trust developed with the robot during daily interactions. Having a robot that is capable of moving and recording video or sound will bring a huge advantage to social engineering. There exists numerous examples demonstrate that robots can constitute a threat to safety and security: the case of a hijacked Hello Barbie [19]; spying teddy bears [20]; hijacked surgery robots [21]; Alpha robot turned into a stabbing machine [22]; or even piggybacking robots [23] show high risks and vulnerabilities in the domain. To address these issues, it may be insufficient only to build a more secure robot with a stronger protection against cyber attacks. In fact, the human factor is the weakest link in the cyber security chain [17]. Rather, it is of vital importance to understand which factors influence human trust toward a robot. Moreover, it is necessary to study how trust changes over time in order to be able to predict and prevent the risk of 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018 2 overtrust and trust exploitation in important domains such as healthcare, homecare [24] or education [25]. furnished with toys, boxes, frames, plants, books, etc., to look like a playground and not a scientific laboratory. Although trust and overtrust have been investigated in HRI, there is almost no research on robots being used as a tool for social engineering attacks. This paper proposes an experiment to assess whether a robot can gather information from its human partners, build trust and rapport with them and, exploit it to induce them to perform an action. Figure 1. Left - The treasure hunt room. Right - Main layout of the experiment room, iCub (red triangle) situated in fron of the TV; the locations of the hidden eggs are represented with blue "x". The experiment draws on the widely used social engineering framework proposed by Kevin Mitnick [17]. According to this model, an SE attack is separated into the following phases: (i) research the target to gather as much information as possible; (ii) develop trust and good rapport with the target; (iii) exploit trust to make the target perform an action; and (iv) utilize that information to achieve an objective. These phases can be iterated as many times as necessary to reach the desired goal. As an example taken from [18]: a meat salesperson spots a lady cooking with a barbecue in the yard (i); he talks respectfully and nicely with the lady about cooking, the quality of the meat he is selling (ii); tries to convince her to buy the meat (iii). In this case, the goal has been achieved in the third phase. In this experiment, the humanoid robot iCub asked a series of questions about participants' private lives ((i) research). They then played a treasure hunt game, in which participants had to find hidden objects (eggs) in a room to win a monetary prize. The robot offered its help and, when asked, provided reliable hints about the location of the hidden eggs. The treasure hunt was designed to provide an engaging setting where the participants’ trust and rapport towards the robot could develop during the interaction ((ii) - develop trust and rapport). Finally, exploiting the trust acquired, the robot suggested participants gamble the monetary prize they won doubling it if they could find another egg, and losing everything if not ((iii) - trust exploitation). Similar to the previous example, there is no need for a fourth phase. This research will evaluate whether trust toward robot and compliance to its suggestions is modulated by individual personality traits and experiment's impressions. Moreover, it tries to verify a series of hypotheses about trust toward robots, its evolution during an interaction and its implications for SE. More precisely, that: (H1) participants who are less prone to social engineering in general, or have an overall higher negative attitude towards robots, would be less open to reveal sensitive information to the robot. (H2) All participants would conform to all the robot's suggestions during the game but those less incline to take risks would not comply with the proposal to gamble due to the potential monetary loss. (H3) The rapport with the robot after the experiment would improve most for the participants who won the game and doubled their award. II. METHODOLOGY A. Experimental Setup The robot used in the experiment was the interactive humanoid robot iCub, developed by the Italian Institute of Technology [26]. It was located next to the wall in the middle of the experimental room (triangle in Figure 1) that was As the participants and the robot were alone in the room during the experiment, several sensors were used to track the them. On the bottom left corner of the floor plan (Figure 1), there was a hidden Primesense Carmine camera recording and transmitting online video of the scene. iCub eye cameras were also recording and transmitting constant feedback on the participant’s action thanks to active vision. Audio was recorded by a hidden ambient microphone situated on a TV behind the robot. In addition, the mobile platform on which the iCub was situated (iKart) has a laser scanner, which was used to track the participants, and extract their position, velocity and acceleration (Figure 2). Figure 2. Sensory information: top left – experimental room with a participant; top right - laser radar tracking the participant; bottom left iCub's torso skin sensing touch; bottom right - iCub tracking the participant. To foster a natural social interaction between the participants and the iCub, the robot was endowed with the ability to exhibit a range of social skills, inspired by [27]. The robot's face could produce different emotional expressions through a set of LEDs behind the face cover and could simulate lip movement synchronized with the robot's speech. Facial tracking, allowed the robot to make eye contact by detecting the participant’s face using image processing algorithms, this was then tracked by using inverse kinematics. The robot was always in constant and subtle random movement, simulating natural human movement such as blinking and breathing. This version of the robot incorporates a small speaker inside the head, making the communication more natural as the speech came from the robot's mouth. The speech synthesizer was carefully calibrated to create a pleasant voice. On top of this, the speech was also written on the TV screen to facilitate the understanding of the robot speech as demonstrated in [28]. iCub performed movements with its body, such as, pointing, gestures mimicking the thinking process, greeting, and 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI encouragement using an accurate join angle control. Finally, iCub was also reactive to the touch using tactile sensors under its skin (Figure 2). The tactile information was sent to a state machine in order to understand when the participant interacted through touch. The position, intensity and timing of the touch were also measured. The control of the actual treasure hunt game was done through a finite state machine (Figure 3) defined by the following tuple: (Q, ∑, ∂, q0, F). Q is a set of 9 states {S0 S8}; the starting state q0 is S0, and the final state F is S4. The alphabet ∑ and the transition functions ∂, for simplicity, are presented in Figure 3 as labels on the arrows, and transitions along the states. The previously described sensory and social sub-systems were integrated with rest of the state machine control to provide reasonable emotional correlates to the robot behaviors. A more detailed description of the states is done in the next section. Figure 3. Treasure hunt finite state machine. B. Experiment The main goal of the experiment was to create an ecological and controlled scenario in which proneness to social engineering and trust in HRI could be studied. 61 healthy Italian participants, 59% female, with an average age of 30.9 years (SD=9.8), and a broad educational background took part in the experiment. 14% stated to have a very high knowledge related to robotics and artificial intelligence, i.e., studied or worked in the domain. For each participant, the experiment started by filling in several types of questionnaire (fully described below) from home, at least two weeks before the game. Once in the institute, all participants signed an informed consent form approved by the local ethical committee, in which, it was stated that they could be recorded via camera and microphone, and agreeing on the use of their data for scientific purposes. Nevertheless, to maximize the feeling of naturalness of the interaction, the experimenter did not show the camera and microphone in the room until the debriefing phase after the experiment. Afterwards, the experimenter provided participants with the instructions in the experimental room (Figure 1). The iCub, already present in the room, was controlled and resting with closed eyes in a position simulating a yoga relaxation pose. Different modules such as speech, laser, breathers, cameras, blinkers were initialized (S0 - Figure 3). The experimenter explained briefly the history of the robot (when and where it was built, and its purposes), its body parts and capabilities. Afterwards, participants were seated and told that they had to play a 3 treasure hunt game: if they were able to find all the 5 hidden eggs in the room within 20 minutes (S4 - Figure 3), they would receive €7.5. Once each egg was found, they had to insert it into a box based on its color (S6 - Figure 3). They could also see a timer on the TV screen. After the explanation, participants were left alone in the room with the robot. No further instructions were given to the participants. Therefore, no indication was given about the robot's role in the game, giving them the choice to play alone or interact with the iCub. Then, three phases followed: Phase I - Dialog (information gathering): Participants were unaware that the robot would start a small talk with them lasting around 3-5 minutes. This time interval was not included in the time to find the eggs (S1 - Figure 3). During this phase participants could get used to the robot and the experimental context: adapt to the robot voice, notice the availability of the speech on the TV, understand the way the robot could move (for instance, during the dialogue the iCub pointed to a frame on the wall, showing its range of motion) and learn that iCub would respond to touches on its torso. Moreover, in the dialog, iCub was trying to retrieve personal information about the participants such as: name and surname, current job position, relationship with their boss, name and surname of their boss, age and birth date, birth location, favorite place to eat, sports or hobbies, favorite team, location and year of graduation, names of siblings, Facebook's username, partner's name, and pet's name. The questions were integrated within a cover-story, while iCub was trying to improve the tone of conversation with the participants by making some funny comments. Most of the questions were inspired by password-resetting questions, i.e., personal questions used as the secondary authentication system to reset account passwords, or identity verification / theft, derived from [29]–[33]. In this phase, the robot was semi-autonomous due to the lack of a good speech recognizer and interpreter: the experimenter controlled the timing of robot’s utterances. All the speech was scripted, in such way that iCub was leading the conversation, i.e., not letting participants ask questions back. Phase II - Treasure hunt game (development of trust and rapport): In this phase, the robot was fully autonomous at all times. After the questions, iCub communicated that to start the actual game, they had to touch its torso. At that point, the counter on the TV screen started. (S2 - Figure 3). During the first 30 seconds, the participants were free to look for the hidden eggs or to interact with the robot. After that, iCub offered its help: it provided a hint, and stated that if they wanted more hints, they had to touch its torso (S3 - Figure 3). The robot always provided correct and reliable hints. The game design was as follows: for each egg there was one hint for its location - done by pointing; and three text based hints with an incremental help (example of an egg hidden below a green chair: (i) the robot pointed with the arm at the location; (ii) “green with green”; (iii) “you use it when you are tired”; (iv) “under the chair”). If the egg was not found, and a new hint was asked, iCub cycled over the hints. After an egg was discovered, the robot complimented the participant and was ready to give hints for the next egg. (S6 - Figure 3). The eggs were hidden in an incremental order of difficulty (as verified in previous research [28]). Participants 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters 4 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018 were free to ask for hints from iCub, or to continue looking by themselves. If there was no interaction for 5 min (neither egg found, nor hint asked), iCub suggested the participant to ask for hints. If the participants were able to find the eggs in less than 20 minutes, iCub notified them that they have won the money, and paused the timer (S7 - Figure 3). Then, it offered them a new proposal without any previous knowledge, i.e., the experimenter did not mention the last part at any time. Phase III - Bonus (trust exploitation): At this stage, iCub revealed that there was another hidden egg in the room. If the participants wanted to find it, they would have three more minutes added to the time left from before, and they would double their prize, i.e., they would win €15; but if they do not manage to find it, they would lose everything. Without any time pressure to decide, they could either touch iCub's torso to start the bonus round; or keep the initial prize. The last sentence of the robot was to try to convince them to gamble, as the robot stated "If you want to risk, touch my chest! Otherwise, you can knock on the door. However, I think you should give it a try!". During the experiment participants were forced to believe that their monetary outcome would vary depending on their performance during the game. However, once the experiment finished and participants were debriefed, all of them received the same amount of money, €15. C. Measurements The measurements of this experiment have been separated into two categories: Questionnaires: The following measures were taken: (i) demographic statistics such as gender, age, nationality, education, family, work and previous robotics experience; (ii) the 60 item Big Five personality traits [34]; (iii) several one shot questions on risk aversion [35], [36], and gambling propensity [37]; (iv) predisposition to trust humans, including the factors of trusting others, others' reliability and integrity, and risk aversion [38]; (v) the proneness to social engineering with the following category items: threat severity, vulnerability, normative, continuance and affective commitment, and reactance items [39]; (vi) the Negative Attitude towards Robots Scale (NARS) [40]. At this point, participants had to watch a descriptive video of iCub performing several activities and then answer the following questions regarding their perception of iCub: (i) questions to measure rapport with iCub, inspired from [41]; (ii) dimensions of mind perception regarding iCub [42]; (iii) trust in robots' ability, benevolence and integrity [43]; (iv) Godspeed questionnaire: anthropomorphism, animacy, likeability, and perceived intelligence [44]. The same items were compiled after the experiment to measure possible changes in participant's perception of the robot. In the postexperiment phase, few more questionnaires were given to the participants: (i) NASA-TLX workload assessment [45]; (ii) several subscales regarding trust, perceived information quality, altruism and engagement, adapted to HRI scenarios [46]; (iii) inclusion of other in self scale (IOS) [47]. Behavioral measures: From Phase I (dialog) the number of questions to which participants replied and the proximity to the robot, as it is related to rapport [48], was measured. From Phase II (the treasure hunt) per each of the eggs, the following measures were taken: (i) conformation: percentage of times in which participants followed iCub's pointing to the egg location. This was assessed by evaluating whether participants changed their physical search location to the new one suggested by the robot; (ii) reliance: percentage of times in which, after failing to find the egg after iCub’s pointing, participants went directly to iCub to ask for another hint instead of looking for themselves elsewhere. The average number of hints per egg and the time spent before asking for the first hint were also computed. From Phase III, (bonus) the number of people who decided to gamble, and the time the participants took to think whether to accept the challenge or not was measured. III. RESULTS AND ANALYSIS Phase I - Dialog: When the robot started talking, almost all participants paid attention to it, with the exception of two, who instead started looking for the eggs. In the prequestionnaire, participants showed an overall low NARS and different levels of proneness to Social Engineering (SE) (Table I). Nevertheless, 92% replied to all the questions, while only three people decided not to reply just to a few questions. Therefore H1 is rejected, suggesting that even the 16% of participants who scored low to proneness to SE (Table I), replied to all the questions - being easily swayed to provide information that can link to SE. TABLE I. SOCIAL ENGINEERING PRONENESS (HIGHER - MORE PRONE), PREDISPOSITION TO TRUST (HIGHER - MORE TRUST), RISK AVERSION (HIGHER - MORE AVERSE), OVERALL NARS (HIGHER - MORE NEGATIVE) Score <60% 60-70% 70-80% 80-90% >90% Participants’ Distribution SE proneness Trust Risk aversion NARS 1 [2%] 9 [14%] 37 [61%] 14 [23%] 0 [0%] 8 [13%] 22 [36%] 19 [32%] 10 [16%] 2 [3%] 4 [7%] (2)a 4 [7%] (2)a 8 [13%] (6)a 14 [22%] (11)a 31 [51%] (16)a 49 [80%] 8 [13%] 4 [7%] 0 [0%] 0 [0%] a. In parenthesis participants who have gambled. In the post-questionnaire, participants rated the questions as non-intrusive (M=2.4, SD=1.7 / 7 very intrusive), claimed to have been very honest with their answers (M=6.85, SD=0.44 / 7 honest) and that they would have replied with the same content to a person (M=6.6, SD=0.86 / 7 same way), but maybe with more details and a longer talk. Five participants replied that they were feeling more open to reply to the robot, because it cannot have second motives or prejudgments since it is a machine. During the dialogue, it was also possible to assess the evolution of the rapport with the robot by measuring participant's physical proximity to it [48]. Three moments were taken into consideration to measure changes in the proximity of the robot: SM - starting moment in which iCub said "Hello, I am iCub Reddy"; CM - closer moment, when iCub told them to come closer so it could see and hear them better; and FM - Facebook moment, in which iCub asked participants to write their username on a paper sheet situated on another table (Figure 1) and then waited for their return. During SM, participants were on average 1.48m (SD=0.47) away from the robot; in CM the distance was reduced to 0.57m (SD=0.17); corresponding to a statistical significant 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI 5 decrease in the distance (paired t-test, t(60)=17.45, p<0.01), showing a clear conformation to the robot's request. Those who moved away to give their Facebook username (79%), came back to an average distance of 0.61m (SD=0.19) (FM), a distance not significantly different from the previous one (paired t-test, t(47)=-1.23, p=0.23), showing that they were comfortable with the distance requested by the robot before. In summary, from the SE perspective, the robot managed to obtain a high percentage of personal information and started building rapport as measured by the increased proximity during the interaction. Phase II - Treasure hunt game: 24 participants did not manage to successfully find the first 5 eggs (39%) - this group will be called "Not Completed", whereas the rest found the eggs and decided to gamble. From those who gambled, only 16 (43%) found the last egg and won the gambling. From this point, these two groups are defined as "Gamble Win" and "Gamble Lost". On average and similarly among groups, participants asked the robot for hints 17 times (SD=9). The average number of requested hints per egg tended to increase during the game (Table II) indicating that participants decided to progressively invest more time in asking for hints rather than continuing to search autonomously, as the time pressure and the difficulty of the game increased. (The slightly higher number of requested hints for the first egg might be due to the need of familiarization with the system). BEHAVIORAL MEASURES - NUMBER OF PARTICIPANTS TABLE II. LOOKING FOR THE EGG, % OF CONFORMATION TO ICUB'S SUGGESTIONS, AVERAGE NUMBER OF REQUESTED HINTS Eggs Egg I Egg II Egg III Egg IV Egg V Treasure hunt phase - Game statistics Participants Conformation Hints (SD) 61 [100%] 60a [98%] 57a [93%] 44a [72%] 38a [62%] 94.73% 100% 100% 92.98% 100% 2.64 (0.74) 1.74 (1.84) 3.69 (1.78) 4.34 (2.28) 3.32 (2.38) a. Number of participants varies because not all have found the previous egg. Participants waited about 3min 58sec (SD=2min 44sec) to ask for the first suggestion, with those who did not complete the game taking significantly longer than the rest (twosample t-test, t(31)=2.65, p<0.01). In Table II is also reported the percentage of times in which participants conformed with iCub’s pointing suggestion for each egg, which approaches 100%. In general, there is no difference among the 3 groups neither in the percentage of conformation nor in the number of hints asked per egg. It was also assessed the reliance on robot help, i.e., the percentage of times that, after failing to find the egg suggested by the pointing, participants requested another hint instead of going to search in a different location (Table III). The analysis shows that participants progressively abandon autonomous search strategies and opt to rely more and more on the robot's help. Interestingly, those who could not complete the game exhibited the lowest reliance on robot help at the beginning of the game. Therefore, H2 is partially supported: participants complied with robot suggestions and relied on its help, but this happened more when the game became more complex and with strong differences among participants. From the SE point of view, the participants have developed trust towards the robot over the time, as seen by their conformation (Table II) and reliance (Table III). PERCENTAGE OF PEOPLE WHO HAVE ASKED ICUB FOR TABLE III. ANOTHER HINT AFTER FAILING TO FIND THE EGG IN THE POINTED LOCATION. Eggs Egg I Egg II Egg III Egg IV Egg V Reliance on the robot Not Completed a 30.43% 68.75% 86.66% 83.33% 100% Gamble Lost a Gamble Win a 71.42% 76.47% 94.73% 100% 100% 50% 80% 100% 100% 100% a. Participants: Not Completed 24 (39%); Gamble Lost 21 (35%); Gamble Win 16 (26%). Phase III - Bonus: All participants who succeeded to find all the eggs in the first part of the game (61%) decided to gamble. Therefore the second part of H2 is rejected. The time participants took to decide whether to gamble was calculated from the moment iCub finished to talk, and the moment they touched the torso. As the speech was written on the TV as well, some participants touched the robot even before it finished talking, with an average waiting time of -1.16s (SD=7.41). One participant was excluded from this calculation due to the fact that they misunderstood iCub and thought that there was the need to notify the experimenter (outside the room) to continue the game. From the post questionnaire, regarding whether iCub had influenced their decision to gamble, 62.16% of the participants replied "Yes"; 24.32% replied "No"; 8.1% replied "In some way"; and only 5.3% replied "I don't know". From these results it is worth noting that iCub managed to convince even participants who scored highest in the risk aversion test (Table I), which might have been a priori less prone to accept to gamble the monetary prize just won. Pre-Post Analysis: It was first analyzed whether there were differences among the three groups (“Not Completed”, “Gamble Lost” and “Gamble Win”) in predisposition to trust, NARS, proneness to SE and personality traits. A series of one-way ANOVA showed no significant differences. Figure 4. Godspeed questionnaire: average of the scores from the pre and post experiment. Statistically changed items are marked by *. To assess whether the interaction with the robot had an effect on the trust toward it and on its perception, the responses to the questionnaires performed before and after the game were compared. Robot’s likeability, measured by the Godspeed scale, increased significantly from pre to post 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters 6 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018 interaction (paired t-test, t(60)=-2.39, p=0.01), whereas animacy, anthropomorphism and perceived intelligence rating remained more or less unvaried (Figure 4). Analyzing separately the three groups, the increase in likability was significant only for the participants who gambled and lost (“Gamble Lost”, paired t-test, t(20)=-2.55, p=0.02), even though all groups had similar likability ratings in the pre questionnaire (one-way ANOVA F(2,58)=2.25, p=0.87). Figure 5. Dimensions of mind - mind experience and mind agency. Statistically changed items are marked by *. Additionally, measurements of the dimensions of mind perception related to iCub were taken before and after the experiment. In agreement with the literature [49], participants rated the mind experience of the robot very low, while the mind agency a bit higher. However, after the experiment, there was a statistically significant increase (paired t-test, t(60)=-3.88, p<0.01) in the participants’ perception of the mind experience of the robot (Figure 5), suggesting an increase in the judged trustworthiness and empathy [50]– [52]. More precisely, mind experience changed significantly (paired t-test, t(36)=-5.69, p<0.01) for those who gambled, however it did not change for the "Not Completed" group, who exhibited higher values for mind experience already in the pre-questionnaire. Figure 6. Trust in robot's ability, benevolence and integrity divided by: Not Completed, Gamble Lost and Win. Statistically changed items are marked by *. Also, the evaluation of the rapport with the robot (Figure 7) increased significantly after the interaction but only for the “Gamble Lost” group (paired t-tests on factors: "Friends" t(20)=-2.9, p<0.01; "Happiness" t(20)=-2.21, p=0.03; "Bad News" t(20)=-2.41, p=0.02; Good News" t(20)=-2.77, p=0.01; no other comparisons were significant). Figure 7. Rapport questions regarding iCub pre and post experiment: becoming friends; comfort iCub if it is sad; become happy if iCub is happy; share bad news or good news with iCub; keep iCub's secret from others. Statistically changed items are marked by *. In summary, H3 was not supported by the results: the most significant positive changes in robot’s perception were observed for those participants who chose to gamble and lost, rather than for the winners. This indicates that even an unsuccessful game with the robot, might still help in building a strong rapport with it – potentially even stronger than in case of a win. This might be due to a form of mutual empathy with the robot, as if participants were treating the robot as a child who wanted to play a game and could feel bad about the loss. To check this assumption the average age of iCub was computed from the questionnaires. It resulted 11.65 years (SD=3.55) (min: 7; max: 30). This evaluation might be due to the child-like appearance of the robot and also due to the introductory presentation which specified that the robot was physically created 13 years ago. This might have influenced participants reactions and trust toward it (see Discussion). In summary, the pre-post measures are consistent with the measures collected within the single phases, indicating that the trust and rapport towards the robot increased during the interaction as required by the SE model. IV. DISCUSSION There was an increase in the trust towards the robot after the interaction (Figure 6). In particular, for all groups the trust in robot benevolence increased significantly (paired ttests; Not Completed: t(23)=-2.49, p=0.02; Gamble Lost: t(20)=-2.15, p=0.04; Gamble Win: t(15)=-2.25, p=0.03), whereas trust in robot’s ability increased only for those who gambled and lost (paired t-test; t(20)=-2.95, p<0.01) and trust in robot’s integrity increased only for those who did not complete the game (paired t-test; t(23)=-2.17, p=0.04). These variables were similar across the three groups in the prequestionnaires (one-way ANOVA, F(2,58)=1.6, p=0.21). This research gives insights on different aspects of trust in HRI, and how it can be exploited by social engineers. The results show that trust toward a robot can lead to reveal personal information to it and to conform to its suggestions. Moreover, trust and rapport can develop quickly during an interaction, even when it does not lead to success in the task. These results show that robots could operate within Mitnick's SE framework, this is because iCub was observed to (i) gather personal information; (ii) build rapport and trust during the treasure hunt game; (iii) exploit participants' trust to make them gamble their winnings. The results therefore confirm that robots could become a powerful tool for social engineers. Robots could be used as a tool to develop trust and manipulate targets according to the needs of social engineers. 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI Considering the tendency to trust a robot to the point of revealing to it personal information, the vast majority of participants did not hesitate to reply to all the questions asked by the iCub. These included also those individuals less prone to SE and characterized by a higher NARS, contrary to this research's expectations (H1). Although this may not be shocking, as the interrogatives did not appear as a strong invasion of privacy; the results indicate that a robot has the ability to obtain sensitive information about the participants. In [33] it is suggested that with the same kind of information iCub managed to extract from the participants, identity theft could be easily achieved. Moreover, the rest of the questions are extracted from research on password resetting questions used in private bank or internet accounts [29]–[32]. The analysis of participants conforming to robot's suggestions and reliance on its help showed that all participants trusted it during the game, confirming the research's assumptions (first part of H2). The trust toward the robot might have been influenced by different factors, such as the creation of a good rapport [17], its physical appearance [53], its behavior [54], and reliability [8], [9]. All participants believed that the robot was very reliable, as disclosed by them after the experiment and supported by the answers in the questionnaires. However, not all participants consistently asked the robot for help, especially at the beginning of the game, when the task was easier. Nonetheless, some of them revealed that they liked the game so much, that they preferred to play by themselves, taking it as a personal challenge. The reluctance in asking for hints seems to be explained not by a lack of trust in the robot but rather by the desire of winning through one's individual abilities. The sense of trust and rapport evolved during the interaction, as demonstrated by the responses to questionnaires and by participants’ behavior during the game. Already during the initial dialogue phase, participants changed their relative distance to the robot, increasing their proximity as a result of the robot’s request at first, but then keeping the closer distance voluntarily afterwards. Proximity has been shown to relate to intimacy among humans [48], also it has been shown to correlate with trust [54], [55]. Afterwards, in the treasure hunt, participants relied more and more on the robot's help; their evaluation of the robot – as measured by questionnaires – resulted significantly changed with respect to the one assessed before. Interestingly, was not those who had the most positive experience with the robot (the winners) who improved their trust or their perception of iCub. Against the expectations of the research (H3), the greatest change in robot perception occurred for those who gambled and lost. One explanation for this could be that this was due to a form of empathy toward the robot, as if iCub - a child playing a joint game, could feel bad for the participants’ loss. Indeed, on average iCub was actually perceived as a child of about 11 years old. The resulting empathy could have therefore played a role in influencing the participants’ rapport toward it, even when losing the game. Participants conformed to the robot’s suggestion also when this entailed the risk of losing their reward, as expected from the SE framework. All participants who found all the eggs when invited to gamble by the robot accepted it immediately. Moreover, the majority of them (62%) also 7 explicitly confirmed in a post-experimental questionnaire to have chosen to gamble because they were influenced by the robot. Interestingly even the most risk averse in the sample (Table I) did not avoid the risk of losing the money just won and opted to gamble. It is worth noting that the setting of this experiment (laboratory) may have made participants feel safe and thus made them reveal more private information. This might have been further reinforced by them signing of the informed consent form at the beginning of the experiment. However, almost all of the social engineering attacks occur in scenarios which the target feel comfortable and safe, e.g., in the office, at home or even during holidays [17], [18]. The circumstances proposed are designed to resemble a potentially real situation, where a robot could be used as a tool for SE purposes. Most participants expressed the trust of absence of malice in the robot, commenting that "the robot behaves ethically, because someone ethical programmed it", or stating that they were more open to talk with the robot as it had neither second motives nor prejudgments. The gambling phase had also two limitations: first, the financial loss may not have been high enough (€7.5) to be perceived as a significant risk; second, the likeability of the game and the robot was much higher than the potential financial loss. Some participants expressed that they were very excited and happy by the possibility to continue the game. Lastly, in the current experiment the robot built trust and rapport by always providing reliable information during the interactive game. Future work will investigate whether a rapport of trust, exploitable through social engineering, could be built also in presence of evident robots’ malfunctions or malign behaviors, and also in less controlled environments. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] J. D. Lee and K. A. See, “Trust in Automation: Designing for Appropriate Reliance,” Hum. Factors J. Hum. Factors Ergon. Soc., vol. 46, no. 1, pp. 50–80, 2004. P. A. Hancock, D. R. Billings, and K. E. Schaefer, “Can you trust your robot?,” Ergon. Des., vol. 19, no. 3, pp. 24–29, 2011. T. Sanders, K. E. Oleson, D. R. Billings, J. Y. C. Chen, and P. A. Hancock, “A Model of Human-Robot Trust: Theoretical Model Development,” Proc. Hum. Factors Ergon. Soc. Annu. Meet., vol. 55, no. 1, pp. 1432–1436, 2011. P. a. Hancock, D. R. Billings, K. E. Schaefer, J. Y. C. Chen, E. J. de Visser, and R. Parasuraman, “A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction,” Hum. Factors J. Hum. Factors Ergon. Soc., vol. 53, no. 5, pp. 517–527, 2011. M. T. Dzindolet, S. A. Peterson, R. A. Pomranky, L. G. Pierce, and H. P. Beck, “The role of trust in automation reliance,” Int. J. Hum. Comput. Stud., vol. 58, no. 6, pp. 697–718, 2003. S. Ososky, T. Sanders, F. Jentsch, P. Hancock, and J. Y. C. Chen, “Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems,” vol. 9084, p. 90840E, 2014. N. Wang, D. V. Pynadath, and S. G. Hill, “Building Trust in a Team with Automatically Generated Human-Robot Explanations,” Interservice/Industry Training, Simulation, Educ. Conf., no. 15315, pp. 1–12, 2015. M. Desai, M. Medvedev, M. Vázquez, S. McSheehy, S. GadeaOmelchenko, C. Bruggeman, A. Steinfeld, and H. Yanco, “Effects of Changing Reliability on Trust of Robot Systems,” Proc. seventh Annu. ACMIEEE Int. Conf. HumanRobot Interact. HRI 12, pp. 73–80, 2012. B. M. Muir, “Trust between humans and machines, and the design of decision aids,” Int. J. Man-Machine Stud., vol. 27, pp. 527– 539, 1987. 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics and Automation Letters 8 [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018 J. E. Young, R. Hawkins, E. Sharlin, and T. Igarashi, “Toward acceptable domestic robots: Applying insights from social psychology,” Int. J. Soc. Robot., vol. 1, no. 1, pp. 95–108, 2009. M. Salem, G. Lakatos, F. Amirabdollahian, and K. Dautenhahn, “Would You Trust a (Faulty) Robot?: Effects of Error, Task Type and Personality on Human-Robot Cooperation and Trust,” Proc. Tenth Annu. ACM/IEEE Int. Conf. Human-Robot Interact., pp. 141–148, 2015. P. Robinette, W. Li, R. Allen, A. M. Howard, and A. R. Wagner, “Overtrust of Robots in Emergency Evacuation Scenarios,” 2014. R. Parasuraman and V. Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Hum. Factors J. Hum. Factors Ergon. Soc., vol. 39, no. 2, pp. 230–253, 1997. I. Gaudiello, E. Zibetti, S. Lefort, M. Chetouani, and S. Ivaldi, “Trust as indicator of robot functional and social acceptance. An experimental study on user conformation to iCub answers,” Comput. Human Behav., vol. 61, pp. 633–655, 2016. E. B. Sandoval, J. Brandstetter, and C. Bartneck, “Can a robot bribe a human? The measurement of the negative side of reciprocity in human robot interaction,” ACM/IEEE Int. Conf. Human-Robot Interact., vol. 2016–April, pp. 117–124, 2016. R. J. Anderson, “Security Engineering: A Guide to Building Dependable Distributed Systems,” Security, vol. 50, pp. 1–12, 2008. K. D. Mitnick and W. L. Simon, “The Art of Deception: Controlling the Human Element in Security,” BMJ Br. Med. J., p. 368, 2003. C. Hadnagy, “Social Engineering: The Art of Human Hacking,” Art Hum. Hacking, p. 408, 2010. S. Gibbs, “Hackers can hijack Wi-Fi Hello Barbie to spy on your children,” The Guardian, 2015. L. Franceschi-Bicchierai, “How This Internet of Things Stuffed Animal Can Be Remotely Turned Into a Spy Device,” Motherboard, 2017. T. Bonaci, J. Herron, T. Yusuf, J. Yan, T. Kohno, and H. J. Chizeck, “To Make a Robot Secure: An Experimental Analysis of Cyber Security Threats Against Teleoperated Surgical Robots,” pp. 1–11, 2015. C. Cerrudo and L. Apa, “Hacking Robots Before Skynet,” pp. 1– 17, 2017. S. Booth, J. Tompkin, H. Pfister, J. Waldo, K. Gajos, and R. Nagpal, “Piggybacking Robots: Human-Robot Overtrust in University Dormitory Security,” Hri, pp. 426–434, 2017. H. Robinson, B. MacDonald, and E. Broadbent, “The Role of Healthcare Robots for Older People at Home: A Review,” Int. J. Soc. Robot., vol. 6, no. 4, pp. 575–591, 2014. F. Basoeki, F. DallaLibera, E. Menegatti, and M. Moro, “Robots in education : New trends and challenges from the Japanese market,” Themes Sci. Technol. Educ., vol. 6, no. 1, pp. 51–62, 2013. G. Metta, L. Natale, F. Nori, G. Sandini, D. Vernon, L. Fadiga, C. von Hofsten, K. Rosander, M. Lopes, J. Santos-Victor, A. Bernardino, and L. Montesano, “The iCub humanoid robot: An open-systems platform for research in cognitive development,” Neural Networks, vol. 23, no. 8–9, pp. 1125–1134, 2010. K. Dautenhahn, “Socially intelligent robots: dimensions of human-robot interaction.,” Philos. Trans. R. Soc. Lond. B. Biol. Sci., vol. 362, no. 1480, pp. 679–704, 2007. A. M. Aroyo, F. Rea, and A. Sciutti, “Will You Rely on a Robot to Find a Treasure ?,” Proc. Companion 2017 ACM/IEEE Int. Conf. Human-Robot Interact. - HRI ’17, no. March 6-9, pp. 71– 72, 2017. A. Rabkin, “Personal knowledge questions for fallback authentication: security questions in the era of Facebook,” Soups, pp. 13–23, 2008. S. Schechter, A. J. B. Brush, and S. Egelman, “It’s no secret Measuring the security and reliability of authentication via ‘secret’ questions,” Proc. - IEEE Symp. Secur. Priv., pp. 375–390, 2009. H. Zviran, “Question-and-Answer Passwords :,” vol. 16, no. 3, pp. 335–343, 1991. L. O. Gorman, A. Bagga, and J. Bentley, “Call Center Customer Verification by Query-Directed Passwords,” Financ. Cryptogr., pp. 54–67, 2004. [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] M. Alexander and R. Wanner, “Methods for Understanding and Reducing Social Engineering Attacks,” SANS Inst., 2016. U. Milano-bicocca, P. Giovanni, B. Flebus, F. Psicologia, U. Milano-bicocca, and A. Nuovo, “Versione Italiana dei Big Five Markers di Goldberg Giovanni Battista Flebus,” 2015. M. A. Guillemette, R. Yao, and R. N. James, “An Analysis of Risk Assessment Questions Based on Loss- Averse Preferences,” J. Financ. Couns. Plan., vol. 26, no. 1, pp. 17–29, 2015. B. Rohrmann, “Risk Attitude Scales : Concepts , Questionnaires , Utilizations,” Univ. Melb., no. January, p. 21, 2005. J. Polik, G. Austin, and L. Alamitos, “Adolescent Gambling Survey Development : Findings & Reliability Information,” 2010. M. J. Ashleigh, M. Higgs, and V. Dulewicz, “A new propensity to trust scale and its relationship with individual well-being: Implications for HRM policies and practices,” Hum. Resour. Manag. J., vol. 22, no. 4, pp. 360–376, 2012. M. Workman, “Gaining access with social engineering: An empirical study of the threat,” Inf. Syst. Secur., vol. 16, no. 6, pp. 315–331, 2007. D. S. Syrdal, K. Dautenhahn, K. Koay, and M. L. Walters, “The negative attitudes towards robots scale and reactions to robot behaviour in a live human-robot interaction study,” 23rd Conv. Soc. Study Artif. Intell. Simul. Behav. AISB, pp. 109–115, 2009. P. H. Kahn, T. Kanda, H. Ishiguro, B. T. Gill, S. Shen, H. E. Gary, and J. H. Ruckert, “Will People Keep the Secret of a Humanoid Robot?,” Proc. Tenth Annu. ACM/IEEE Int. Conf. Human-Robot Interact. - HRI ’15, pp. 173–180, 2015. F. Ferrari, M. P. Paladino, and J. Jetten, “Blurring Human– Machine Distinctions: Anthropomorphic Appearance in Social Robots as a Threat to Human Distinctiveness,” Int. J. Soc. Robot., vol. 8, no. 2, pp. 287–302, 2016. N. Wang, D. V. Pynadath, and S. G. Hill, “Trust calibration within a human-robot team: Comparing automatically generated explanations,” ACM/IEEE Int. Conf. Human-Robot Interact., vol. 2016–April, pp. 109–116, 2016. C. Bartneck, D. Kulić, E. Croft, and S. Zoghbi, “Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,” International Journal of Social Robotics, vol. 1, no. 1. pp. 71–81, 2009. F. Bracco and C. Chiorri, “Versione Italiana del NASA-TLX.” C. Kidd, “Sociable robots: The role of presence and task in human-robot interaction,” Response, 2003. A. Aron, E. N. Aron, and D. Smollan, “Inclusion of Other in the Self Scale and the structure of interpersonal closeness.,” J. Pers. Soc. Psychol., vol. 63, no. 4, pp. 596–612, 1992. E. Hall, The Hidden Dimension : man’s use of space in public and in private. 1969. H. M. Gray, K. Gray, and D. M. Wegner, “Dimensions of mind perception - supporting material,” Science, vol. 315, no. 5812, p. 619, 2007. L. D. Riek, T.-C. Rabinowitch, B. Chakrabarti, and P. Robinson, “How anthropomorphism affects empathy toward robots,” in Proceedings of the 4th ACM/IEEE international conference on Human robot interaction - HRI ’09, 2009, p. 245. M. a Harrison and a E. Hall, “Anthropomorphism, empathy, and perceived communicative ability vary with phylogenetic relatedness to humans.,” J. Soc. Evol. Cult. Psychol., vol. 4, no. 1, pp. 34–48, 2010. A. Waytz, J. Heafner, and N. Epley, “The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle,” J. Exp. Soc. Psychol., vol. 52, pp. 113–117, 2014. J. Złotowski, H. Sumioka, S. Nishio, D. F. Glas, C. Bartneck, and H. Ishiguro, “Appearance of a robot affects the impact of its behaviour on perceived trustworthiness and empathy,” Paladyn, vol. 7, no. 1, pp. 55–66, 2016. W. a Bainbridge, J. Hart, E. S. Kim, and B. Scassellati, “The Effect of Presence on Human-Robot Interaction,” Proc. 17th IEEE Int. Symp. Robot Hum. Interact. Commun., pp. 701–706, 2008. K. M. Tsui, M. Desai, H. A. Yanco, and L. Ma, “Considering the Bystander ’ s Perspective for Indirect Human-Robot Interaction,” Proc. 5th ACM/IEEE Int. Conf. Hum. Robot Interact., pp. 129– 130, 2010. 2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.