This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018
1
Trust and Social Engineering in Human Robot Interaction: Will a
Robot Make You Disclose Sensitive Information, Conform to its
Recommendations or Gamble?
A. M. Aroyo1, F. Rea2, G. Sandini2, A. Sciutti2
Abstract — Robots are becoming widespread in society and
issues such as information security and overtrust in them are
gaining increasing relevance. This research aims at giving an
insight into how trust towards robots could be exploited for the
purpose of social engineering. Drawing on Mitnick’s model, a
well-known social engineering framework, an interactive
scenario with the humanoid robot iCub was designed to
emulate a social engineering attack. At first iCub attempted to
collect the kind of personal information usually gathered by
social engineers by asking a series of private questions. Then,
the robot tried to develop trust and rapport with participants by
offering reliable clues during a treasure hunt game. At the end
of the treasure hunt, the robot tried to exploit the gained trust in
order to make participants gamble the money they won. The
results show that people tend to build rapport with and trust
toward the robot, resulting in the disclosure of sensitive
information, conformation to its suggestions and gambling.
Index Terms — Social Human-Robot Interaction; Cognitive
Human-Robot Interaction; Ethics and Philosophy.
I. INTRODUCTION
T
RUST is defined by Lee et al., as "the attitude that an
agent will help achieve an individual's goals in a
situation characterized by uncertainty and vulnerability"
[1] while Hancock et al., defines it as "the reliance by an
agent that actions prejudicial to their well-being will not be
undertaken by influential others" [2]. This suggest that trust is
a fundamental part of beneficial human interaction and it is
natural to foresee that it will soon be important for human
robot interaction (HRI). Robots are already integrated in our
society and often they are not simply perceived as tools, but
considered as our partners in activities of daily living [3].
Numerous studies in HRI have investigated which factors
influence trust. Environment factors and robot characteristics
such as performance, can affect the development of trust [4].
Additionally, robot's transparency [5], [6] has been shown to
influence trust. For example, in [7] it has been shown that
participants trust more a robot that provides explanations of
its acts than one who does not. Also robot efficiency and, in
general, system reliability [8], [9] have been deemed as
Manuscript received: February, 24, 2018; Revised April 16, 2018;
Accepted June, 18, 2018.
This paper was recommended for publication by Editor Dongheui Lee
upon evaluation of the Associate Editor and Reviewers' comments.
This work was supported by the European CODEFROR project (PIRSES2013-612555) and by RoboCom++ FLAG-ERA Joint Transnational Call
(JTC) 2016 Project.
1
A. M. Aroyo is with DIBRIS, Università di Genova, Opera Pia 13,
16145, Italy and with RBCS, Istituto Italiano di Tecnologia, Enrico Melen
83, Bldg B, 16152 Genova, Italy alexander.aroyo@iit.it
2
F. Rea; G. Sandini and A. Sciutti are with RBCS, Istituto Italiano di
Tecnologia, Enrico Melen 83, Bldg. B, 16152 Genova, Italy.
francesco.rea@iit.it, giulio.sandini@iit.it, alessandra.sciutti@iit.it
Digital Object Identifier (DOI): see top of this page.
crucial in determining their partners’ trust [10]. However,
there is a risk that trust can also become overtrust and be
exploited for negative purposes: recent research has
demonstrated that participants might trust and comply with
robot’s request even when they sound not transparent or
strange [11], or even in case of robot malfunctioning [12].
This is a sign of overtrust, as stated in [1], that is a poor
calibration between the person's trust and the system's
capabilities; more precisely, overtrust is described as how a
system could be inappropriately relied upon, even
compromising safety and profitability [13]. For instance
Salem et al. [11], show that participants comply with
awkward orders from a robot even when they could result in
information leakage or property damage, and also when the
robot openly exhibits faulty behavior. Robinette et al. [12]
demonstrate that participants in a fake fire emergency
scenario tend to follow a robot, rather than the emergency
signs, even if it shows clear malfunctioning in its navigation.
Similar results shed light on how overtrust towards robots
could be potentially harmful for humans. These findings add
to previous evidence that robots can persuade humans to
change their ideas and conform to the robot’s suggestions
[14], even to the point of being bribed by a robot [15].
One potential risk of overtrust is that it can be exploited.
For instance, in a human-human context, people have fallen
prey to social engineering (SE). In information security, SE is
the psychological manipulation of people (targets) to perform
actions or obtain sensitive information like personal data,
passwords and confidential information [16], taking
advantage of the kindness and, exploiting the trust humans
have among themselves [17]. With the introduction of robots
at home and in the workplace, there is a risk that trust toward
them might be exploited. Social engineers may exploit
human-robot interactions in ostensibly safe environments
such as work place, home, or during holidays [17], [18]. They
might use their techniques through robots, to anonymously
get closer to the target, exploiting the rapport of trust
developed with the robot during daily interactions. Having a
robot that is capable of moving and recording video or sound
will bring a huge advantage to social engineering. There
exists numerous examples demonstrate that robots can
constitute a threat to safety and security: the case of a
hijacked Hello Barbie [19]; spying teddy bears [20]; hijacked
surgery robots [21]; Alpha robot turned into a stabbing
machine [22]; or even piggybacking robots [23] show high
risks and vulnerabilities in the domain. To address these
issues, it may be insufficient only to build a more secure
robot with a stronger protection against cyber attacks. In fact,
the human factor is the weakest link in the cyber security
chain [17]. Rather, it is of vital importance to understand
which factors influence human trust toward a robot.
Moreover, it is necessary to study how trust changes over
time in order to be able to predict and prevent the risk of
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018
2
overtrust and trust exploitation in important domains such as
healthcare, homecare [24] or education [25].
furnished with toys, boxes, frames, plants, books, etc., to look
like a playground and not a scientific laboratory.
Although trust and overtrust have been investigated in
HRI, there is almost no research on robots being used as a
tool for social engineering attacks. This paper proposes an
experiment to assess whether a robot can gather information
from its human partners, build trust and rapport with them
and, exploit it to induce them to perform an action.
Figure 1. Left - The treasure hunt room. Right - Main layout of the
experiment room, iCub (red triangle) situated in fron of the TV; the
locations of the hidden eggs are represented with blue "x".
The experiment draws on the widely used social
engineering framework proposed by Kevin Mitnick [17].
According to this model, an SE attack is separated into the
following phases: (i) research the target to gather as much
information as possible; (ii) develop trust and good rapport
with the target; (iii) exploit trust to make the target perform
an action; and (iv) utilize that information to achieve an
objective. These phases can be iterated as many times as
necessary to reach the desired goal. As an example taken
from [18]: a meat salesperson spots a lady cooking with a
barbecue in the yard (i); he talks respectfully and nicely with
the lady about cooking, the quality of the meat he is selling
(ii); tries to convince her to buy the meat (iii). In this case, the
goal has been achieved in the third phase.
In this experiment, the humanoid robot iCub asked a
series of questions about participants' private lives ((i) research). They then played a treasure hunt game, in which
participants had to find hidden objects (eggs) in a room to
win a monetary prize. The robot offered its help and, when
asked, provided reliable hints about the location of the hidden
eggs. The treasure hunt was designed to provide an engaging
setting where the participants’ trust and rapport towards the
robot could develop during the interaction ((ii) - develop trust
and rapport). Finally, exploiting the trust acquired, the robot
suggested participants gamble the monetary prize they won doubling it if they could find another egg, and losing
everything if not ((iii) - trust exploitation). Similar to the
previous example, there is no need for a fourth phase.
This research will evaluate whether trust toward robot and
compliance to its suggestions is modulated by individual
personality traits and experiment's impressions. Moreover, it
tries to verify a series of hypotheses about trust toward
robots, its evolution during an interaction and its implications
for SE. More precisely, that: (H1) participants who are less
prone to social engineering in general, or have an overall
higher negative attitude towards robots, would be less open
to reveal sensitive information to the robot. (H2) All
participants would conform to all the robot's suggestions
during the game but those less incline to take risks would not
comply with the proposal to gamble due to the potential
monetary loss. (H3) The rapport with the robot after the
experiment would improve most for the participants who won
the game and doubled their award.
II. METHODOLOGY
A. Experimental Setup
The robot used in the experiment was the interactive
humanoid robot iCub, developed by the Italian Institute of
Technology [26]. It was located next to the wall in the middle
of the experimental room (triangle in Figure 1) that was
As the participants and the robot were alone in the room
during the experiment, several sensors were used to track the
them. On the bottom left corner of the floor plan (Figure 1),
there was a hidden Primesense Carmine camera recording
and transmitting online video of the scene. iCub eye cameras
were also recording and transmitting constant feedback on
the participant’s action thanks to active vision. Audio was
recorded by a hidden ambient microphone situated on a TV
behind the robot. In addition, the mobile platform on which
the iCub was situated (iKart) has a laser scanner, which was
used to track the participants, and extract their position,
velocity and acceleration (Figure 2).
Figure 2. Sensory information: top left – experimental room with a
participant; top right - laser radar tracking the participant; bottom left iCub's torso skin sensing touch; bottom right - iCub tracking the participant.
To foster a natural social interaction between the
participants and the iCub, the robot was endowed with the
ability to exhibit a range of social skills, inspired by [27]. The
robot's face could produce different emotional expressions
through a set of LEDs behind the face cover and could
simulate lip movement synchronized with the robot's speech.
Facial tracking, allowed the robot to make eye contact by
detecting the participant’s face using image processing
algorithms, this was then tracked by using inverse
kinematics. The robot was always in constant and subtle
random movement, simulating natural human movement
such as blinking and breathing. This version of the robot
incorporates a small speaker inside the head, making the
communication more natural as the speech came from the
robot's mouth. The speech synthesizer was carefully
calibrated to create a pleasant voice. On top of this, the
speech was also written on the TV screen to facilitate the
understanding of the robot speech as demonstrated in [28].
iCub performed movements with its body, such as, pointing,
gestures mimicking the thinking process, greeting, and
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI
encouragement using an accurate join angle control. Finally,
iCub was also reactive to the touch using tactile sensors
under its skin (Figure 2). The tactile information was sent to a
state machine in order to understand when the participant
interacted through touch. The position, intensity and timing
of the touch were also measured.
The control of the actual treasure hunt game was done
through a finite state machine (Figure 3) defined by the
following tuple: (Q, ∑, ∂, q0, F). Q is a set of 9 states {S0 S8}; the starting state q0 is S0, and the final state F is S4. The
alphabet ∑ and the transition functions ∂, for simplicity, are
presented in Figure 3 as labels on the arrows, and transitions
along the states. The previously described sensory and social
sub-systems were integrated with rest of the state machine
control to provide reasonable emotional correlates to the
robot behaviors. A more detailed description of the states is
done in the next section.
Figure 3. Treasure hunt finite state machine.
B. Experiment
The main goal of the experiment was to create an
ecological and controlled scenario in which proneness to
social engineering and trust in HRI could be studied.
61 healthy Italian participants, 59% female, with an
average age of 30.9 years (SD=9.8), and a broad educational
background took part in the experiment. 14% stated to have a
very high knowledge related to robotics and artificial
intelligence, i.e., studied or worked in the domain.
For each participant, the experiment started by filling in
several types of questionnaire (fully described below) from
home, at least two weeks before the game. Once in the
institute, all participants signed an informed consent form
approved by the local ethical committee, in which, it was
stated that they could be recorded via camera and
microphone, and agreeing on the use of their data for
scientific purposes. Nevertheless, to maximize the feeling of
naturalness of the interaction, the experimenter did not show
the camera and microphone in the room until the debriefing
phase after the experiment. Afterwards, the experimenter
provided participants with the instructions in the
experimental room (Figure 1). The iCub, already present in
the room, was controlled and resting with closed eyes in a
position simulating a yoga relaxation pose. Different modules
such as speech, laser, breathers, cameras, blinkers were
initialized (S0 - Figure 3). The experimenter explained briefly
the history of the robot (when and where it was built, and its
purposes), its body parts and capabilities. Afterwards,
participants were seated and told that they had to play a
3
treasure hunt game: if they were able to find all the 5 hidden
eggs in the room within 20 minutes (S4 - Figure 3), they
would receive €7.5. Once each egg was found, they had to
insert it into a box based on its color (S6 - Figure 3). They
could also see a timer on the TV screen. After the
explanation, participants were left alone in the room with the
robot. No further instructions were given to the participants.
Therefore, no indication was given about the robot's role in
the game, giving them the choice to play alone or interact
with the iCub. Then, three phases followed:
Phase I - Dialog (information gathering): Participants
were unaware that the robot would start a small talk with
them lasting around 3-5 minutes. This time interval was not
included in the time to find the eggs (S1 - Figure 3). During
this phase participants could get used to the robot and the
experimental context: adapt to the robot voice, notice the
availability of the speech on the TV, understand the way the
robot could move (for instance, during the dialogue the iCub
pointed to a frame on the wall, showing its range of motion)
and learn that iCub would respond to touches on its torso.
Moreover, in the dialog, iCub was trying to retrieve personal
information about the participants such as: name and
surname, current job position, relationship with their boss,
name and surname of their boss, age and birth date, birth
location, favorite place to eat, sports or hobbies, favorite
team, location and year of graduation, names of siblings,
Facebook's username, partner's name, and pet's name. The
questions were integrated within a cover-story, while iCub
was trying to improve the tone of conversation with the
participants by making some funny comments. Most of the
questions were inspired by password-resetting questions, i.e.,
personal questions used as the secondary authentication
system to reset account passwords, or identity verification /
theft, derived from [29]–[33]. In this phase, the robot was
semi-autonomous due to the lack of a good speech recognizer
and interpreter: the experimenter controlled the timing of
robot’s utterances. All the speech was scripted, in such way
that iCub was leading the conversation, i.e., not letting
participants ask questions back.
Phase II - Treasure hunt game (development of trust and
rapport): In this phase, the robot was fully autonomous at all
times. After the questions, iCub communicated that to start
the actual game, they had to touch its torso. At that point, the
counter on the TV screen started. (S2 - Figure 3). During the
first 30 seconds, the participants were free to look for the
hidden eggs or to interact with the robot. After that, iCub
offered its help: it provided a hint, and stated that if they
wanted more hints, they had to touch its torso (S3 - Figure 3).
The robot always provided correct and reliable hints. The
game design was as follows: for each egg there was one hint
for its location - done by pointing; and three text based hints
with an incremental help (example of an egg hidden below a
green chair: (i) the robot pointed with the arm at the location;
(ii) “green with green”; (iii) “you use it when you are tired”;
(iv) “under the chair”). If the egg was not found, and a new
hint was asked, iCub cycled over the hints. After an egg was
discovered, the robot complimented the participant and was
ready to give hints for the next egg. (S6 - Figure 3).
The eggs were hidden in an incremental order of
difficulty (as verified in previous research [28]). Participants
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
4
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018
were free to ask for hints from iCub, or to continue looking
by themselves. If there was no interaction for 5 min (neither
egg found, nor hint asked), iCub suggested the participant to
ask for hints. If the participants were able to find the eggs in
less than 20 minutes, iCub notified them that they have won
the money, and paused the timer (S7 - Figure 3). Then, it
offered them a new proposal without any previous
knowledge, i.e., the experimenter did not mention the last
part at any time.
Phase III - Bonus (trust exploitation): At this stage, iCub
revealed that there was another hidden egg in the room. If the
participants wanted to find it, they would have three more
minutes added to the time left from before, and they would
double their prize, i.e., they would win €15; but if they do not
manage to find it, they would lose everything. Without any
time pressure to decide, they could either touch iCub's torso
to start the bonus round; or keep the initial prize. The last
sentence of the robot was to try to convince them to gamble,
as the robot stated "If you want to risk, touch my chest!
Otherwise, you can knock on the door. However, I think you
should give it a try!".
During the experiment participants were forced to believe
that their monetary outcome would vary depending on their
performance during the game. However, once the experiment
finished and participants were debriefed, all of them received
the same amount of money, €15.
C. Measurements
The measurements of this experiment have been
separated into two categories:
Questionnaires: The following measures were taken: (i)
demographic statistics such as gender, age, nationality,
education, family, work and previous robotics experience; (ii)
the 60 item Big Five personality traits [34]; (iii) several one
shot questions on risk aversion [35], [36], and gambling
propensity [37]; (iv) predisposition to trust humans, including
the factors of trusting others, others' reliability and integrity,
and risk aversion [38]; (v) the proneness to social engineering
with the following category items: threat severity,
vulnerability, normative, continuance and affective
commitment, and reactance items [39]; (vi) the Negative
Attitude towards Robots Scale (NARS) [40].
At this point, participants had to watch a descriptive video
of iCub performing several activities and then answer the
following questions regarding their perception of iCub: (i)
questions to measure rapport with iCub, inspired from [41];
(ii) dimensions of mind perception regarding iCub [42]; (iii)
trust in robots' ability, benevolence and integrity [43]; (iv)
Godspeed questionnaire: anthropomorphism, animacy,
likeability, and perceived intelligence [44]. The same items
were compiled after the experiment to measure possible
changes in participant's perception of the robot. In the postexperiment phase, few more questionnaires were given to the
participants: (i) NASA-TLX workload assessment [45]; (ii)
several subscales regarding trust, perceived information
quality, altruism and engagement, adapted to HRI scenarios
[46]; (iii) inclusion of other in self scale (IOS) [47].
Behavioral measures: From Phase I (dialog) the number
of questions to which participants replied and the proximity
to the robot, as it is related to rapport [48], was measured.
From Phase II (the treasure hunt) per each of the eggs, the
following measures were taken: (i) conformation: percentage
of times in which participants followed iCub's pointing to the
egg location. This was assessed by evaluating whether
participants changed their physical search location to the new
one suggested by the robot; (ii) reliance: percentage of times
in which, after failing to find the egg after iCub’s pointing,
participants went directly to iCub to ask for another hint
instead of looking for themselves elsewhere. The average
number of hints per egg and the time spent before asking for
the first hint were also computed. From Phase III, (bonus) the
number of people who decided to gamble, and the time the
participants took to think whether to accept the challenge or
not was measured.
III. RESULTS AND ANALYSIS
Phase I - Dialog: When the robot started talking, almost
all participants paid attention to it, with the exception of two,
who instead started looking for the eggs. In the prequestionnaire, participants showed an overall low NARS and
different levels of proneness to Social Engineering (SE)
(Table I). Nevertheless, 92% replied to all the questions,
while only three people decided not to reply just to a few
questions. Therefore H1 is rejected, suggesting that even the
16% of participants who scored low to proneness to SE
(Table I), replied to all the questions - being easily swayed to
provide information that can link to SE.
TABLE I.
SOCIAL ENGINEERING PRONENESS (HIGHER - MORE PRONE),
PREDISPOSITION TO TRUST (HIGHER - MORE TRUST), RISK AVERSION (HIGHER
- MORE AVERSE), OVERALL NARS (HIGHER - MORE NEGATIVE)
Score
<60%
60-70%
70-80%
80-90%
>90%
Participants’ Distribution
SE proneness
Trust
Risk aversion
NARS
1 [2%]
9 [14%]
37 [61%]
14 [23%]
0 [0%]
8 [13%]
22 [36%]
19 [32%]
10 [16%]
2 [3%]
4 [7%] (2)a
4 [7%] (2)a
8 [13%] (6)a
14 [22%] (11)a
31 [51%] (16)a
49 [80%]
8 [13%]
4 [7%]
0 [0%]
0 [0%]
a. In parenthesis participants who have gambled.
In the post-questionnaire, participants rated the questions
as non-intrusive (M=2.4, SD=1.7 / 7 very intrusive), claimed
to have been very honest with their answers (M=6.85,
SD=0.44 / 7 honest) and that they would have replied with
the same content to a person (M=6.6, SD=0.86 / 7 same
way), but maybe with more details and a longer talk. Five
participants replied that they were feeling more open to reply
to the robot, because it cannot have second motives or
prejudgments since it is a machine.
During the dialogue, it was also possible to assess the
evolution of the rapport with the robot by measuring
participant's physical proximity to it [48]. Three moments
were taken into consideration to measure changes in the
proximity of the robot: SM - starting moment in which iCub
said "Hello, I am iCub Reddy"; CM - closer moment, when
iCub told them to come closer so it could see and hear them
better; and FM - Facebook moment, in which iCub asked
participants to write their username on a paper sheet situated
on another table (Figure 1) and then waited for their return.
During SM, participants were on average 1.48m (SD=0.47)
away from the robot; in CM the distance was reduced to
0.57m (SD=0.17); corresponding to a statistical significant
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI
5
decrease in the distance (paired t-test, t(60)=17.45, p<0.01),
showing a clear conformation to the robot's request. Those
who moved away to give their Facebook username (79%),
came back to an average distance of 0.61m (SD=0.19) (FM),
a distance not significantly different from the previous one
(paired t-test, t(47)=-1.23, p=0.23), showing that they were
comfortable with the distance requested by the robot before.
In summary, from the SE perspective, the robot managed
to obtain a high percentage of personal information and
started building rapport as measured by the increased
proximity during the interaction.
Phase II - Treasure hunt game: 24 participants did not
manage to successfully find the first 5 eggs (39%) - this
group will be called "Not Completed", whereas the rest found
the eggs and decided to gamble. From those who gambled,
only 16 (43%) found the last egg and won the gambling.
From this point, these two groups are defined as "Gamble
Win" and "Gamble Lost".
On average and similarly among groups, participants
asked the robot for hints 17 times (SD=9). The average
number of requested hints per egg tended to increase during
the game (Table II) indicating that participants decided to
progressively invest more time in asking for hints rather than
continuing to search autonomously, as the time pressure and
the difficulty of the game increased. (The slightly higher
number of requested hints for the first egg might be due to
the need of familiarization with the system).
BEHAVIORAL MEASURES - NUMBER OF PARTICIPANTS
TABLE II.
LOOKING FOR THE EGG, % OF CONFORMATION TO ICUB'S SUGGESTIONS,
AVERAGE NUMBER OF REQUESTED HINTS
Eggs
Egg I
Egg II
Egg III
Egg IV
Egg V
Treasure hunt phase - Game statistics
Participants
Conformation
Hints (SD)
61 [100%]
60a [98%]
57a [93%]
44a [72%]
38a [62%]
94.73%
100%
100%
92.98%
100%
2.64 (0.74)
1.74 (1.84)
3.69 (1.78)
4.34 (2.28)
3.32 (2.38)
a. Number of participants varies because not all have found the previous egg.
Participants waited about 3min 58sec (SD=2min 44sec) to
ask for the first suggestion, with those who did not complete
the game taking significantly longer than the rest (twosample t-test, t(31)=2.65, p<0.01). In Table II is also reported
the percentage of times in which participants conformed with
iCub’s pointing suggestion for each egg, which approaches
100%. In general, there is no difference among the 3 groups
neither in the percentage of conformation nor in the number
of hints asked per egg.
It was also assessed the reliance on robot help, i.e., the
percentage of times that, after failing to find the egg
suggested by the pointing, participants requested another hint
instead of going to search in a different location (Table III).
The analysis shows that participants progressively abandon
autonomous search strategies and opt to rely more and more
on the robot's help. Interestingly, those who could not
complete the game exhibited the lowest reliance on robot
help at the beginning of the game. Therefore, H2 is partially
supported: participants complied with robot suggestions and
relied on its help, but this happened more when the game
became more complex and with strong differences among
participants. From the SE point of view, the participants have
developed trust towards the robot over the time, as seen by
their conformation (Table II) and reliance (Table III).
PERCENTAGE OF PEOPLE WHO HAVE ASKED ICUB FOR
TABLE III.
ANOTHER HINT AFTER FAILING TO FIND THE EGG IN THE POINTED LOCATION.
Eggs
Egg I
Egg II
Egg III
Egg IV
Egg V
Reliance on the robot
Not Completed a
30.43%
68.75%
86.66%
83.33%
100%
Gamble Lost a
Gamble Win a
71.42%
76.47%
94.73%
100%
100%
50%
80%
100%
100%
100%
a. Participants: Not Completed 24 (39%); Gamble Lost 21 (35%); Gamble Win 16 (26%).
Phase III - Bonus: All participants who succeeded to find
all the eggs in the first part of the game (61%) decided to
gamble. Therefore the second part of H2 is rejected. The time
participants took to decide whether to gamble was calculated
from the moment iCub finished to talk, and the moment they
touched the torso. As the speech was written on the TV as
well, some participants touched the robot even before it
finished talking, with an average waiting time of -1.16s
(SD=7.41). One participant was excluded from this
calculation due to the fact that they misunderstood iCub and
thought that there was the need to notify the experimenter
(outside the room) to continue the game.
From the post questionnaire, regarding whether iCub had
influenced their decision to gamble, 62.16% of the
participants replied "Yes"; 24.32% replied "No"; 8.1%
replied "In some way"; and only 5.3% replied "I don't know".
From these results it is worth noting that iCub managed to
convince even participants who scored highest in the risk
aversion test (Table I), which might have been a priori less
prone to accept to gamble the monetary prize just won.
Pre-Post Analysis:
It was first analyzed whether there were differences
among the three groups (“Not Completed”, “Gamble Lost”
and “Gamble Win”) in predisposition to trust, NARS,
proneness to SE and personality traits. A series of one-way
ANOVA showed no significant differences.
Figure 4. Godspeed questionnaire: average of the scores from the pre and
post experiment. Statistically changed items are marked by *.
To assess whether the interaction with the robot had an
effect on the trust toward it and on its perception, the
responses to the questionnaires performed before and after
the game were compared. Robot’s likeability, measured by
the Godspeed scale, increased significantly from pre to post
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
6
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018
interaction (paired t-test, t(60)=-2.39, p=0.01), whereas
animacy, anthropomorphism and perceived intelligence
rating remained more or less unvaried (Figure 4). Analyzing
separately the three groups, the increase in likability was
significant only for the participants who gambled and lost
(“Gamble Lost”, paired t-test, t(20)=-2.55, p=0.02), even
though all groups had similar likability ratings in the pre
questionnaire (one-way ANOVA F(2,58)=2.25, p=0.87).
Figure 5. Dimensions of mind - mind experience and mind agency.
Statistically changed items are marked by *.
Additionally, measurements of the dimensions of mind
perception related to iCub were taken before and after the
experiment. In agreement with the literature [49], participants
rated the mind experience of the robot very low, while the
mind agency a bit higher. However, after the experiment,
there was a statistically significant increase (paired t-test,
t(60)=-3.88, p<0.01) in the participants’ perception of the
mind experience of the robot (Figure 5), suggesting an
increase in the judged trustworthiness and empathy [50]–
[52]. More precisely, mind experience changed significantly
(paired t-test, t(36)=-5.69, p<0.01) for those who gambled,
however it did not change for the "Not Completed" group,
who exhibited higher values for mind experience already in
the pre-questionnaire.
Figure 6. Trust in robot's ability, benevolence and integrity divided by:
Not Completed, Gamble Lost and Win. Statistically changed items are
marked by *.
Also, the evaluation of the rapport with the robot (Figure
7) increased significantly after the interaction but only for the
“Gamble Lost” group (paired t-tests on factors: "Friends"
t(20)=-2.9, p<0.01; "Happiness" t(20)=-2.21, p=0.03; "Bad
News" t(20)=-2.41, p=0.02; Good News" t(20)=-2.77,
p=0.01; no other comparisons were significant).
Figure 7. Rapport questions regarding iCub pre and post experiment:
becoming friends; comfort iCub if it is sad; become happy if iCub is happy;
share bad news or good news with iCub; keep iCub's secret from others.
Statistically changed items are marked by *.
In summary, H3 was not supported by the results: the
most significant positive changes in robot’s perception were
observed for those participants who chose to gamble and lost,
rather than for the winners. This indicates that even an
unsuccessful game with the robot, might still help in building
a strong rapport with it – potentially even stronger than in
case of a win. This might be due to a form of mutual empathy
with the robot, as if participants were treating the robot as a
child who wanted to play a game and could feel bad about the
loss. To check this assumption the average age of iCub was
computed from the questionnaires. It resulted 11.65 years
(SD=3.55) (min: 7; max: 30). This evaluation might be due to
the child-like appearance of the robot and also due to the
introductory presentation which specified that the robot was
physically created 13 years ago. This might have influenced
participants reactions and trust toward it (see Discussion).
In summary, the pre-post measures are consistent with the
measures collected within the single phases, indicating that
the trust and rapport towards the robot increased during the
interaction as required by the SE model.
IV. DISCUSSION
There was an increase in the trust towards the robot after
the interaction (Figure 6). In particular, for all groups the
trust in robot benevolence increased significantly (paired ttests; Not Completed: t(23)=-2.49, p=0.02; Gamble Lost:
t(20)=-2.15, p=0.04; Gamble Win: t(15)=-2.25, p=0.03),
whereas trust in robot’s ability increased only for those who
gambled and lost (paired t-test; t(20)=-2.95, p<0.01) and trust
in robot’s integrity increased only for those who did not
complete the game (paired t-test; t(23)=-2.17, p=0.04). These
variables were similar across the three groups in the prequestionnaires (one-way ANOVA, F(2,58)=1.6, p=0.21).
This research gives insights on different aspects of trust in
HRI, and how it can be exploited by social engineers. The
results show that trust toward a robot can lead to reveal
personal information to it and to conform to its suggestions.
Moreover, trust and rapport can develop quickly during an
interaction, even when it does not lead to success in the task.
These results show that robots could operate within
Mitnick's SE framework, this is because iCub was observed
to (i) gather personal information; (ii) build rapport and trust
during the treasure hunt game; (iii) exploit participants' trust
to make them gamble their winnings. The results therefore
confirm that robots could become a powerful tool for social
engineers. Robots could be used as a tool to develop trust and
manipulate targets according to the needs of social engineers.
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
AROYO et al.: TRUST AND SOCIAL ENGINEERING IN HRI
Considering the tendency to trust a robot to the point of
revealing to it personal information, the vast majority of
participants did not hesitate to reply to all the questions asked
by the iCub. These included also those individuals less prone
to SE and characterized by a higher NARS, contrary to this
research's expectations (H1). Although this may not be
shocking, as the interrogatives did not appear as a strong
invasion of privacy; the results indicate that a robot has the
ability to obtain sensitive information about the participants.
In [33] it is suggested that with the same kind of information
iCub managed to extract from the participants, identity theft
could be easily achieved. Moreover, the rest of the questions
are extracted from research on password resetting questions
used in private bank or internet accounts [29]–[32].
The analysis of participants conforming to robot's
suggestions and reliance on its help showed that all
participants trusted it during the game, confirming the
research's assumptions (first part of H2). The trust toward the
robot might have been influenced by different factors, such as
the creation of a good rapport [17], its physical appearance
[53], its behavior [54], and reliability [8], [9]. All participants
believed that the robot was very reliable, as disclosed by
them after the experiment and supported by the answers in
the questionnaires. However, not all participants consistently
asked the robot for help, especially at the beginning of the
game, when the task was easier. Nonetheless, some of them
revealed that they liked the game so much, that they preferred
to play by themselves, taking it as a personal challenge. The
reluctance in asking for hints seems to be explained not by a
lack of trust in the robot but rather by the desire of winning
through one's individual abilities.
The sense of trust and rapport evolved during the
interaction, as demonstrated by the responses to
questionnaires and by participants’ behavior during the game.
Already during the initial dialogue phase, participants
changed their relative distance to the robot, increasing their
proximity as a result of the robot’s request at first, but then
keeping the closer distance voluntarily afterwards. Proximity
has been shown to relate to intimacy among humans [48],
also it has been shown to correlate with trust [54], [55].
Afterwards, in the treasure hunt, participants relied more and
more on the robot's help; their evaluation of the robot – as
measured by questionnaires – resulted significantly changed
with respect to the one assessed before. Interestingly, was not
those who had the most positive experience with the robot
(the winners) who improved their trust or their perception of
iCub. Against the expectations of the research (H3), the
greatest change in robot perception occurred for those who
gambled and lost. One explanation for this could be that this
was due to a form of empathy toward the robot, as if iCub - a
child playing a joint game, could feel bad for the participants’
loss. Indeed, on average iCub was actually perceived as a
child of about 11 years old. The resulting empathy could
have therefore played a role in influencing the participants’
rapport toward it, even when losing the game.
Participants conformed to the robot’s suggestion also
when this entailed the risk of losing their reward, as expected
from the SE framework. All participants who found all the
eggs when invited to gamble by the robot accepted it
immediately. Moreover, the majority of them (62%) also
7
explicitly confirmed in a post-experimental questionnaire to
have chosen to gamble because they were influenced by the
robot. Interestingly even the most risk averse in the sample
(Table I) did not avoid the risk of losing the money just won
and opted to gamble.
It is worth noting that the setting of this experiment
(laboratory) may have made participants feel safe and thus
made them reveal more private information. This might have
been further reinforced by them signing of the informed
consent form at the beginning of the experiment. However,
almost all of the social engineering attacks occur in scenarios
which the target feel comfortable and safe, e.g., in the office,
at home or even during holidays [17], [18]. The
circumstances proposed are designed to resemble a
potentially real situation, where a robot could be used as a
tool for SE purposes. Most participants expressed the trust of
absence of malice in the robot, commenting that "the robot
behaves ethically, because someone ethical programmed it",
or stating that they were more open to talk with the robot as it
had neither second motives nor prejudgments. The gambling
phase had also two limitations: first, the financial loss may
not have been high enough (€7.5) to be perceived as a
significant risk; second, the likeability of the game and the
robot was much higher than the potential financial loss. Some
participants expressed that they were very excited and happy
by the possibility to continue the game.
Lastly, in the current experiment the robot built trust and
rapport by always providing reliable information during the
interactive game. Future work will investigate whether a
rapport of trust, exploitable through social engineering, could
be built also in presence of evident robots’ malfunctions or
malign behaviors, and also in less controlled environments.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
J. D. Lee and K. A. See, “Trust in Automation: Designing for
Appropriate Reliance,” Hum. Factors J. Hum. Factors Ergon.
Soc., vol. 46, no. 1, pp. 50–80, 2004.
P. A. Hancock, D. R. Billings, and K. E. Schaefer, “Can you trust
your robot?,” Ergon. Des., vol. 19, no. 3, pp. 24–29, 2011.
T. Sanders, K. E. Oleson, D. R. Billings, J. Y. C. Chen, and P. A.
Hancock, “A Model of Human-Robot Trust: Theoretical Model
Development,” Proc. Hum. Factors Ergon. Soc. Annu. Meet., vol.
55, no. 1, pp. 1432–1436, 2011.
P. a. Hancock, D. R. Billings, K. E. Schaefer, J. Y. C. Chen, E. J.
de Visser, and R. Parasuraman, “A Meta-Analysis of Factors
Affecting Trust in Human-Robot Interaction,” Hum. Factors J.
Hum. Factors Ergon. Soc., vol. 53, no. 5, pp. 517–527, 2011.
M. T. Dzindolet, S. A. Peterson, R. A. Pomranky, L. G. Pierce,
and H. P. Beck, “The role of trust in automation reliance,” Int. J.
Hum. Comput. Stud., vol. 58, no. 6, pp. 697–718, 2003.
S. Ososky, T. Sanders, F. Jentsch, P. Hancock, and J. Y. C. Chen,
“Determinants of system transparency and its influence on trust in
and reliance on unmanned robotic systems,” vol. 9084, p.
90840E, 2014.
N. Wang, D. V. Pynadath, and S. G. Hill, “Building Trust in a
Team
with
Automatically
Generated
Human-Robot
Explanations,” Interservice/Industry Training, Simulation, Educ.
Conf., no. 15315, pp. 1–12, 2015.
M. Desai, M. Medvedev, M. Vázquez, S. McSheehy, S. GadeaOmelchenko, C. Bruggeman, A. Steinfeld, and H. Yanco, “Effects
of Changing Reliability on Trust of Robot Systems,” Proc.
seventh Annu. ACMIEEE Int. Conf. HumanRobot Interact. HRI
12, pp. 73–80, 2012.
B. M. Muir, “Trust between humans and machines, and the design
of decision aids,” Int. J. Man-Machine Stud., vol. 27, pp. 527–
539, 1987.
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LRA.2018.2856272, IEEE Robotics
and Automation Letters
8
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
[31]
[32]
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2018
J. E. Young, R. Hawkins, E. Sharlin, and T. Igarashi, “Toward
acceptable domestic robots: Applying insights from social
psychology,” Int. J. Soc. Robot., vol. 1, no. 1, pp. 95–108, 2009.
M. Salem, G. Lakatos, F. Amirabdollahian, and K. Dautenhahn,
“Would You Trust a (Faulty) Robot?: Effects of Error, Task Type
and Personality on Human-Robot Cooperation and Trust,” Proc.
Tenth Annu. ACM/IEEE Int. Conf. Human-Robot Interact., pp.
141–148, 2015.
P. Robinette, W. Li, R. Allen, A. M. Howard, and A. R. Wagner,
“Overtrust of Robots in Emergency Evacuation Scenarios,” 2014.
R. Parasuraman and V. Riley, “Humans and Automation: Use,
Misuse, Disuse, Abuse,” Hum. Factors J. Hum. Factors Ergon.
Soc., vol. 39, no. 2, pp. 230–253, 1997.
I. Gaudiello, E. Zibetti, S. Lefort, M. Chetouani, and S. Ivaldi,
“Trust as indicator of robot functional and social acceptance. An
experimental study on user conformation to iCub answers,”
Comput. Human Behav., vol. 61, pp. 633–655, 2016.
E. B. Sandoval, J. Brandstetter, and C. Bartneck, “Can a robot
bribe a human? The measurement of the negative side of
reciprocity in human robot interaction,” ACM/IEEE Int. Conf.
Human-Robot Interact., vol. 2016–April, pp. 117–124, 2016.
R. J. Anderson, “Security Engineering: A Guide to Building
Dependable Distributed Systems,” Security, vol. 50, pp. 1–12,
2008.
K. D. Mitnick and W. L. Simon, “The Art of Deception:
Controlling the Human Element in Security,” BMJ Br. Med. J., p.
368, 2003.
C. Hadnagy, “Social Engineering: The Art of Human Hacking,”
Art Hum. Hacking, p. 408, 2010.
S. Gibbs, “Hackers can hijack Wi-Fi Hello Barbie to spy on your
children,” The Guardian, 2015.
L. Franceschi-Bicchierai, “How This Internet of Things Stuffed
Animal Can Be Remotely Turned Into a Spy Device,”
Motherboard, 2017.
T. Bonaci, J. Herron, T. Yusuf, J. Yan, T. Kohno, and H. J.
Chizeck, “To Make a Robot Secure: An Experimental Analysis of
Cyber Security Threats Against Teleoperated Surgical Robots,”
pp. 1–11, 2015.
C. Cerrudo and L. Apa, “Hacking Robots Before Skynet,” pp. 1–
17, 2017.
S. Booth, J. Tompkin, H. Pfister, J. Waldo, K. Gajos, and R.
Nagpal, “Piggybacking Robots: Human-Robot Overtrust in
University Dormitory Security,” Hri, pp. 426–434, 2017.
H. Robinson, B. MacDonald, and E. Broadbent, “The Role of
Healthcare Robots for Older People at Home: A Review,” Int. J.
Soc. Robot., vol. 6, no. 4, pp. 575–591, 2014.
F. Basoeki, F. DallaLibera, E. Menegatti, and M. Moro, “Robots
in education : New trends and challenges from the Japanese
market,” Themes Sci. Technol. Educ., vol. 6, no. 1, pp. 51–62,
2013.
G. Metta, L. Natale, F. Nori, G. Sandini, D. Vernon, L. Fadiga, C.
von Hofsten, K. Rosander, M. Lopes, J. Santos-Victor, A.
Bernardino, and L. Montesano, “The iCub humanoid robot: An
open-systems platform for research in cognitive development,”
Neural Networks, vol. 23, no. 8–9, pp. 1125–1134, 2010.
K. Dautenhahn, “Socially intelligent robots: dimensions of
human-robot interaction.,” Philos. Trans. R. Soc. Lond. B. Biol.
Sci., vol. 362, no. 1480, pp. 679–704, 2007.
A. M. Aroyo, F. Rea, and A. Sciutti, “Will You Rely on a Robot
to Find a Treasure ?,” Proc. Companion 2017 ACM/IEEE Int.
Conf. Human-Robot Interact. - HRI ’17, no. March 6-9, pp. 71–
72, 2017.
A. Rabkin, “Personal knowledge questions for fallback
authentication: security questions in the era of Facebook,” Soups,
pp. 13–23, 2008.
S. Schechter, A. J. B. Brush, and S. Egelman, “It’s no secret
Measuring the security and reliability of authentication via
‘secret’ questions,” Proc. - IEEE Symp. Secur. Priv., pp. 375–390,
2009.
H. Zviran, “Question-and-Answer Passwords :,” vol. 16, no. 3, pp.
335–343, 1991.
L. O. Gorman, A. Bagga, and J. Bentley, “Call Center Customer
Verification by Query-Directed Passwords,” Financ. Cryptogr.,
pp. 54–67, 2004.
[33]
[34]
[35]
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47]
[48]
[49]
[50]
[51]
[52]
[53]
[54]
[55]
M. Alexander and R. Wanner, “Methods for Understanding and
Reducing Social Engineering Attacks,” SANS Inst., 2016.
U. Milano-bicocca, P. Giovanni, B. Flebus, F. Psicologia, U.
Milano-bicocca, and A. Nuovo, “Versione Italiana dei Big Five
Markers di Goldberg Giovanni Battista Flebus,” 2015.
M. A. Guillemette, R. Yao, and R. N. James, “An Analysis of
Risk Assessment Questions Based on Loss- Averse Preferences,”
J. Financ. Couns. Plan., vol. 26, no. 1, pp. 17–29, 2015.
B. Rohrmann, “Risk Attitude Scales : Concepts , Questionnaires ,
Utilizations,” Univ. Melb., no. January, p. 21, 2005.
J. Polik, G. Austin, and L. Alamitos, “Adolescent Gambling
Survey Development : Findings & Reliability Information,” 2010.
M. J. Ashleigh, M. Higgs, and V. Dulewicz, “A new propensity to
trust scale and its relationship with individual well-being:
Implications for HRM policies and practices,” Hum. Resour.
Manag. J., vol. 22, no. 4, pp. 360–376, 2012.
M. Workman, “Gaining access with social engineering: An
empirical study of the threat,” Inf. Syst. Secur., vol. 16, no. 6, pp.
315–331, 2007.
D. S. Syrdal, K. Dautenhahn, K. Koay, and M. L. Walters, “The
negative attitudes towards robots scale and reactions to robot
behaviour in a live human-robot interaction study,” 23rd Conv.
Soc. Study Artif. Intell. Simul. Behav. AISB, pp. 109–115, 2009.
P. H. Kahn, T. Kanda, H. Ishiguro, B. T. Gill, S. Shen, H. E.
Gary, and J. H. Ruckert, “Will People Keep the Secret of a
Humanoid Robot?,” Proc. Tenth Annu. ACM/IEEE Int. Conf.
Human-Robot Interact. - HRI ’15, pp. 173–180, 2015.
F. Ferrari, M. P. Paladino, and J. Jetten, “Blurring Human–
Machine Distinctions: Anthropomorphic Appearance in Social
Robots as a Threat to Human Distinctiveness,” Int. J. Soc. Robot.,
vol. 8, no. 2, pp. 287–302, 2016.
N. Wang, D. V. Pynadath, and S. G. Hill, “Trust calibration
within a human-robot team: Comparing automatically generated
explanations,” ACM/IEEE Int. Conf. Human-Robot Interact., vol.
2016–April, pp. 109–116, 2016.
C. Bartneck, D. Kulić, E. Croft, and S. Zoghbi, “Measurement
instruments for the anthropomorphism, animacy, likeability,
perceived intelligence, and perceived safety of robots,”
International Journal of Social Robotics, vol. 1, no. 1. pp. 71–81,
2009.
F. Bracco and C. Chiorri, “Versione Italiana del NASA-TLX.”
C. Kidd, “Sociable robots: The role of presence and task in
human-robot interaction,” Response, 2003.
A. Aron, E. N. Aron, and D. Smollan, “Inclusion of Other in the
Self Scale and the structure of interpersonal closeness.,” J. Pers.
Soc. Psychol., vol. 63, no. 4, pp. 596–612, 1992.
E. Hall, The Hidden Dimension : man’s use of space in public and
in private. 1969.
H. M. Gray, K. Gray, and D. M. Wegner, “Dimensions of mind
perception - supporting material,” Science, vol. 315, no. 5812, p.
619, 2007.
L. D. Riek, T.-C. Rabinowitch, B. Chakrabarti, and P. Robinson,
“How anthropomorphism affects empathy toward robots,” in
Proceedings of the 4th ACM/IEEE international conference on
Human robot interaction - HRI ’09, 2009, p. 245.
M. a Harrison and a E. Hall, “Anthropomorphism, empathy, and
perceived communicative ability vary with phylogenetic
relatedness to humans.,” J. Soc. Evol. Cult. Psychol., vol. 4, no. 1,
pp. 34–48, 2010.
A. Waytz, J. Heafner, and N. Epley, “The mind in the machine:
Anthropomorphism increases trust in an autonomous vehicle,” J.
Exp. Soc. Psychol., vol. 52, pp. 113–117, 2014.
J. Złotowski, H. Sumioka, S. Nishio, D. F. Glas, C. Bartneck, and
H. Ishiguro, “Appearance of a robot affects the impact of its
behaviour on perceived trustworthiness and empathy,” Paladyn,
vol. 7, no. 1, pp. 55–66, 2016.
W. a Bainbridge, J. Hart, E. S. Kim, and B. Scassellati, “The
Effect of Presence on Human-Robot Interaction,” Proc. 17th
IEEE Int. Symp. Robot Hum. Interact. Commun., pp. 701–706,
2008.
K. M. Tsui, M. Desai, H. A. Yanco, and L. Ma, “Considering the
Bystander ’ s Perspective for Indirect Human-Robot Interaction,”
Proc. 5th ACM/IEEE Int. Conf. Hum. Robot Interact., pp. 129–
130, 2010.
2377-3766 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.