Movement acts in breakdown situations: How a robot's recovery procedure affects participants' opinions

2.1 Communication through movements

Speech act theory posits that humans are attuned to a speaker’s intended meaning (i.e., the content of the words and sentences themselves) and to the speaker’s utterances (i.e., the acts of speaking or not speaking). The utterance itself can contain “requests, warnings, invitations, promises, apologies, predictions, and the like” [9, p. 1]. The theory draws parallels to Watzlawick et al.’s [10] first axiom of communication that states “[...]no matter how one might try, one cannot not communicate. Activity or inactivity, words or silence all have message value” [10, p. 30]. That is, it is impossible to not communicate and there is no such thing as a non-behavior. Expanding this to include movement, humans, as social beings, are sensitive to both the implicit and explicit dimensions of movement as well. They actively look for and interpret signals of social behavior.

While all explicit communication signals transport information with a defined and intended meaning from the sender to the receiver on purpose, implicit communication requires interpretation of the information on the receiver’s end [11]. This implicit communication can be misinterpreted as other information is often inadvertently conveyed that may or may not be incidental. This information could include the sender’s emotional state, inner motivation, or intention behind an utterance or an action [7], and can be interpreted by the receiver consciously and unconsciously. That is, information can be sent and received without an intended message, and the unintended message can lead to misunderstandings. For example, some movements are intended to explicitly signal a message, like waving to a friend. Upon receiving such a signal, the receiver might interpret the intended message, while at the same time be sensitive to all layers of social information implicit in the act of waving and the context in which it occurs [12]. Yet, many movements and behaviors are often merely incidental. For example, a friend moving and extending arms trying to find the proper angle for stretching gets incorrectly interpreted by you that your friend is waving hello to you.

Moreover, movement itself can generate meaning for humans even if it is not exhibited by a living being. It is now generally recognized that most people will assume intentions of objects and figures that move in a certain way, even though they are aware that the objects and figures are not actually alive. The phenomenon, usually referred to as anthropomorphizing, was demonstrated in a study where humans observed the movements of geometrical shapes and the observers assigned the shapes agency, motive, and personality [13]. Recently, the phenomenon was categorized as a type of experienced sociality [14]. A related but slightly different kind of experienced sociality is sociomorphing. It occurs when a person interacts with a non-human agent and attributes to the agent social capabilities although it might not necessarily have human-like properties [14].

How do we examine these phenomena? One solution is to use semiotics, the study of signs and their usage. The most common understanding of signs is a dyadic relationship between the signifier and the signified: A sign represents its object in some respect. Semiotics is often associated with text and media analysis, but signs do not necessarily need to be linguistic symbols. Furthermore, the study of signs is not exclusively looking for symbolism and hidden meaning in the different forms of storytelling in text and media. In Pierce’s pragmatic tradition of semiotics [15], a sign is not a dyadic relationship; instead a sign is a triadic relationship between the signifier, the object signified, and an interpreter (or “translator”) of what is represented. The study of signs in the pragmatic tradition of semiotics is thus concerned with the study of how meaning is generated in this triadic relation. So, according to the pragmatic tradition of semiotics, communication can be unconscious and pre-reflexive, and forms of unconscious communication and sign processing exist beyond human language [16]. Thus, anthropomorphizing and sociomorphing are two examples of pre-linguistic meaning-making phenomenon occurring in everyday experiences.

Because of the asymmetrical social capabilities of humans and robots [17], the first axiom of communication might not translate perfectly to HRI; robots move and behave in human social spaces, but cannot truly be considered to have feelings, moods, purpose, etc. The semiotic perspective, however, is sensitive to any layer of meaning implicit within any and every movement and enables us to analyze robot movements and non-movements, for example in a breakdown situation, as meaningful, even if no message was intended to be communicated to a user.

2.2 Interpreting robot behaviors socially

A robot’s core functionality is often enhanced using social features to make the interaction more robust [18]. That is, by using shapes that can be socially interpreted or by actively communicating the robot’s current state, it is easier for people to interpret the robot’s function and behavior [19]. Knepper et al. [20] argued that actions performed in collaboration between humans and robots will be interpreted as functional and communicative. As with other humans and inanimate objects, humans interpret a robot’s signals and cues even when these signals and cues might not have an intended or well-designed social meaning. That is, the robot’s blinking lights, noises from motors, or body movements sometimes have an unintended effect on a robot’s social perception [8]. For example, even though robots may deliberately make sounds intended to communicate with people (intentional sounds), the noise produced by actuation servos for robot functionality (consequential sounds) also shaped people’s interaction with the robot [21]. Because consequential movements and noises are inevitable to get the robot to move, designers and developers were encouraged to consider what might be implicitly communicated to the user through these modalities, especially considering that robots do not need to be anthropomorphic to be sociable [22]. For example, the Fetch robot (Figure 1) uses its pan-tilt camera in its head to support its navigation algorithm. In our experiment (Section 4), the robot’s movements of this part could be misinterpreted as head movements bearing social gaze.

Figure 1

The Fetch robot at Robot House, its arm configuration, and the basket used for the experiment.

Modeling and exhibiting social signals appropriately can aid the robot in communicating its current state [19] and guide users through an interaction situation [23]. Several studies have found that the intentional use of explicit and implicit cues, such as verbal, vocal, gaze, gestures, and proximity, can influence people’s opinion of the robot [24]. Some examples of embodied cues influencing people’s opinion include using motion that communicate the robot’s collision avoidance strategy instead of its destination [25] or expressing the internal state of the robot by timing the robot’s movements [26]. Techniques from animation, such as the 12 animation principles [27], have also been used to communicate a robot’s intent to people watching or working with a robot [28].

Purely movement-based interactions can also be successfully implemented. The creators of a mechanical ottoman made it move in such a way as if to ask if the person in the room was willing or available to interact with it [22]. The study illustrated that the designers were aware that there is an explicit and an implicit dimension to the ottoman’s movement. Another study had a robot move its arm using what the researchers characterized as legible motion. The human collaborator could better infer the robot’s goal and resulted in better collaboration on a shared task [29]. Cooperation between humans and robots also improves when developers carefully consider how to use a robot’s movement for expressing its purpose, intent, state, mood, personality, attention, etc. [30].

There are also examples of what can happen when robot motion does not take into account how a robot may appear socially, even when it is not regarded as a social robot. A mismatch between the expectation and reality may, for example, lead to a loss of trust [2]. In one instance, a military robot was deactivated after it made unanticipated movements, and people distrusted it [31]. Another example is in a study where people showed tendencies toward anxiety and discomfort when they were uncertain how a robot arm would move as they worked together in proximity on a task [32]. In one study, people viewed a robot in virtual reality and on video sorting balls according to color. Participants watching the video trusted the robot when it moved fluidly, but less when it trembled doing its task. This finding was not confirmed when the robot and person cooperated on the same task [33]. This suggests a robot’s motion may be more noticeable when the person is only watching the robot instead of working directly with the robot.

In summary, all robots’ actions explicitly and implicitly communicate information even when their actions are not intended to communicate anything. Thus, it might be interesting to examine an unexpected breakdown in an experiment and investigate how people interpret and understand the robot’s unintended communication in such a situation. In our study, we apply a semiotic perspective to the perception and interpretation of robot motion as we have a special interest in the sociability of robots. We are interested in examining how a person might see the movement of a robot as a sign of “something.”

3.1 Effects on user perception

A number of studies investigate how a planned breakdown alters people’s perception of a robot and hence how such situations influence the robot’s acceptability and usefulness. In general, it appears that different contexts lead to different implications for a robot’s breakdown or errant behavior. For example, in one study where children were to engage with a robot, a robot that displayed unexpected behavior elicited more engagement from the children than one that behaved as expected [34]. In a different study, a human and robot worked together on memory and sequence completion tasks. When the robot made mistakes, it triggered a positive attitude for the human, but lowered human performance [35]. Yet another study found that participants preferred a robot that made mistakes in social norms and made small technical errors in an interview and instruction-giving process than one that performed flawlessly, but the study found no differences in the robot’s perceived intelligence or anthropomorphism [36]. In contrast, we suspect that participants in our original experiment might have been frustrated or irritated by the breakdown instead.

Some studies have examined specifically how breakdowns affect human trust in robots. A meta-analysis of factors influencing trust in HRI found that the robot’s task performance had a large impact on people’s trust [37]. In another study, researchers looked at how willing people were to follow odd commands, such as watering a plant with orange juice, from a robot that was acting faulty [38]. Although the robot’s behavior affected participants’ opinion of the robot’s trustworthiness and the participants had different opinions about the odd requests, many of the participants honored the requests. The researchers speculated that this could be due to some participants feeling they were in an experiment and actions therefore had low stakes. Similarly, we are interested in examining how erroneous behavior that cannot easily be interpreted might have affected the users and their perception of the robot.

Another study provided different ways that a robot could handle a breakdown while playing a cooperative game with someone and looked at people’s trust in the robot afterward [39]. For some participants, the robot would freeze while speaking in mid-sentence during the game. It would then either start from the beginning or pick up from where it left off. It could then provide a justification for why it froze or offer no explanation. The robot’s freeze had a negative effect on the participants’ perceived trust of the robot, but restarting the interaction had a more negative impact on the perceived trust than if the robot continued. Robots that continued and provided a justification for freezing further reduced the negative perceived impact of trust. Similarly, in our study, we are interested in examining if a robot’s freezing and recovery behaviors might have caused negative effects on the users’ perception.

3.2 Repair and mitigation strategies

Some studies have evaluated different mitigation techniques to salvage an interaction despite the occurrence of breakdowns. One experiment investigated whether some robot action can repair the situation after a planned breakdown [40]. In the experiment, participants observed a scenario between a robot and person. The observers then rated the robot and the service it provided. The robot’s breakdown had a negative influence on how observers rated their satisfaction with a robot and the service, but different mitigation techniques (no mitigation, apologizing, or offering compensation) could change the observer’s opinion of the robot’s service or the interaction. There was also a correlation between an observer’s orientation to service (more relational versus more utilitarian) and how well the mitigation performed. As long as a mitigation was provided, observers rated the robots as more human-like regardless of the robot’s form.

A different approach is the strategy of calibrated trust where the person’s expectations are tuned to the robot’s shortcomings or potential malfunctions [2]. For example, participants in the study above by Lee et al. rated the task more difficult for the robot if the robot warned early that it might not complete the task correctly [40]. This article examines both aspects, i.e., it provides help with planning and adjusting the robot’s behaviors to the participants’ expectations, and it provides tools to design fallback strategies for unexpected cases.

3.3 Unexpected breakdowns

Most breakdowns do not happen according to plan and study data from these breakdowns are often discarded to make data analysis easier. A few researchers have argued that there is value hidden in data discarded due to robot breakdowns and other error situations [41]. For example, Barakova et al. have documented a robot’s unexpected behavior in pilot studies with children with autism and how the unexpected behavior affected the children [42]. The unexpected errors were the result of a mistake by the leader of the session, software problems, or issues with the robot. The experiences lead the researchers to document their redesign of the study and changes to the robot’s software to eliminate the issues for the final study [42]. The documented changes are useful for other researchers designing similar studies.

Our purpose here was to look at people’s opinion of the robot’s breakdown situation in an experiment that was not designed for a breakdown and where breakdowns were not present in the pilot study. We wanted to examine the participant comments and see what lessons we could learn for future experiments and robot design. We performed this examination through the lens of explicit and implicit communication.

4.1 Experiment setting, questionnaire, robot, and navigation system

The procedure was approved by the University of Hertfordshire Health, Science, Engineering and Technology Ethics Committee (Protocol Number COM/SF/UH/03491) and took place at the University of Hertfordshire’s Robot House. Robot House is a place that people can visit and experience robots and sensors in a home environment instead of a typical lab environment. Since the overarching goal of the research is to have a robot be a part of a home and that the robot’s movement should appear more friendly and ultimately lead to better trust in the robot, it seemed appropriate to run the experiment in a physical area that resembled a home environment rather than a lab.

The questionnaire for the original study included the Godspeed series [43]. We also included an additional Likert item about how well the person could predict where the robot would go, and an open question: “What do you think about how the robot handled this task?” We included the prediction item as we wondered if the different velocity profiles would affect how easy the person could predict the robot’s movement. The results from the Godspeed series were reported previously [1]. The open question gathered qualitative information and is the basis of our analysis below (Section 5).

The robot we used was a Fetch Mobile Manipulator from Fetch Robotics [44] hereafter referred to as Fetch (Figure 1). We selected Fetch since it can move at a rate of 1 m/s. This speed is slower than an average person’s walking speed of 1.4 m/s [45], but accelerating up to this speed takes enough time that it is possible to create different velocity profiles.

The linear and slow in and slow out velocity profiles were based on the algorithm described in Schulz et al. [46] and adapted to a plugin for the local navigation planner in Fetch’s navigation stack, which is the navigation stack from the Robot Operating System (ROS) [47]. Fetch’s local planner uses the trajectory roll out scheme [48]. This method of integration is similar to a set up suggested by Gielniak et al. [49] for integrating stylized motion into a velocity profile for a task. The plugin included dynamic parameters for setting the velocity profile (linear or slow in and slow out). This allowed us to change the velocity profile without restarting the robot’s navigation system. The changes only affected Fetch’s linear velocity (i.e., moving forward); the angular velocity (i.e., turning in place) was unchanged from the original plugin and thus always used a linear velocity profile.

We considered ignoring the environment and simply issuing pre-recorded velocity commands to Fetch. This technique would have resulted in smoother velocity curves, but we were concerned that small inaccuracies would occur while turning, starting, and stopping would lead to large inaccuracies as Fetch moved through the house. Fetch’s navigation stack had already been extensively tested for moving the robot around and avoiding obstacles. After investigating both, we found that Fetch’s navigation stack with our developed plugin worked better than any solution we could develop from scratch in the time given for the experiment.

Fetch moved between several preassigned destinations in the house: (Positions 1, 2, and 3, respectively, on Figure 2). Each spot had two poses, one for facing the person and one for facing away from the person toward the next location. The two poses per location were done to keep the performance of Fetch’s navigation similar across conditions. Position 1 had a slightly different location for its poses to make it easier to remove and add items to the basket without the participant noticing.

Figure 2

Floor plan and position of people for the experiment. The robot would move between the numbered positions, starting at Position 1, using either a linear or slow in and slow out velocity profile.

4.2 Experimental procedure and data collection

Participants that had consented to being part of the experiment entered Robot House and filled out demographic information of age, gender, and if the participant had any experience with robots.

After the participant filled out this questionnaire, we went through safety information with the participant for interacting with the robot. We explained that they would be interacting with a Fetch robot during the experiment and that two of us would be constantly monitoring the robot. Fetch was brought over and controlled with the remote control during this explanation. We told the participants that we did not expect any safety issues, but advised them not to approach the robot while it was moving and that if the emergency stop was engaged, that the robot would keep its momentum and move unexpectedly. Participants were told they could end the experiment at any time if they felt unsafe (none of the participants ended their participation). As the safety information was being explained, one of the facilitators remotely controlled the robot and moved it towards the participant so the participant could see how the robot moved and see its size. Participants could ask additional questions regarding safety at this time.

Then, the scenario was explained. The participant was visiting a friend’s house to help in cleaning up the home (the facilitator that had controlled the robot was introduced as the friend). Cups had been placed on the dining table and the coffee table near some couches. These cups needed to be returned to the kitchen. The robot would aid in the cleanup by collecting cups from the participant and taking them to the kitchen. Since we did not want to draw attention to the robot’s motion, we explained we were interested in how the robot handles the hand over of objects from the participant. We instructed the participant where to stand, what to do, and what the robot would be doing (Figure 2).

The facilitators and the participant would then take their positions. Video recording of the procedure from one camera was started for participants that consented. One facilitator would stand in the kitchen (near Position 1 in Figure 2); the participant and the facilitator helping in cleaning the house would stand near the dining table and sit on the couch, respectively (near Positions 2 and 3, respectively, in Figure 2). Fetch would be sent to Position 1 in Figure 2. The remote control was placed on the table near the sofa to indicate the robot was not being teleoperated, but the remote control was in easy reach of the facilitator if something were to go wrong.

Starting at Position 1, the procedure was the following: (1) The robot moved from Position 1 to Position 2. (2) The participant took one of the cups from the dining table and put it in the robot’s basket (Figure 1). (3) The robot moved from Position 2 to Position 3. (4) The facilitator on the couch took a cup from the coffee table and put it in the robot’s basket. (5) The robot moved to Position 1. (6) The facilitator in the kitchen removed the cups and put a copy of the questionnaire in the basket. (7) The robot moved to Position 2. (8) The participant took the questionnaire from the robot and filled it out. (9) Once the questionnaire was complete, the participant put the questionnaire back in the robot’s basket. (10) The robot moved to Position 1. (11) Finally, the facilitator in the kitchen removed the questionnaire and prepared the robot for the next iteration.

This procedure was performed for four iterations: two times the movement was with a linear velocity profile, and two times the movement was with a slow in and slow out velocity profile. The profiles were counterbalanced to avoid ordering effects. The counterbalancing was achieved by taking the six possible combinations of two linear and two slow in and slow out velocity profiles, and randomly selecting an ordering for each participant.

After the final iteration, any video recording was stopped and participants went through an ending procedure. They filled out a questionnaire with open-ended questions concerning the overall interactions: “Do you have any general impressions about the robot during your interaction with it?” and “Do you have any questions you would like to ask us?”

We also informed participants that we were actually interested in the robot’s movement and not the handover. We used this opportunity to answer their questions and go into more technical details about how the robot sensed its environment and moved around the house. Participants were encouraged to ask any additional questions about the set up, the robot, and the experiment. Finally, we thanked participants for their time and, as noted in the informed consent form, gave them a £ 10 gift card for Amazon as compensation for time and traveling to Robot House.

Since we were concerned how the robot’s movement affected people’s perception of the robot, we removed some confounding factors to improve the internal validity of the experiment. For example, we chose to use a basket for collecting cups and the questionnaire since Fetch’s arm movement is not deterministic and would confuse participants. In addition, Fetch only moved and did not use speech recognition or sound. A pilot study revealed that it was confusing for the person to know when it was OK to put a cup in the basket. To signal to the person that Fetch was ready to receive a cup or take and return a questionnaire, it would raise its torso 10 cm when it had arrived at the pose facing that person.

Participants were asked to stand if able while giving the cup to the robot and receiving the questionnaire (all participants were able to stand). They could sit while filling out the questionnaire. The primary reason was to allow a better view of Fetch and keep the base for participants’ perceptions similar since a standing participant is taller than the robot, which might not be true with a sitting participant. A lesser, secondary reason was to make participants feel safer as the robot approached since we reasoned that participants may feel easier to move away from a robot when they are already standing versus having to get up from a chair. For an additional level of safety, having a person in the kitchen and the couch also allowed two people to watch the robot and activate an emergency stop if Fetch was going to run into something.

Fetch was partially controlled via Wizard of Oz. In line with recommendations from Riek [50], we include the additional information about our use of Wizard of Oz. The wizard, the facilitator in the kitchen and this paper’s first author, acted as the robot’s eyes and as a conductor for the robot. The wizard was in charge of noticing when the participant or the facilitator had put the cup into the basket. Then, the wizard would signal for the robot to go to the next pose. The robot would then navigate to the next position using its navigation stack. We chose to use a Wizard of Oz component to reduce variability of time for the experiment with the robot detecting the cup or questionnaire was added to the basket. Given that Fetch traveled a fixed route and the participant’s role was rigidly defined, the Wizard of Oz component could have been eliminated given enough time. The wizard also noted down observations for each iteration.

For additional data, we collected the robot’s odometry information and the time from when a request to move was made to move to the next location until the time that the robot arrived at the location and raised its torso.

5.1 Robot’s recovery procedure

From the Godspeed questionnaire, the participants’ responses were not different enough between the linear or slow in and slow out velocity curve [1]. When we were examining the qualitative, free-text comments, we noticed how some comments expressed feelings of discomfort, curiosity, or confusion from interacting with Fetch, especially during its recovery procedure. The unplanned phenomenon became part of the interaction in a substantial amount of the trials, happening 68 times in total or around 9% of the time when the robot moved. A total of 31 of the 38 participants experienced at least one of these issues.

The recovery procedure occurred when Fetch encountered problems in calculating its path for moving across the room. The recovery procedure caused the robot to move differently than its intended behavior and was not planned for by the facilitators. As part of the navigation stack, the recovery procedure was likely implemented to get Fetch to move again without considering what an observer would see. During the pilot and testing, the breakdown situation did not occur. Once discovered, the functionality could have been disabled, but it would have increased the chance the robot did nothing, which would have caused an even larger interruption during the experiment.

Our interest is in the case of the unexpected breakdown. Furthermore, the irregularity of its occurrence makes it unfit for quantitative analysis. Instead, we analyze the comments qualitatively and cross-reference them with our notations of when Fetch had issues, and how that issue manifested in the robot’s movement and behavior.

Normally, the robot navigated competently between the positions in Figure 2. When Fetch received an instruction to proceed to the next navigation point, its head would look up and down as it calculated the path and speed to travel. The head movement would pan-tilt the depth camera inside Fetch’s head and support its navigation algorithm. When a path had been calculated, it would straighten its head and proceed on the path.

This process would normally be completed within a second or two, and Fetch would begin to move. Sometimes, however, it encountered problems in calculating its navigation path. In these situations, it would continue trying to calculate a path and the head would continue to move up and down until one of the following things happened: (1) it succeeded in calculating the path and started on the path after the delay or (2) if the navigation software had not calculated the path after 25 s, it decided that that Fetch was “stuck.”

If Fetch was stuck, the navigation software would rotate the robot 36 0 ∘ to make the robot “unstuck” (Figure 3). After completing the rotation, Fetch would quickly find its path and proceed. The turn took around 8 s to complete. Adding the 25 s from attempting to calculate the path and around 4 s for Fetch to turn after receiving the cup results in the participant experiencing an approximate 40 s wait during Fetch’s recovery procedure.

Figure 3

Example of the recovery procedure when Fetch traveled from Position 3 to Position 1. In (a), Fetch is at Position 3 and has received the cup. Then, it rotates to go to Position 1 (b). Once the rotation finishes, it tries to compute the route to Position 1. If this takes longer than 25 s, Fetch is considered “stuck” and rotates 36 0 ∘ to get unstuck (c) before continuing on its path.

5.2 Analytical procedure

To begin our analysis, we looked at the logs of the robot’s performance and noted when it paused or when it became stuck and initiated its recovery procedure. We then arranged the participants’ responses to the open questions on each iteration and their overall opinions into tables, arranged first by trial and later by participant. This resulted in tables that charted the journey of the robot, and we could easily follow the robot, the problems it had, and the responses from the participant. The answers from all the participants were manually annotated through emergent coding, first deductively and then inductively [51]. The themes presented in the results emerged from the coding during the inductive approach, which was performed by two researchers who then met to harmonize on the themes. Exactly how many iterations and re-reading of the comments were not counted: Each researcher read through as many times as they needed to make sense of the data.

5.3 Results: themes based on comments

The comments from participants when Fetch performed correctly were generally positive about how Fetch handled the task. While the responses made after an iteration where Fetch did not have any navigational issues are not excluded from our analysis. Going through the tables, however, it was clear that the more interesting comments were made when Fetch did not perform as expected.

We report participants’ comments from the themes of form, competence, and sentience around the delays and recovery procedures, and another unrelated event that came up often enough that we include it as well. Since there were more problems with the slow in and slow out profile iterations, there are more comments from those iterations. We did not find a difference in the nature of the comments and therefore do not differentiate between the velocity profiles below. To avoid repetition, we do not report on all comments from the legs that had problems, but instead report a representative amount.

6.1 Examining the themes

The codes that emerged during the analysis of the responses were categorized into three themes: sentience, form, and competence. These themes emerged from the coding process and were not mutually exclusive.

Sentience had the largest variety of codes. Yet, what the codes all have in common is the clear use of either directly or metaphorically describe what the robot did or did not do. Following this line of thinking, we also noticed that it was possible to further distinguish the comments (especially) belonging in the sentience category between those that directly anthropomorphize (or sociomorph) Fetch, and those where such anthropomorphism is implicit in the language used to describe it. For instance, in many of the comments coded within the sentience theme, participants explain how it “felt like,” or “was as if,” or “seemed to me like” the robot did something that gave it a life-like character. Even though Fetch was not intended to be sociable in the experiment and thus limited in its social function, this was a recurring pattern. Perhaps it was due to the context of cooperating being of a social nature. It is also possible that the Fetch’s movements in the room gave the participants an experienced sociality (such as sociomorphing or anthropomorphizing) in their interactions with Fetch.

Form had the fewest occurrences of codes related to it, and the codes that did emerge concerned only three “body parts.” Still, there were some interesting trends that appeared. During coding, we noticed a certain overlap between the comments that described Fetch’s form using anthropomorphic or zoomorphic terminology and comments that sociomorphed Fetch. For instance, small movements of the camera were perceived as social gazes or nods of a head. On the other hand, the comments about the arm were all about the lack of its function. Fetch not extending its arm when it stopped to receive the cup from the participant might be one reason for the lack of experienced sociability toward this particular part of the interaction. Perhaps putting Fetch’s arm in a sling would have helped indicate that its arm could not be extended in the interaction. On the other hand, this change could have other unintended effects on the robot’s sociability and might look inappropriate to the participants. The study of Heider and Simmel [13] shows that the exact visual appearance of a moving object is only marginally perceived by a human observer. Instead, the quality of the movement is the predominant characteristic. The experiment does, however, show that this is an area that could be examined further.

Competence had the smallest variety in codes, of which there were only two. Yet, these two codes occurred more often than the others. A large number of responses explaining how the participants felt that the robot was slow. We understood these comments as concerning the nature of the interaction in a collaborative action, which often gave rise to frustration at the robot being slow at completing the task, even if it felt safe to collaborate with it. Another reason for this high occurrence of codes regarding Fetch’s competence could be that the participants were answering an open question specifically asking how well they thought the robot handled the task.

While this is not included in our coding one interesting observation made going through the data was an apparent distinction in the responses of participants who appeared to use words with a sentience connotation without any apparent inner strife, and those who appeared less inclined to do this, but seemed to either not find, or not to bother finding other words to describe what is happening in the interaction. Fussell et al. [52] argued that it is easy for people to anthropomorphize robots in casual descriptions of robots because they use “ordinary” words. This can be related to the work of Seibt [53] who describes that varieties of “as if” (either explicit or implicit) in descriptions of human interactions with robots masks the social asymmetry of the interaction.

In general, it is difficult to discern whether the participants perceived Fetch as actually having the social and sentient abilities their words in their answers described, or if they were applied for a lack of a better way of expressing the experience. Many comments regarding Fetch’s competence were direct answer to how well the robot handled the task, and might not be the result of any sociomorphing or anthropomorphism. On the other hand, if a participant described that they felt Fetch was “checking the room” it would imply a kind of perceived competence in the robot, as checking could be characterized as knowing what to look for and getting an overview of the situation. Furthermore, a checking function like “looking” requires intent and purpose, which are relying on sentience. Further still, it would also be a kind of comment on how the participant perceived Fetch’s form, because “checking” then also requires having visual perception. Furthermore, a comment regarding Fetch “checking” could thus belong to all three categories, “looking” in two. Several of the comments were annotated with codes belonging to two or even all three themes. One method that could be used to examine this more thoroughly is the Linguistic Category Model [54], as is done by Fussell et al. [52] to examine linguistic anthropomorphism at different abstraction levels.

6.2 Movement acts

Knepper et al.’s [20] classification of intentional and consequential sounds can also apply to robot movement. A similar categorization for movements can enable us to better understand what is explicitly communicated in Fetch’s movements and what may be implicit communicated (both intended and unintended). During normal operation Fetch’s movements were primarily functional, even though Fetch did not have any movements that were designed purposefully for social interaction and giving social cues. Because Fetch did not communicate explicitly with language or sound in the experiment, the communication was purely expressed through Fetch’s movement across the room and what was explained via the facilitators. This was due to the original experiment examining different velocity profiles. The purpose was to see if the difference in the profiles communicated different information to the participants.

For example, Fetch’s journey in each iteration was functional and intentional to collect cups and return it to the kitchen, but Fetch’s rotations were functional and consequential as the movement “calculated a path” and “performed a recovery procedure,” respectively, without communicating any intended message. Still, it does not cover how meaning arises in a semiotic, triadic relationship between signifier, signified, and interpreter. That consequential movement or non-movement is present in the world for all present to observe, which can result in unintended interpretations of what that movement or non-movement meant [12].

Before the experiment, the participants were explicitly told that Fetch would be collecting cups. They were therefore aware that Fetch moved to collect cups and knew that this would be the purpose of the robot’s approaching and stopping (having the implicit meaning of “now’s the time to give the cup”). This means that even if the participants knew what the purpose was, when and how they should hand over the cup became unclear to many participants because they were expecting a social cue and hence still waited for the robot.

That the intention behind the implementation of the rotation has no explicit communication purpose, however, does not invalidate the experiences of people who interpret robot movement with a different meaning than intended – even if they are not quite sure what to make of it. Our case study has further confirmed that functional or consequential movements still communicate “something.” But as the “message” being interpreted was not intentionally sent, what this “something” ends up meaning to an observer can be difficult to predict. As this case study has demonstrated, a robot’s movement in the breakdown situation leads to different, possibly incompatible, interpretations by the participants.

Another Fetch’s consequential movement or rather non-movement was its occasional delays, where it paused longer than usual before leaving a station. These pauses also brought forth puzzled comments from the participants about what the purpose of the delay was. During our analysis, we clearly saw that these pauses, even when the participants were not quite sure what to make of it, did not go by unnoticed.

Currently, there is no framework or concept that covers the triadic relationship of different meanings that might arise during interaction with robots and that acknowledges both movement and non-movement as social signs. Therefore, we draw inspiration from the concept of speech act [55] and introduce the concept of movement act. A movement act entails the understanding that both intentional and expressive movements and intentional and consequential movements might be interpreted by an observer as communication of inner state and intention. Just as not speaking is itself an act open to interpretation by the surroundings and its inhabitants, so will also not moving be an act (as a conscious or less conscious choice) open to interpretation. For example, a pause may only be a pause, but it may also imply a sense of insecurity or confusion.

That the situation is interpreted differently based on the robot’s movements is in line with what others have already suggested: A robot, despite its limited social capabilities, is capable of communicating implicitly and explicitly using movements only. Using the concept of movement acts, we can isolate, identify, and characterize this phenomenon. We can then take each movement act and individually examine its implicit and explicit dimension. Movement acts can make sense of what a robot’s movement communicates explicitly (or lacks to communicate). Being aware of the implicit dimension allows one to systematically look for interpretations that might happen during an interaction. The notion of movement acts facilitates a behavior design process that aims for an effective and clear communication between a robot and the people who interact with it.

6.3 Applying movement acts to robots

If we apply the movement act concept to the original experiment, it can help explain some issues or provide suggestions for a better movement design.

First, although a robot’s movement can communicate information, the original experiment did not find any significant difference in the perception of the slow in and slow out and regular velocity curves. So, was the slow in and slow out motion worth the effort? The previous article [1] outlined multiple reasons why that might have been the case. Yet given the participant’s comments in the case study, it would appear that the robot’s motion in a breakdown situation drew attention away from any other type of motion. That is, the movement acts in the rotation and delay captured more attention than the movement act in the velocity profile. Although the slow in and slow out movement act was meant to be implicit in its communication, it could have been too subtle. Perhaps a slow in and slow out velocity profile cannot be used alone and may need to be used in concert with one or more animation principles – for example, exaggeration or anticipation – to capture sufficient attention.

The movement act of Fetch tilting its head up and down as it calculated its path gave depth information to the navigation stack and provided some context to participants watching that something was happening, but the participants’ comments indicated that this movement act was ambiguous and communicated different information. The act must communicate more explicitly that Fetch needed more time. One way to do this could be additional movements such as slowing its head movement or performing a quick “double take” when the calculation started to take more time. Another possibility could be to combine the movement with other cues such as sound and light.

Likewise, Fetch’s rotation movement act focused on the functional purpose for the navigation stack (re-calibrating its obstacles and position). On the one hand, we could have put more effort to avoid the situation entirely in the original experiment. On the other hand, this movement act could be modified to communicate its purpose to observers as an explicit, communicative motion. For example, perhaps Fetch might quickly raise and lower its torso before rotating, or it could just lower its head completely in a sign of defeat before rotating. As it is unlikely to avoid all breakdown situations, we recommend paying attention to the implicit dimension of all movement acts, including functional ones, to may make it easier to communicate a robot’s current state.

This is where knowledge from other studies may be helpful. A model for mitigating breakdowns in HRI has been proposed based on a literature review [56]. The model suggested using visual indicators (LEDs, icons, emojis), secondary screens, and audio [56], but motion is not mentioned. The responses from the participants in our case study showed that motion communicates information as well. So, incorporating motion with these other modalities could strengthen communication for mitigating a breakdown. But, as Aéraïz-Bekkis et al. already reported, the discomfort that some participants expressed can be related to uncertainty about the robot’s movements and its intentions [32]. Hence, if a robot’s unexpected movement behaviors are causing discomfort (or fear), trust in the robot might be eroded, as the robot’s performance is a large factor affecting trust [37]. This is congruent with the observation by Ogreten et al. [31] that soldiers never used a specific kind of robot in the field due to this robot’s unexpected movements. Isolating the movement into movement acts can help identify where and why the uncertainty is happening and provide places where additional or different motion may communicate more explicitly and remove the uncertainty.

Returning to the humans’ expectations of a robot based on the robot’s appearance [57], designing a movement act to express the navigation issue may help calibrate people’s expectation that the robot may not be an expert navigator yet. Similarly, using movement acts to isolate the motion in a breakdown situation could lead to more legible motion for people to understand what is happening in the situation [58]. Using movement acts may also show that there is a need to add additional functionality to the robot (e.g., adding sound, lights, or extra moving parts) to aid in legibility or provide multiple modalities for communication.

There are many areas designers can turn to for inspiration to explicitly or implicitly communicate information through motion. Some sources of inspiration can be from animals or art. For example, Koay et al. [59] looked at how hearing dogs use movement to communicate with their deaf owners and transferred it to a humanoid robot. Participants were able to understand the robot’s movement as communication and act upon them to solve a problem even when they had not been told the nature of the study. The original experiment drew inspiration from animation [28], but other areas such as puppetry [60] or dance [61] also offer inspiration. All these fields have dealt with issues of designing motion that can be understood by others, provide some expression, and set expectations by the people viewing the motion.

6.4 Making unexpected breakdowns expected

There are multiple ways to reflect on the case study. The case study might be seen as a cautionary tale. Researchers can try to control as much of the variables in an experiment, but issues still can show up. In this case, the robot may have built-in behavior that will take over if things don’t work. It is good that a built-in behavior can resolve a problem, but one should consider how the people interacting with a robot will interpret the behavior. One might conclude that researchers should prioritize making robot robust, making the experiment meticulously planned, or controlling the entire experience by filming it and having participants watch it.

We would instead present this as a call to embrace the unexpected and design the breakdown situation into a study. Using the metaphor from Hoffman and Ju’s designing with movement in mind [30], we would encourage researchers to design their experiments with the possibility of “robot breakdowns in mind.” This does not absolve researchers and engineers from designing robust robots and well-designed experiments, but to accept that a breakdown may occur and have a plan to get data out of those situations. Moreover, we want to encourage authors to extensively report unexpected breakdowns to gain a deeper understanding of HRI.

Since these breakdowns may not happen for every encounter in a study, researchers will likely need to employ qualitative methods to explore the breakdown. One way of doing this could be to have a qualitative, semi-structured interview with the participants if a breakdown situation and see how they interpreted the breakdown or even if they noticed any sort of breakdown. This may mean that even if the participants’ quantitative data may not be useful due to a breakdown, they can still provide qualitative information about their experiences and interpretations of the breakdown situation.

If experimenters desire more control and consistent experience, they could intentionally insert or trigger a breakdown situation during an experiment, even if the experiment does not primarily look at breakdowns. Since the breakdown is known in these cases, experimenters could design better ways of gathering data from the participants about the breakdown and how the participants interpret it. An inspiration for this approach comes from a long-term case study where participants developed their mental models of a robot shoe rack over several encounters with the robot changing behaviors every 2 weeks (with some unintentional errors from the Wizard) [62].

For example, if we had designed our experiment from Section 4 with breakdowns in mind, we could have used the opportunity to go deeper on things participants wrote and explored their opinions. It might have been possible to examine what participants meant when they said the robot was “waiting” or was “confused”? What actions from the robot made them think this? What made them feel uncomfortable and why? Alternatively, if the person felt that everything worked fine, why do they think that? Yet another approach could have been explored in the built-in navigation recovery. We could have found a reliable way to trigger the error to make the breakdown part of the experiment.

Answers to the qualitative questions may not be directly connected to the quantitative question being investigated in a study (our case study was not linked to the earlier experiment). The data collected from the interview questions, however, can provide a better understanding in future robot design and interaction. This could lead to insight into how to make breakdown situations easier to understand, or make people feel safer and more comfortable when such a situation occurs.

Of course, quantitative scales may also be useful for getting data about breakdowns. In the original experiment, there may have been an issue that the Godspeed Series might not have been sensitive enough to capture the change in perception during the breakdown situations. A different scale, such as the robot social attributes scale (RoSAS) [63], might have picked up participants’ different perceptions of the robot that occurred during the breakdown situation.

Designing with breakdowns in mind could be formalized so that it can be part of any HRI experiment. During an experiment’s design phase, experimenters could dedicate time to imagining possible breakdowns or other things that could go wrong. Additional sources for inspiration could include breakdowns that occurred in other pilot studies or experiments. From this work, the list of breakdowns would provide a starting point to determine what breakdowns could be prevented. For the breakdowns that are not prevented – either because they are unpreventable or they can be triggered in a controlled way – the experimenters could then plan qualitative or quantitative measures to record participants’ reactions. This results in a set of breakdowns that the experimenters can prevent; a set of breakdowns that are not preventable, but expected; and a set of breakdowns the experimenters could choose to trigger. These sets will never be complete, but the steps in creating them provide preparation to handle the unexpected breakdowns not in the sets as well. Additionally, following these steps will provide a better simulation of real world conditions and could give participants a feeling that the experiment is realistic and not contrived, regardless of whether the experiment takes place in the field or in a lab.

Breakdowns happen in many situations, inside and outside of HRI. It benefits all researchers to gather data from breakdowns to help improve future experiments and to understand how HRI can help improve a breakdown situation. Being willing to collect data from random, but expected, breakdowns in an experiment are also compatible with calls for bold HRI research [64] and to try research that goes beyond experimental psychology [65]. We can expect that as robots spend more time in less well-controlled environments, it will be necessary to also understand the extreme cases when interactions do not go as planned and researchers armed with methods to examine this area will find rich data that will improve future breakdown situations and HRI.

6.5 Limitations

We mentioned in Section 4 that we chose a within-subject design. This decision could have affected the quantitative results, for example there could be a learning effect throughout iterations. Counter-balancing can help mitigate this effect, but it is difficult to say if it had an effect here. We also mentioned that a choice of within-subjects is less important for the case study as we are interested in participants’ opinions during the breakdown situation and not the answers to the Godspeed questionnaire.

Regardless, even though all participants had a different experience with the breakdown situation, there still may be some learning effect for some participants who witnessed the breakdown situation more than once. This is where using the qualitative data in the case study is useful as the goal here is not to generalize, but to examine a phenomenon and learn from it to create better future interactions.

Having the kind of data set we had, our analysis could have benefited from using a framework such as the Linguistic Category Model [54], and would have strengthened this study. It would allow us to conduct an analysis in which the descriptive action verbs used by participants to describe the robot were examined.

One could also argue that the participants from the studies about breakdown situations presented earlier also were not aware of the planned malfunctions. Those studies are looking at results that they can generalize. Our goal here was to show that even when things in an experiment are unexpected for the researchers and the participants, there are still possibilities to get data out the situation that may be useful. Here we were limited to one qualitative question that did not specifically consider the breakdown situation, but the themes from the comments led us to develop a concept for better analyzing motion and understanding how the motion communicates information to humans.

Breakdowns are often an opportunity to return to the study design. If breakdowns have a fair enough chance of happening (e.g., one could argue 9% is fairly often in our case), it might be a good idea to spend time incorporating the breakdown into the study.