Tracking Anthropomorphizing Behavior in Human-Robot Interaction

Questionnaires have been used ubiquitously in the study of anthropomorphism, for instance, to investigate the effects of robots’ appearance and design [e.g. Hegel 2012; Philipps et al. 2017; Reeves & Hancock 2020]. Questionnaires allow the researcher to ask explicitly or implicitly about the effects of certain design decisions regarding robots’ appearance or the implementation of social behaviors, such as gaze following, presence or absence of gestures, body postures and the like [e.g. Ham et al. 2012]. The questions asked may concern anthropomorphism directly by addressing topics like the attribution of mind, intention and social characteristics; for instance, Sims et al. [2005] had participants rate images that varied according to robot characteristics (movement method, body shape, presence of limbs, etc.) in a questionnaire addressing the degree of perceived aggression, intelligence, and animacy. Another study using questionnaire data is Rosenthal-von der Pütten & Kramer [2014], in which questionnaire responses to 40 robot pictures were elicited and clustered. The authors found six clusters emerged, depending on how threatening, submissive, likable and unfamiliar the robots were perceived. More recently, Mieczkowski et al. [2019] asked people to rate 342 images of robots concerning warmth and competence, and they elicited behavioral tendencies, such as participants’ willingness to help or harm the robot. Several standardized questionnaires have been developed to investigate anthropomorphism; very common is the Godspeed questionnaire developed by Bartneck et al. [2009]. More recent proposals are the Robotic Social Attributes Scale [Carpinella et al. 2017], Ruijten et al.’s [2019] survey that distinguishes between different types of attributions, and Damgaard et al.’s [2020] questionnaire, which distinguishes between anthropomorphing and sociomorphing. Thus, the first use of questionnaires is to understand what features people attribute to robots (and to what extent), based on design properties of the robot or based on the ways in which it is introduced or presented.

A related use of questionnaires is to test for social categorization. For example, Eyssel & Kuchenbrandt [2012] presented a robot to participants as either an in-group or an out-group member and asked them to rate the robot regarding anthropomorphism (mind attribution, warmth), psychological closeness, contact intentions, and design. Participants rated the in-group robot more favorably and anthropomorphized it more strongly.

Questionnaires are also used to measure the effects of certain robot behaviors or properties of behaviors. For instance, Nass and Brave [2005] investigated whether people respond to characteristics of robot voices in similar ways as to human voices.

Another use of questionnaires is to study effects of robot framing on the extent to which people anthropomorphize robots; Paepke & Takayama [2009], for instance, presented participants with a Pleo robot, introducing it either as highly sophisticated or as rather basic. They found that people's experience of the interaction as measured by a post-experimental questionnaire was influenced by the way the robot was framed. A slightly different use of questionnaires is not to ask people for evaluations of the robot, but instead to have them assess the robot's suspected knowledge. Lee et al. [2005], for instance, presented participants with a robot that was introduced to them as either built in China or in the US. Participants furthermore witnessed a brief interaction between the experimenter and the robot in either Mandarin or English, before they had to rate the likelihood that the robot knew landmarks from China, the US and other regions of the world in a questionnaire. The results showed that participants assumed the robot to know more landmarks of the country in which it was built, and thus that participants overused social categories, like the assumption of common ground [Clark 1996].

Besides asking about trait attributions, attributions of group membership and cultural knowledge to robots based on their design, description or behavior, questionnaires may also elicit other kinds of attributions, such as moral capability. For instance, they may ask about the extent to which participants hold robots morally accountable for their actions and the consequences of these actions [e.g. Arnold & Scheutz 2017], which then provides indirect evidence for anthropomorphism.

If questionnaires are used to study anthropomorphic design, they often make use of images or of videos of robots [e.g. Philipps et al. 2017; Rosenthal-von der Pütten & Kramer 2014; Hegel 2012]. If questionnaires are used in human-robot interactions, they are usually administered post hoc, due to the fact that filling them out constitutes an activity by itself. Sometimes questionnaires are administered between different phases of an experiment [e.g. Ham et al. 2012], yet they do not capture people's responses in the here and now of the interaction.

Another methodology for studying anthropomorphism is to elicit behavioral effects of robot design, robot behavior or robot framing. In comparison with questionnaires, studies targeting behavioral effects have the advantage of studying anthropomorphism in the moment, i.e. during the interactions in which they arise. A considerable body of work rests on previous findings on the effects of social cues in human interaction, which can be replicated in human-robot interaction such that when produced by a robot, social cues are similarly responded to as when such cues are produced by humans. Nass [2010, page 12] describes the method he and his collaborators have been using [e.g. Reeves & Nass 1996; Nass & Moon 2000; Nass & Brave 2005; among many others] as follows:

“I would find any conclusion by a social science researcher and change the sentence ‘People will do X when interacting with other people’ to ‘People will do X when interacting with a computer.’ I constantly challenged myself to uncover ever more unlikely social rules that applied to technology in defiance of all common sense.”

Examples of observable human behaviors that may serve as indicators for the respective degree of anthropomorphism include their preferences for spatial distance towards a robot. For instance, Ham et al. [2012] measured how closely participants approached the robot to read numbers on a screen mounted on the robot depending on the ostensible approachability of the robot's body posture and depending on their cognitive load. Other studies investigate the effects of anthropomorphic design or behavior on people's willingness to lie for the robot [Kahn et al. 2015], on the persuasiveness of a robot [Andrist et al. 2013; 2015], or on compliance with robot requests, such as not to turn it off when the robot is afraid of losing its memory [Seo et al. 2015].

Behavioral studies are generally designed to elicit one measurable response, such as the distance to the robot chosen, secrets kept or revealed, or whether the robot is turned off or not. This single behavior is then taken as an indicator of, for instance, mindless transfer [Nass & Moon 2000]. Sometimes, not only the single behavior itself but also factors that possibly influence the respective response behavior are analyzed in order to identify the conditions under which anthropomorphism occurs, for example, by varying participants’ cognitive load [Ham et al. 2012] or by analyzing user characteristics [e.g. Lee 2010].

An exception from the more narrow focus on one type of response are studies that analyze participants’ verbal behavior in response to social clues from the robot. One variant of this method is to let participants witness human-robot interactions and have them summarize what they have observed in their own words. An example of this method is the study by Fussell et al. [2008, page 145] who gave participants scripts from healthcare interviews that were allegedly carried out either by human or by robot interviewers; the manipulation was supported by presenting participants with images of either a person or a robot. Results show that participants use similar vocabulary to describe the robotic and the human interviewers’ behavior, which in turn is taken as an indicator for anthropomorphism.

Another variant of this method is to have people interact with a robot via speech, and to analyze the degree to which their speech is anthropomorphizing [cf. Fischer et al. 2012]. For instance, Fischer [2006; 2011] finds people to differ in their initial responses to the robot's greeting, such that their behavior shows more or less anthropomorphism, which correlates with their behavior later in the interactions. Using this approach, Lee et al. [2010] analyzed people's verbal behavior for indicators of anthropomorphism depending on their initial greeting, and they observed specific behavioral patterns, for instance, concerning politeness, grounding, attention to relevancy and negative behaviors. This method relied on the fact that we always design our utterances in such a way that our intended recipients can understand it; therefore, utterances show traces of who they have been intended for, i.e. they allow ‘reverse recipient design’ [Fischer et al. 2012; Fischer 2016]. Thus, the method also provides indicators for how people understand the robot and the extent to which they anthropomorphize it.

Both types of studies of verbal behavior focus on participants’ overall use of anthropomorphizing language; that is, they analyze the quantitative distribution of features with respect to the total number of words spoken. Consequently, while they rely on the investigation of a broad range of linguistic behaviors, their focus is not on the dynamics of anthropomorphism over the course of the interaction, nor on its development over time, but on the pervasiveness of anthropomorphism in participants’ verbal behavior.

Participants' (mostly) verbal behaviors also form the basis of ethnographic analyses of anthropomorphism [cf. Chun & Knight 2020]. Specifically, participants' verbal behavior observed and documented in field notes or elicited in interviews can be analyzed for traces of anthropomorphism; for instance, Forlizzi [2007] reports instances of social attributions to a Roomba robot. Chun & Knight [2020] additionally asked the participants in their study, employees at a robot manufacturing company, directly for reports or stories about “interesting interactions” with the robots, as well as about the robots' gender and personality, in addition to a survey in which social attributions were elicited in open and Likert-scale questions. These analyses target stakeholders' “holistic understandings of integrated technologies” [Chun & Knight 2020, p. 16:5], i.e. how particular social groups of people, such as families or the different teams in the manufacturing company, make sense of robots depending on their cultural situatedness, and thus aim at understanding the general concepts and practices of groups, rather than at the dynamic unfolding of anthropomorphizing behavior in interaction.

Another type of behavioral study measures how quickly people answer questions about social attributions to robots in comparison with attributions to people [e.g. Fussell et al. 2008] or how quickly they respond when primed with images of either humans or robots (Złotowski et al. [2018]). A variant of this type of investigations is to study the conditions that slow reaction times down, in particular, interference effects, such as the ‘Social Simon Effect.’ This phenomenon describes the motor interference from observing another person carrying out a slightly different behavior. That is, if a person tries to draw a straight line or hit a button as quickly as possible, she may be more inaccurate or slower when watching another person carrying out a slightly different task [cf. Förster et al. 2020]. Stenzel et al. [2012] investigated to what extent robots are perceived as similar enough to cause such an interference. In these experiments, participants were seated next to a quite mechanically looking, humanoid robot, where they had to hit the left key when a square appeared on the screen, whereas the robot hit the right key when a diamond appeared. Participants were told either that the robot was active and intelligent and responding autonomously, or that it was completely controlled by the computer, and thus mechanically executing motor commands. The results showed that only when participants believed that the robot was active and intentional, their own behavior was influenced by the robot's behavior. In contrast, if they believed the robot to be an unintelligent machine, no interference was observed.

Similarly, Klapper et al. [2014] investigated the degree to which anthropomorphic design and beliefs about the robot interferes with participants’ own finger movements. They found that human form and the belief to be observing a human hand together created inhibition effects, but not if only one of them is present. The two reaction time studies thus suggested that while automatic responses to anthropomorphic design can be observed, these responses are mediated by participants’ mental models.

To sum up, behavioral measures provide indirect evidence for the extent to which people anthropomorphize a robotic interaction partner. Usually the baseline is the human communication partner; for instance, if people's reaction times, degrees of interference, ways of speaking or responses to a particular cue (like the request to lie for the robot) are similar to those observable in interactions with humans, then these behaviors are taken as evidence for anthropomorphism. This approach produces information about anthropomorphism in the moment, but studies usually measure only a single behavior and only once. This approach thus cannot inform us about the time course of anthropomorphism in interaction dynamically because not enough data points are elicited.

Brain imaging techniques have been used to investigate the extent to which people respond to robots like to other people. For instance, Krach et al. [2008] had people play a computer game and made them believe that they were playing against a human partner, a humanoid robot, a functional robot and a laptop computer. In fact, participants interacted with the same computer program in all four cases. However, the participants not only rated their interaction partners as increasingly more fun, intelligent, competitive and sympathetic with increasing degrees of human-likeness, but their brain activations in interactions with the artificial interaction partners were also increasingly similar to those in interactions with the human partner the more human-like the communication partner was expected to be. Kuz et al. [2015] compared people's recognition of movements by two robotic agents, one mechanical, the other anthropomorphic, and found that while the actions by both agents were generally recognized, in contrast to Krach et al.’s [2008] results, they did give rise to different brain patterns. In a review paper on people's responses to embodied artificial agents in general, Wykowska et al. [2016] concluded that responses to such agents are so similar to responses to humans that they can even be used as testbeds to study human social cognition.

To sum up, while previous work has demonstrated anthropomorphism in a large variety of situations and has shed much light on the factors conditioning anthropomorphizing behavior and the attribution of human-like properties into robots, the methods used in previous work are not designed to trace anthropomorphism over the course of an interaction. To have such a method to complement the set of methods already available would, however, be very useful to address inter- and intrapersonal variation in response to robots designed as social actors, to reconcile findings about anthropomorphism with findings on anti-social behaviors towards robots, and to understand the relationship between the automatic, mindless characteristics of anthropomorphism on the one hand and its cognitive mediatedness on the other. In the following, I thus suggest a method to describe the dynamics of anthropomorphism in interaction.

In order to describe how people anthropomorphize robots over the course of an interaction, we need a model that can track whether and, if so, in which ways or to what extent, people respond to a robot in a human-like way. Such a procedure would have to rely on people's behavior, since their beliefs and attitudes are not directly available, and stopping them to fill out a questionnaire would disrupt the very interaction the analysis is setting out to explain. Similarly, we cannot rely on robots’ anthropomorphic design or behavior since it is the responses to these designs and behaviors that are at issue. Thus, any method that aims to investigate anthropomorphism in interaction has to rely on the observation of people's behaviors, and not just on single behaviors, such as whether or not a secret is kept [Kahn et al. 2015] or whether or not people agree to the robot's memory being erased [Seo et al. 2015], but potentially on all behaviors observable in interaction. That is, since anthropomorphism may express itself in mimics, gesture, polite verbal behavior and in many other ways, we would want to account for anthropomorphic responses with respect to all of these modalities, not just single actions. A method to track anthropomorphizing behavior in interaction over time thus has to rely on observing and classifying a broad range of people's behaviors towards robots in interaction.

Now, a minimalistic classification would mean to classify people's behavior as either anthropomorphizing or not, whereas a maximalist version of such a classification might classify the type or extent of anthropomorphization. One might, for instance, identify, based on each of a person's observable behaviors, what kind of trait is attributed, how uniquely human it is and correspondingly, how easy or difficult it is to attribute this trait to the artifact in question [cf. Ruijten et al. 2019]. This would however mean that specific attributions can be identified and intersubjectively classified, given that all we have is how people behave. The question is whether such a classification is feasible and how the relevant judgements can be made based on what is observable; it seems therefore safer to (a) keep the number of distinctions small, and (b) to allow an easy identification of the level oriented to.

The proposal made here is to use a model of depiction [Clark 2016] because it allows us to describe people's responses to robots without presupposing that we know what this behavior is caused by. For instance, Nass & Moon [2000] suggest that people respond to technological artifacts like robots like to other humans out of mindlessness; Nass [2004] argues that anthropomorphizing behavior is basically a mistake, based on our human evolution in which we were only faced with human social actors. In contrast, Ruijten et al. [2019], for instance, regard anthropomorphism to be a predisposition instead, whereas social-constructivist approaches may take it as an instance of social practice, so that anthropomorphizing behavior would just be ‘how things get done’ [cf. Edwards 1994; Hutchby 2001]. In order to avoid circularity, a model that describes observable behaviors should not presuppose the one or other explanation and should thus be largely theory neutral. Therefore, I analyze the robot's behavior based on Clark's three levels of depiction [Clark 2016], and the person's behavior in terms of orientation to these levels. From this perspective (Clark & Fischer under revision), the robot's design is viewed as an act of staging, in which the robot is depicted as a social actor.

We understand by depiction a very common strategy of human interaction in which one person stages a behavior for another person to evoke a temporally and spatially removed situation. This strategy is most obvious in plays and other kinds of fictional staging (in movies, video games, pretend play etc.), but also occurs frequently in conversation in general when people report on events involving other people, for instance, a person telling her friend about her tyrannical boss. Clark's model involves (a) a base scene, which consists of the concrete means by which the depiction is carried out, e.g. gesture, mimics, loud voice etc., i.e. the respective ‘trigger’, (b) the proximal scene, which comprises the scene depicted, e.g. an angry person with a loud voice and exaggerated gestures, and (c) the distal scene, for instance, the speaker's boss getting outraged about a mistake someone made. Robots are special kinds of depictions since the scene evoked is not necessarily spatially or temporally removed, but rather concerns a fictitious being (cf. Clark & Fischer under revision for more details on the full model).

Concerning human-robot interaction, the base scene is relatively straight-forward to define: it concerns the mechanical properties of a robot as a machine. Less straight-forward is the definition of the proximal scene, which can be understood as the encoded meaning of a robot's behavior. That is, the relationship between base and proximal scene can be understood as one of signaling [Peirce 1998], such that the base scene (e.g. a robot moving its actuators) may evoke a particular meaning (e.g. the robot waving) by means of indexical, iconic or symbolic relations between the movement and the interpretation. An index is a sign in which the form consists of a pointer to a particular referent; an example is the English word “I,” which indexes the speaker, but it does not ‘mean’ John, Mary, Herb or Kerstin itself. For instance, directing the robot head towards a participant's position indicates attention towards him or her. An icon is a sign where the form is connected to its meaning by means of similarity; for instance, the robot moving its gripper from left to right and back is similar to a human waving gesture, i.e. it can be taken to iconically represent waving. Finally, a symbol is a sign with an arbitrary, conventional relationship between form and interpretation, the prime example being language, where the sequence of the letters c-a-t are conventionally taken to mean ‘cat’. When a robot says “hello,” it conventionally symbolizes a greeting.

In contrast, the distal scene goes beyond the encoded meaning and concerns the pragmatic interpretation of an action. The definition of the distal scene is thus also relatively unproblematic since it concerns the completely fictional, live-like character evoked by the robot, for instance, as a companion, friend or collaborator. For example, the robot moving its hand encodes a waving gesture, but the interpretation that it sees me, recognizes me, wants to attract my attention and initiate an interaction by waving at me is an interpretation that goes beyond what is encoded. Both the proximal and the distal scene thus involve anthropomorphizing behavior; however, while the distal scene evokes the robot as a character, i.e. as a social actor, the proximal scene only assumes that the robot is engaged in meaningful action, i.e. that its behavior means something.

The two levels of depictions assumed thus differ in the underlying processes: whereas the proximal scene relies on semiotic processes, i.e. on the interpretation of signs, the distal scene is essentially fictional and relies on imagination [e.g. Walton 1993] and thus involves a certain degree of pretending on the side of the recipient [Clark 1999]. The distinction is useful for our analysis because it helps distinguish between features that are encoded in the (behavior) design of the robot and features that go beyond this; for instance, using a voice with features characteristic of a human female, for example, indexes a female speaker; that is, there is an objective, indexical relationship between certain voice characteristics and human gender. In contrast, if people associate gender stereotypes to a (human or robotic) speaker with these voice characteristics, then these stereotypes are not encoded in, but associated with the voice [e.g. Tay et al. 2014; Bryant et al. 2020; Law et al. 2020]. That is, by means of its speech characteristics, the robot is depicted as female, but an expected higher competence in a certain occupation is not part of the robot's (behavior) design, but rather “in the eye of the beholder”, i.e. in the distal scene evoked.

In our model (Clark & Fischer under revision), a robot is thus similar to Mickey Mouse in Disney Land: The man in the costume constitutes the basis for the depiction; the mouse-shaped costume evokes the depiction proper, a large mouse with eyes, ears and a mouth. The distal scene is then that Mickey Mouse, a character one is familiar with from numerous comics, is present in the here and now. Regarding Mickey Mouse, kids can orient to the man in the costume (e.g. “is it hot in there?”), to the depiction of the mouse (e.g. “you have a large head!”) or to the character (e.g. “Mickey, where is Minnie?”). To which of the three scenes they orient can thus be identified from their behavior.

Similarly, the robot has a mechanical base that enables the robot's functionality; the robot's design, functionality and presentation in context suggest that it can be interacted with by talking to it. And finally, the staging of the robot evokes a distal scene in which the robot can be understood as a human-like character. Like kids' interactions with Mickey Mouse, people's behavior in response to a robot's actions is revealing regarding to which of the three scenes they orient.

Table 1 illustrates how the depiction model applies to various different kinds of depictions [see also Clark 2016].

Thus, the depiction model assumes that depictions allow attention to three different perspectives or scenes, which exist in parallel, and all three can be in focus at different points in time. For instance, concerning the conversational depiction of the angry boss, the listener can focus on the base scene by saying ‘not so loud’, on the proximal scene by saying ‘you sound more cynical than angry’, or on the distal scene by saying ‘your boss is really not a nice person’.

Thus, what we are proposing is the anatomy of a ‘social cue’, where on the side of the robot, we understand a social cue as the staging of a design choice or behavior that is intended to signal a certain human-like behavior, which is meant to evoke a particular character or fictional being (for the explanatory power of this model, see Clark & Fischer under revision). In a second step, I employ qualitative methods to analyze to which of these three levels people's behavior is oriented (see Section 3.3 below).

For the description of robots’ behavior, we can use the depiction model to account for the discrepancy between their mechanical nature as artefacts, the behaviors designed and the intended effects. In particular, the base scene corresponds to the fact that robots are artefacts that operate mechanically or are preprogrammed or remotely controlled and describes the robot's actions on the mechanical level, such as moving its joints. Responses to the base scene are thus clearly non-anthropomorphic. The proximal scene corresponds to the recognition of this scene as a purposeful action by the robot, such as lifting its arm. Responses to this scene are anthropomorphic since they rely on the attribution of certain properties that the robot does not have, but which are signaled by the design or behavior of the robot, such as having eyes or looking at something. The distal scene finally accounts for the possible function of the respective behavior in context, which may in this case be that the robot is trying to reach for something. It concerns a fictional scene in which the robot is a character, for instance, a human-like being.

Let us consider some examples from human-robot interaction (see Table 2); for instance, Nakata et al. [1998] introduced subjects to a fish-shaped experimental robot which is sensitive to touch. In one of the three behaviors implemented, the robot lifts its ‘head’ when sensing human touch. The authors define this behavior as “a repelling tactile reaction” and label it “rebellious.” Using the depiction model, we can distinguish between the base scene, in which the robot's motors push up the upper part of the robot when the input to the touch sensor exceeds a threshold; the proximal scene, in which the robot ‘pushes back;’ and the distal scene, in which the robot's behavior is understood as expressing a rebellious intention.

As another example consider Tielmann et al.’s [2014] model of robot emotional expression in child-robot interaction, in which the robot's behavior differs concerning features like speech volume, pitch height, speech rate, head pose, eye color, arm and trunk pose. The robot is designed to match the child's measured emotional state, which is analyzed in terms of arousal, valence and level of extroversion. The authors describe the relationship as follows: “The voice of the robot will be influenced by its arousal. The higher arousal, the louder the robot will speak, the higher pitched its voice will be and the higher the speech rate. Speech volume is also influenced by extroversion, the higher the extroversion, the louder the voice.” Thus, according to our model, the base scene is constituted by a machine producing speech in a loud and high-pitched voice, the proximal scene is to signal arousal, and the distal scene is to indicate to the child that the robot is as excited about something as the child is.

As a last example, consider the implementation of different types of gestures in a narrating robot [Huang & Mutlu 2013]. The robot's gestures were carefully modeled after human gestures previously elicited in human narrations and coordinated with particular lexical material in the robot's speech stream. The authors find, for instance, that the “robot's use of iconic gestures significantly predicted males’ perceptions of the robot's competence”. Using our model, we can distinguish between the robot producing a previously demonstrated and recorded posture of its ‘hand’ at the same time as a particular word is used, which constitutes the base scene; the interpretation of the hand posture as iconic gesture, which constitutes the proximal scene, and the impression of a competent robot that can accompany its words with appropriate gestures that indicate that it understands what it is saying, which constitutes the distal scene.

By conceptualizing the robot's behavior as an act of depiction, we can analyze in the next step what level of the robot's behavior users rely on while interacting with the robot, moment by moment. We thus apply the depiction model for describing the robot's behavior, and in a second step use it to describe the ways in which people respond to the robot's actions.

For the analysis of participants’ responses to the robot in interaction, we take the micro-analytic perspective on participants’ actions developed in the framework of ethnomethodological conversation analysis [cf. Garfinkel 1972; Sacks 1992]. In this methodological framework [cf. Sacks et al. 1974], like in other collaborative models of communication [such as Clark 1996; Bangerter & Mayor 2013], it is assumed that in order for interaction to succeed, participants need to display to each other continuously how they understand each others’ contributions and what they consider the relevant context of the current action to be [cf. Clark & Schaefer 1989]. For instance, if speaker A says “sit down”, this utterance per se is highly ambiguous; it could be a suggestion, an invitation, or a command, among many other things. However, once speaker B has responded to it, for instance by saying “thank you”, the interpretation of A's utterance is specified as an invitation. In the next turn, speaker A may reject this interpretation, for instance, by saying “this was an order!”; however, if she does not challenge speaker B's interpretation, ‘invitation’ is the intersubjectively established interpretation of A's utterance [cf. Sacks et al. 1974]. Because speakers in interaction need to display to each other the interpretations of each others’ utterances in order to arrive at intersubjectively available common ground [cf. Clark & Schaefer 1989], their interpretations of each others’ behaviors are also available to the observer [see Sacks 1984].

Correspondingly, in the method proposed, we look at the ways in which the participants display how they make sense of the robot's behavior.¹ Our micro-analysis of the users’ responses to the robot's depiction of human social behavior can indicate to which scene the user attends in the sense that the human is displaying an orientation to a certain aspect of the situation; for instance, in the example above, by saying “thank you,” speaker B displays her orientation to speaker A's utterance as an invitation. While in this example, the display concerns the interpretation of the function of the utterance, people also design their behaviors specifically for their current interaction partners. That is, each utterance is so designed that the particular recipient can understand it [Levinson 2006]; who the speaker takes the recipient of the utterance to be, for example, whether he or she assumes to be talking to a machine or to a live-like character, can thus also be identified from the way the utterance is designed [i.e. by means of ‘reverse recipient design’, cf. Fischer et al. 2012]. Using this approach, we can analyze participants’ behaviors towards robots by classifying them according to (a) whether they display an orientation to the base scene by treating the robot as a mechanical tool; (b) whether they display an orientation to the proximal scene by displaying an understanding of the robot's behavior as being intentional and meaningful; or (c) whether they attend to the distal scene by displaying an understanding of the robot's behavior as human-like.

In particular, for each response to a robot's actions, we identify to which of the three scenes proposed it is a response to. For instance, a robot's greeting “Hello, how are you?”, can be responded to in several different ways [see Fischer 2011]: One possibility is to reciprocate the greeting by replying “I'm fine, thanks, how are you?”. This is a response at the distal level, where two people are having a polite conversation. A second possibility is to respond to the robot's utterance as the production of canned speech by a machine in order to indicate the beginning of the interaction. In this case, people may say “drive forward”. This utterance signals an orientation towards the base scene. Furthermore, people may acknowledge that the robot was asking a question and producing a greeting by responding “fine” or “hello”. In this case, people orient towards the proximal scene, in which the robot is taken to produce some type of social action. However, the human interlocutor does not enter into the joint pretense [cf. Clark 1999] of having a polite conversation, but acknowledges that the robot's behavior may have a social function, such as opening the interaction (‘hello’) and asking a question (‘fine’). These responses would therefore be located at the proximal level.

We can thus classify people's behavior as being oriented to the base, proximal or distal scene, where responses to the proximal and distal scene are anthropomorphizing since they involve sense-making by drawing on the human domain. While the distal scene clearly evokes a fictional, human-like being, the proximal scene also involves human-like categories, such as the attribution of human functions (e.g. the robot “sees” something), and can be (and has been) considered as anthropomorphism [e.g. Fussell et al. 2008], yet the mechanisms underlying these two scenes are most likely quite different (that is, semiotic versus inferential processes, [cf. Fischer 2016]), and thus we rather keep them distinct in our classification.

Given the multimodal nature of interaction, users’ displays of how they understand the partner's actions may potentially comprise a very broad range of signals. For instance, on the one hand, we can look at what people say, on the other, also the timely response to the partner's behavior, i.e. attending to the response time typical of human interaction, about 300msecs, [see Jefferson 2004], can indicate to the analyst that the person is orienting to the robot in similar ways as to a human communication partner [cf. Hutchby 2001]. Similarly, people may or may not gaze towards robots at times when they would also gaze towards human interaction partners [e.g. Andrist et al. 2014]. Anthropomorphizing responses may consequently become evident in all kinds of communicative behaviors.

The method proposed thus targets anthropomorphism in terms of anthropomorphizing behavior, i.e. behavior that attributes human-like qualities to a robot. It is consequently in line with work investigating behavioral responses, such as keeping a robot's secret [e.g. Kahn et al. 2015], saving a computer's face [Nass 2004] or describing a robot in intentionalistic ways [Fussell et al. 2008]. However, the method proposed does not make claims regarding people's subjective understandings of robots, i.e. the psychological basis for the observable behavior. For instance, the method proposed does not make assumptions about psychological mechanisms such as mindless transfer [Nass & Moon 2000], automatic trait attribution [Roebroek 2014], or the real belief that the robot under consideration has a set of human-like characteristics. That is, our study is neutral with respect to possible explanations for the anthropomorphizing behavior observable. Nevertheless, as we shall see (Section 6), the results of tracking people's anthropomorphizing behaviors in interaction over time will have considerable consequences for explanatory models of anthropomorphism in human-robot interaction. Bridging the gap between people's cognitive processes and their observable behavior goes unfortunately far beyond the limits of this paper (but see Clark & Fischer under revision). What the method proposed here however does allow us to do is to describe people's responses to robotic action and track anthropomorphizing behavior dynamically over the course of an interaction.

The data re-used here are spontaneous interactions gathered in the lobby of Willow Garage for another study [Fischer et al. 2014]. In these interactions, participants were left alone in the lobby to fill out an initial questionnaire. While participants were busy filling out the form, a PR2 robot, placed directly opposite the participants across the counter on which they were writing, addressed them either with speech or with a beep, and performed a series of gestures intended to be understood as gesturing for help.

The data set comprises 19 interactions (nine men, nine women, one unreported gender) of participants between 19 and 63 years old and from various occupations and educational backgrounds. They had been recruited to participate in human-robot interaction experiments but did not know at the time of the interaction that it was part of the experiment (but were debriefed later). They were on average not very acquainted with robots (3.4 on a 7-item scale), and they scored M = 82 out of 110 on other-oriented empathy (sd = 7.3) and M = 25.5 out of 40 (sd = 4.3) for helpfulness [cf. Fischer et al. 2014] in a standardized questionnaire on prosocial personality [Penner 2002].

The robot's behaviors were all prerecorded and were initiated by the experimenter remotely from another room according to a fixed script. Robot behavior was not fully automatized in order to ensure participants’ safety. The experimenter came back into the lobby when the participants had stopped interacting with the robot for at least 30 seconds.

In Table 3, we analyze the robot's behavior in the speech condition on the basis of our depiction model; the robot behaviors were implemented by the engineer using the so-called ‘puppet-interface’ [cf. Pantofaru et al. 2013]. This interface allows the researcher to program different robot movements and to store them in sequence. These actions constitute the base scene, which comprises the concrete means by which the depiction is carried out, such as gesture, head orientation or presynthesized, canned speech. This behavior is meant to evoke a proximal scene, such as the robot reaching for something or turning towards the participant. The third level is the distal scene evoked, such as the robot intending to collect a plastic cup from the shelf or turning to the participant for help. The distal scene evoked thus corresponds to the robot behavior designer's intentions (see Table 3). That is, from a social interaction perspective (i.e. from the perspective of the distal scene evoked), the robot first attempts to attract the participant's attention, then gestures towards an object on the shelf that is out of its reach, and finally looks back and forth between the object and the participant in order to get help.

Eight participants saw the robot in the speech condition, while eleven saw it in the beep condition, in which the robot's greeting was replaced by a beep series; thus, in the second condition, the robot leaves the polite welcome message away and instead simply uses a series of beeps to attract the participant's attention; in addition, it looks back and forth between the desired object and the participant one more time than in the speech condition.

The interactions were analyzed concerning the scene attended-to in response to each of the robot's behaviors. That is, we investigated whether each participant displays an orientation to the base scene, the proximal scene or the distal scene and encode each response as one of the three. In particular, we code a user's behavior as attention to the base scene (see Table 3) if there is no reaction to the robot's behavior at all (because in interactions between humans, since interactants need to signal to each other how they understand each others’ behaviors, lack of response to a partner's action is rare),² or when the participants draw attention to the material properties of the robot, for example, by pointing out that it cannot do anything, or by walking around it, reading posters behind it, etc.

In contrast, we code as attention to the proximal scene when people attend to the robot (e.g. by watching it) and thus display an understanding of the robot being in the course of carrying out an extended activity, i.e. when they observe it for longer periods of time. Another example of attention to the proximal scene is when participants assume that the robot is doing something meaningful, but do not attribute emotions or its own agenda into it, for instance, when the robot is turning to the shelf to get help while participants wave at it to make it perceive them and thus display their interpretation of the robot's movement as an attempt to scan the room.

Finally, we code as attention to the distal scene when participants indicate that they expect the robot to have its own agenda (for instance, by asking what it wants), when they suspect human wishes, such as hoping to get a glass of water, or when they respond politely and reciprocate social behavior [Nass & Moon 2000].

Below is an informal rendering of an example interaction; because of spatial constraints only one analysis is presented in detail, and to make it as random as possible, the analysis presented concerns the first interaction elicited (participant H101)³ (see Table 4). The participant's behavior can now be analyzed according to which scene it is oriented to.

The robot's first action is to move its upper part, which was intended to indicate that the robot sees the participant and thus turns its head towards him or her. The participant does not notice this action, however. While we cannot be certain that participants would have noticed such a move from a human in the same position, the robot is still treated like any other piece of furniture in the room by not being attended to (i.e. at base level).

The robot's next action is to play canned speech, namely a polite greeting. During the greeting, the participant briefly looks up and down again. After a pause of about two seconds, the participant responds to the robot's greeting by reciprocating it, thus attending to the distal scene in which two equals participate in a polite exchange. At the same time, he does not look at the robot while it is speaking (which is unlike communication among humans, since listeners usually look at speakers while they are speaking [cf., for instance, Andrist et al. 2014]), and also the long pause before the greeting would indicate trouble in human interaction [e.g. Levinson 1983] – the usual response time in conversation is about 300 to 500 milliseconds [Jefferson 2004], with some slight intercultural variation [cf. Stivers et al. 2009]. Thus, while the participant's greeting itself is oriented towards the distal scene, other interactional behaviors are rather oriented at a communication partner that does not require attention to the social norms of human conversation (and thus relate more to the base scene). To sum up, the participant's looking at the robot during the greeting is encoded as attention to the proximal scene, the long pause is understood as oriented to the base scene, while the reciprocal greeting is coded as orientation to the distal scene.

In response to the robot's next utterance, the participant does not react at all, thus withholding feedback necessary for the joint management of interaction [cf. Sacks et al. 1974] by not providing evidence of his understanding of the partner's utterance. This behavior is consequently coded as attention to the base scene.

The robot produces a series of beeps then, which the participant repeats after he finishes his questionnaire; he thus displays that he recognizes it as attention-getting, but treats it as non-social (compare this to the ways R2D2’s beeps are responded to in Star Wars). We therefore suggest this behavior to respond to the proximal scene: a robot signaling to get attention.

The robot's turning to the shelf is then interpreted as ‘looking,’ as can be seen from the waving gesture and the question “do you see me?”, i.e. it is treated as a very basic, but directed behavior; thus, the participant responds to the proximal scene. Similarly, he does not understand the robot's gesture towards the shelf as based on its own agenda; instead, he provides the robot with a reason to move the arm towards the shelf by giving an instruction (“pick a book”). This means that he does not attribute intention or volition to the robot, which corresponds to attention to the proximal scene.

However, he then offers the robot, not just the cup, but “a glass of water,” i.e. addressing a human need, hence responding to the distal scene. In the next step, the participant exaggerates the action, squeezing the cup into the robot's gripper and indicating the futility of the operation. Thus, he responds first to the proximal and then to the base scene. Finally, he walks away, indicating no social obligation to negotiate the end of the interaction [cf. Schegloff & Sacks 1973 in contrast], thus orienting towards the base scene again.

The participant's behavior is visualized in Figure 1, which illustrates how his behavior changes between the base, proximal and distal scenes (y-axis) in response to the different robot behaviors (x-axis). A line diagram was chosen to highlight the progression of anthropomorphizing behavior over time (even though there are no transitions between the three levels as the connections might suggest), because the novelty of the method proposed here is that it can document the dynamic unfolding of anthropomorphizing behavior over the course of the interaction.

As the diagram summarizes, the participant switches back and forth between attention to the distal, proximal and base scene within this one minute of interaction (59 seconds).

All 19 interactions were analyzed based on the depiction model in the same way as described above. In order to measure the interrater reliability of the coding scheme presented, four interactions were analyzed by an independent coder, a master student in an IT program on web communication design, on the basis of the text in section 3 and the example analyses in section 4, which served as his only instruction. The comparison of the analyses by the first author and the independent coder reveals 93.74% agreement and a Cohen's kappa of 0.874.

Data analysis shows that every single interaction is unique in terms of attending to the robot, following the robot's attention to the shelf, moving towards the robot, and helping.

For example, participant H111 behaved very differently from participant H101 by first ignoring the robot (base scene – the robot is a machine that needs no attention) and then by trying to make sense of the robot's actions (proximal scene – the robot's actions indicate something) (see Table 5).

Another example showing another type of response is produced by H112, who responds to the robot in social ways initially, but then largely ignores it (see Table 6):

The variability in participants’ behaviors towards the robot is reflected in Figure 2, which represents the eight participants’ responses to the robot's nine different behaviors (on the x-axis) in the speech condition. As Figure 2 illustrates, the eight participants in the speech condition employ a wide range of behaviors in unique combinations in interaction with the robot. That is, no robot behavior is responded to in the same way by all participants. It shows that most, but not all, participants respond to the distal scene evoked when the robot produces a polite greeting, and again many but not all participants attend to the distal scene after the robot's gesturing for help. However, as Figure 2 additionally illustrates, all participants also produce behaviors in between that indicate their interpretation of the robot as machine-like, and in general the time courses of their responses to the base, proximal and distal scene vary considerably. The analysis shows that no direct correspondence between robot behavior and participant responses is emerging, so that participants’ actions are not ‘triggered’ by the robot's actions in similar ways.

Comparing the behavior of the participants in the speech condition now with the participants in the ‘beep’-condition, in which the robot did not greet the participants but attracted their attention by means of a series of beeps, shows that even though much intrapersonal variation can be observed here, too, and no single robot behavior is responded to by the participants in a way that would appropriately be described as ‘triggered’, the overall distribution indicates that the initial use of speech in the speech condition has an impact on people's overall behavior. Thus, while the robot's behaviors do not trigger any specific responses, the anthropomorphic design of its behavior in Condition 1, i.e. the choice of human language, has an effect on how the robot is responded to in general.

Figure 3 shows that when the verbal greeting by the robot is replaced by a beep series, responses at the distal level occur later in the interaction, if at all. While some participants do not respond to the robot's depictions at the distal level at all, five participants do, but at different moments in the interaction. In fact, like the analysis of participants’ behavior in the speech condition, the analysis of the beep condition indicates unique patterns and frequent switches between levels by the participants.

The different distributions of responses at the base, proximal and distal level in the two conditions can now be quantitatively compared. An independent sample Chi-Square test was used to compare participants’ behaviors in the speech versus the beep condition; the results (Χ² = 12.94, df = 2, p = .00155) show a significantly different distribution in the extent to which people treat the robot as a machine compared to the extent to which they show anthropomorphizing behavior, based on the presence or absence of the verbal framing utterance at the beginning of the interaction.