Charting User Experience in Physical Human–Robot Interaction

Previous work refers to physical interactions with robots as tactile or pHRI. In the Springer Handbook of Robotics, Haddadin and Croft categorize pHRI as a form of proximate interaction where humans and robots are collocated, and the robot has autonomy in performing (part of) a task [43]. Others use tactile HRI to refer to physical interactions with robots. For example, Argall and Billard discuss that tactile HRI is at the intersection of two areas: (1) tactile sensing and (2) interactions between humans and robots [11]. No clear distinction exists between tactile HRI and pHRI in the literature. While “touch” has more social and experiential connotations than pHRI, both tactile HRI and pHRI have been used to refer to either technical or social aspects in the HRI literature. For example, when the purpose of interaction was purely pragmatic such as in object handover or kinesthetic teaching, the users could associate social and emotional attributes to the robot or interaction. Pan et al. conducted an instance of such a study where users evaluated the social attributes of the robot based on an object handover task [98]. We adopt the definition by Haddadin and Croft and treat pHRI as any collocated interaction that involves an exchange of haptic signals (e.g., force, contact, accelerations) between a human and an autonomous embodied agent. Specifically, in our definition, pHRI is the umbrella category that includes direct tactile interactions where humans are touching or being touched by a robot as well as indirect physical interactions through an object (e.g., during a handover). Also, in our definition, pHRI can refer to both pragmatic and experiential aspects of physical contact.

The literature on pHRI diverges into two areas, one investigating technical engineering challenges of pHRI in the robotics literature, while the second area focuses on the social experiential aspects of physical interactions with robots. Existing pHRI surveys primarily cover the former literature on the technical developments in the field. In their atlas of pHRI, De Santis et al. proposed the safety and dependability of robots as two key criteria for assessing physical interactions [35]. They provided an overview of work on robot hardware and software design toward these two criteria. Relatedly, Hadadin and Croft provided an overview of the technical pHRI literature focusing on human injury analysis and safety standards for pHRI. They presented progress toward human-friendly hardware and algorithm design for robots [43]. Losey et al. reviewed techniques and algorithms related to sharing control between humans and robots in collaborative physical tasks [80]. Argall and Billard categorized the literature according to progress in tactile sensor development and types of physical interactions [11]. The categorization of tactile sensors was purely technical, focusing on the composition of the sensor and the applications of various approaches (e.g., hard sensors vs. soft sensors). They categorized the types of physical interactions into three groups where the touch (1) interferes with robot behavior execution such as in accidental touch, (2) contributes to behavior execution such as in collaborative assembly or robot-assisted therapy, or (3) contributes to behavior development, for instance, when the robot is learning a skill through physical contact.

Others have focused on one type of physical interaction. Most recently, Ortenzi et al. published a review of object handover studies [95]. They described two phases of a handover task as pre-handover and physical handover, noting that important cognitive and physical processes start before the physical part of the interaction. The authors reviewed the progress and gaps in the literature according to these two stages of the object handover task. In addition, they summarized the user evaluation metrics into two categories of objective task performance and UX metrics, and further divided the UX metrics into subjective and psycho-physiological metrics. The above surveys informed our initial codes for analyzing the studies.

Our work complements the above literature by providing a systematic review of UX in pHRI. In contrast to the above surveys, our review focuses on the UX of pHRI. Also, while informative, the above surveys do not explicitly report the sample of articles and the analytical process of the authors. In contrast, we present a systematic review of the literature where we define a sample based on clear inclusion and exclusion criteria and code the articles according to a code book to provide statistics on the trends observed in the sample. With this approach, we also present a data-driven categorization of subjective and objective UX metrics for pHRI across various physical tasks.

Previous work in the HCI domain (e.g., games and mobile phones) has outlined the definition and methods for evaluating the UX of interactive technology. UX research aims to provide a holistic view of human interactions and reactions to technology. Hassenzahl and Tractinsky divided the UX of an interactive product into pragmatic and hedonic factors [49, 50]. Pragmatic or usability factors provide a task-centered view, focusing on the effectiveness and efficiency with which target users can complete specific tasks with a given technology [55, 62]. In contrast, UX is a broader multidimensional concept that also encompasses the positive aspects of interactions such as user motivation, emotions, and esthetics, and reflects the context-dependent and dynamic nature of interacting with technology [15, 50]. Bargas-Avila and Hornbæk provided a systematic review of 51 empirical studies of UX, analyzing the range of products (e.g., websites, and games), UX dimensions, and data collection methodologies employed in their sample. The studies in their review showed a prominent focus on using qualitative methods (e.g., interviews) and evaluating emotions, enjoyment, and aesthetics as measures of UX. We follow a similar systematic review approach and compare evaluation trends in pHRI with those reported by Bargas-Avila and Hornbæk for other forms of interactive technology.

In the HRI domain, researchers have emphasized the need for adopting the UX design practices from HCI [7, 78] and proposed new frameworks to capture the unique nature of interacting with robots [129]. Lindblom et al. described three primary challenges for HRI as the need for robot designers to adopt an iterative process, incorporate UX goals in the development process, and learn about UX evaluation methods and theory [78]. Young et al. discussed adopting three perspectives when designing and evaluating HRI. These perspectives account for (1) visceral factors such as emotions, (2) social mechanics such as gestures and facial expressions, and (3) social structures involving the interaction context [129]. Similarly, Weis et al. provided a framework and guidelines for the holistic evaluation of HRI with four key factors: usability, UX, social acceptance, and societal impact [119]. Grounded in the researchers’ experience and knowledge of the literature, these frameworks highlight the complexity of capturing the UX of interacting with robots. Yet, the extent to which these frameworks or their underlying factors are employed in pHRI studies is unknown.

A few questionnaires exist for evaluating the UX of robots. HRI researchers commonly employ general-purpose questionnaires to measure user emotions [22, 87, 117] or workload [48]. In addition, the Godspeed questionnaire [16] and Robotic Social Attributes Scale (RoSAS) [29] are specifically designed and widely used for robots. The Godspeed questionnaire consists of 24 Likert-scale user ratings about the robot’s anthropomorphism, animacy, likeability, intelligence, and safety. While the questionnaire is not validated, its wide adoption reflects the field’s demand for established evaluation practices. In a series of online studies, Carpinella et al. analyzed the Godspeed questionnaire and developed the RoSAS with 18 items that capture a robot’s perceived warmth, competence, and comfort. RoSAS has been validated in online studies using descriptions of robots or images of robot faces, but the questionnaire is also used in pHRI studies recently [98]. Other instruments such as the Negative Attitude toward Robots Scale (NARS) [91, 93] and Robot Social Anxiety Scale (RAS) [92] capture overall user beliefs and feelings toward robots. Building on this literature, we provide a detailed account of how empirical studies of pHRI measure UX and discuss how our results can inform future evaluation practices and questionnaire development in the field.

To gather high-impact articles on pHRI user studies, we searched through three top HRI venues: the ACM/IEEE International Conference on Human Robot Interaction (HRI), ACM; the ACM Transactions on Human–Robot Interaction (THRI), ACM; and the International Journal of Social Robotics (IJSR), Springer. We used the following query to search in the title and abstract of every publication in the selected venues:

Since we focused on UX in pHRI, our query filtered for articles with both a physical interaction and a user study of the interaction. The first part of the query aimed to capture articles with touch interaction, while the second part included articles with a user study. The wild cards (*) helped us capture different possible wording such as “handover” or “handshake” and “evaluate” or “evaluation.” For HRI and THRI, we could directly run our query using the ACM Digital Library search options. In the case of IJSR, Springer only supports searching on the full text of the article. Hence, we used a custom script to further filter the results obtained from the Springer website to ensure that the search terms only appear in the title or abstract. We included all full articles published before October 2021, when we ran the search query. The query returned 215 articles: 92 from HRI, 32 from THRI, and 91 from IJSR.

To further screen relevant articles from the search query results, we defined the following inclusion criteria:

(1) Physical Autonomous Robots: The article needs to involve a physical robot that appears as an autonomous agent to the user. This criterion ensures that the user thinks they are interacting with a robot rather than a human. We include studies using a Wizard of Oz setup [104] where a human operates the robot without the user’s knowledge. Articles involving virtual avatars or simulations [86], haptic devices operated by a human [107], and teleoperated robots [113] are excluded according to this criterion.

(2) Touch Interaction: The article needs to have a direct touch interaction or indirect interaction through an object (e.g., handover) where physical forces or tactile feedback is exchanged between a human and a robot. The interaction should not be via an intermediate interface such as through a joystick [115], a touchscreen [84], or haptic devices [107, 131].

(3) User Study: Physical interaction needs to be evaluated through a quantitative or qualitative user study by having a research question or a hypothesis about the touch interaction. The study should focus on user response and evaluation rather than hardware design [112] or algorithm development [33]. The participants in the study must either participate in the touch action or observe it.

Two authors independently determined and marked the inclusion of all the articles (0 for excluded, 1 for included) by skimming through the full-text articles. The overall agreement percentage was 90.69%, and Cohen’s kappa was 0.746. Each article with a rating discrepancy was discussed among all three authors, with the third author providing an independent opinion. In this phase, 169 articles were excluded, resulting in a sample of 46 articles for further coding. During the coding process, we further removed eight articles that did not meet our inclusion criteria or were duplicates due to a conference article being extended and later published in a journal. We included the journal version in our analysis.

The final set of 38 articles consisted of 44 unique studies, which we included in the review.

We coded the final set of 44 studies in two rounds. The first round provided summary codes, and the second round resulted in detailed codes for the user studies and physical interactions. In the first round, two authors extracted free-form text from the articles about the study goal(s), physical interactions and their context, user evaluation methods, and high-level outcomes of the article. We copied the article’s text and paraphrased it for brevity whenever necessary. These text descriptions (i.e., summary codes) served as a memory aid for the coders and guided the next coding round.

In the second round, the same two authors defined detailed codes for the physical interactions and user evaluation methods. For the physical interactions, we defined five detailed codes describing the role, type, location, actor(s), and duration of the physical interaction (Table 1). For the user evaluation methods, we coded the time of data collection, methods of data collection, and the questionnaires used in the studies (Table 3) similar to the review by Bargas-Avila and Hornbæk [15]. Also, we further extracted all the subjective user-reported data (e.g., questionnaire items, interview questions) and measurements (e.g., task completion time and user behavior coded from videos). Finally, we extracted parameters that were varied in the studies (i.e., independent variables) as well as the robot model, number of participants, and the participant background data that were collected in the studies (Table 2). We established an interrater agreement for the detailed codes on a random 25% subset of the studies (i.e., 11 studies) before coding the rest of the studies. The interrater reliability was full agreement on 78.03% of the codes, partial agreement on 10.61% of the codes, and a mismatch on 11.36% of the codes. The two authors discussed the differences and updated the descriptions in a coding sheet. The rest of the studies ( \(\textit{n}=33\) ) were coded by one author. The aggregated results of this coding process are reported as numbers, percentages, and instances in the 44 studies (Tables 1–3). The final coding scheme and the codes for the 44 studies are included as supplementary materials.

We further analyzed all the measurements and questionnaire statements from the previous coding phase to identify UX metrics in the pHRI studies (see Supplemental Materials). These measurements and questionnaire statements capture the aspects of UX that the pHRI researchers aimed to cover in their study. Thus, they provided a rich data source for analysis. In two sessions, the three authors created affinity diagrams of the measurements and statements.

We grouped the questionnaire statements that users rated in the studies in one session. We discarded all the ratings where the exact statement or question was not reported in the article (53 rated statements and questions from 14 studies). This left 333 statements and questions for the affinity diagraming session. Before the session, all the statements were printed on article, cut individually, and mixed into a random order. During the affinity diagraming session, the three authors read each statement aloud, discussed the underlying metric, and placed it on a table near statements referring to the same metric. When a statement was unclear, it was set aside for checking in the future. After going through 100 statements, the authors labeled the affinity groups using sticky notes. This process was repeated until all the statements were placed in the groups. If an author proposed a change in the grouping of a statement, all the authors discussed and accepted or rejected the proposed change. In this process, we discarded 17 statements that were either open-ended questions (e.g., “What would you name the robot?”) or the exact measures or their meaning was unclear (e.g., “the participant’s impression of the researcher who proctored the experiment”).

This process resulted in 24 groups (i.e., UX metrics). After the session, one of the authors entered the UX metric for each of the 316 statements (333 statements—17 discarded items) into an Excel sheet. Another author reviewed all the statements and their UX metrics and flagged 31 items that were not consistently categorized. In a subsequent meeting, the three authors discussed these statements and revised the UX metrics for the statements as needed.

In a second session, the authors grouped 72 data measurements (e.g., through datalogs or video coding) from the 44 studies. The three authors followed the same procedure as above to group the measurements. This process led to five groups (i.e., UX metrics). Four of the UX metrics, including accuracy, time, social traits or behavior, and emotion, overlapped with the 24 metrics from the rated statements. One new metric was identified as descriptive measures (e.g., number of actions).

Finally, one of the authors counted the number of studies with a rated or measured item for each UX metric. Table 4 presents the results of the above process.

We discuss the physical interactions according to five parameters that were varied in previous studies: role of touch, who initiates and is active in touching (actor), type of touch, body location, and duration of the physical interaction. Figure 3(f) shows six example physical interactions with different combinations of these parameters. Table 1 gives an overview of physical interaction parameters for all the studies in our sample.

Role of Touch. pHRI has been used in multiple domains such as healthcare, manufacturing, education, technical assistance, and domestic help. Within these domains, we identified four primary roles for physical interactions. First, in most studies ( \(\textit{n }=26\) ), the main purpose of the physical interaction was to communicate or influence the user’s or the robot’s emotions, judgment, or behavior. For example, Law et al. studied if participants’ trust in a robot is impacted after observing the robot touch a person on the shoulder in a video [73]. Shiomi et al. investigated the effect of touch on user behavior. In a study, they examined how touching or being touched by a robot influences user effort in a monotonous task and their judgment of the robot’s friendliness [110]. In another study, the same authors examined if reciprocating the user’s hug by a teddy bear robot can increase the user’s interaction time and amount of self-disclosures [111]. Fitter and Kuchenbecker studied user perception of the social attributes of a Baxter robot during a playful hand-clapping game [39]. Second, in several studies ( \(\textit{n}=8\) ), physical contact was required to support completing a task, such as object handover [8], nursing [32], or opening a path for robot navigation [64]. Third, in a subset of the studies ( \(\textit{n}=10\) ), the physical contact served to guide or teach movement of the robot or the user. In particular, five studies had the user involved in guiding a robot to a position through direct touch [31, 38, 42, 63], four studies examined UX of programming movements of a robot by physical demonstration [5, 52, 79, 123], and one study focused on a robotic platform supporting user movement in a walking rehabilitation task [74]. Fourth, only in one study ( \(\textit{n}=1\) ) the physical contact was unintended, meaning that the contact appeared to be the result of an error [64]. Given that accidental contact with robots can happen as a result of errors or noise in the robot hardware or algorithms, the lack of research on the effects of accidental contact on UX is surprising. We discuss this gap in Section 5.

Who Touches. The studies also differed in whether the human ( \(\textit{n }=19\) ), the robot ( \(\textit{n }=10\) ), or both ( \(\textit{n }=20\) ) were actively engaged in the touch interactions. In this case, analyzing the link between who touches and the role of touch in the reviewed studies revealed interesting patterns. When the human was touching, the role of touch could be any of the four categories mentioned above. Joint touches were used to communicate or influence users ( \(\textit{n }=12\) ), support completing a task ( \(\textit{n}=7\) ), or teach or guide movement ( \(\textit{n }=1\) ). In contrast, when the robot was touching, the role of the touch was mainly to communicate or influence the participant ( \(\textit{n}=9\) ). For example, Yeufang et al. investigated how a robot’s touch impacts the user’s emotion and attitude toward the robot [130]. Only in two study conditions did the robot-initiated touch support completing a task. In one study, the robot cleaned the forearm of the user in a nursing context [32], and in another study, the robot touched the participant to open a navigation path [64]. Robot-initiated touches were not used to teach or guide the user’s movement, and a robot did not initiate an unintended accidental contact. These scenarios are imaginable in physical interaction with robots, but they are underexplored in the literature.

Type of Touch and Body Location. The type of touch interaction was described with various phrases and levels of detail. Several studies ( \(\textit{n}=13\) ) referred to the interaction generally as touching without specifying it in detail. Others used a label (e.g., tapping) to describe the touch. However, we could not discern whether the use of these labels was consistent across different studies. The location of touch was mainly on the hand ( \(\textit{n }=19\) ) or arm of the users or the robots ( \(\textit{n}=7\) ). Others involved multiple areas of the body ( \(\textit{n}=10\) ) such as during a hug, or focused on other body parts ( \(\textit{n}=11\) ) such as the shoulder or waist. The focus on hands and arms is similar to the haptics literature, reflecting the importance of hands in sensing and manipulation. In contrast, haptic devices targeting multiple or other body parts are rare.

Duration. The duration and timing of the touch showed trends and some methodological issues in the reviewed articles. The duration of physical contact was reported in only about half of the studies ( \(\textit{n}=23\) ). When reported, the duration ranged from a few seconds ( \(\textit{n }=13\) ) to minutes ( \(\textit{n }=2\) ). In some cases, the duration was not limited in the study and differed between participants ( \(\textit{n }=8\) ). In all cases, the touch event was part of a longer interaction timeline, usually spanning a single session. Only one study was conducted over several sessions. Huijnen et al. ran a four-session study with children with autism over 4 weeks to compare their attention and interaction with the KASPAR robot and a teacher [59]. The time of the robot-initiated touches ( \(\textit{n }=10\) ) was either implicitly or explicitly communicated to the participants. In two studies, the participants could not anticipate the exact timing of the touch. In one study, the robot occasionally touched the participant during a scary movie [121], and in another study, the robot touched the participant’s shoulder to open a navigation path while passing from behind the participant [64]. In other cases, the participants could either anticipate the contact from the robot’s verbal announcement or the study protocol ( \(\textit{n }=3\) ), or they only observed videos of a robot touching someone ( \(\textit{n }=5\) ). Studies of the robot touching the user without explicit notice or permission can improve the efficiency of human–robot teaming, but they are underexplored in the reviewed studies.

Most reviewed studies ( \(\textit{n}=39\) ) used controlled experiments to study the effect of one or more independent variables on UX or task performance. We examined the independent variables (Table 2) and the data collection methods (Table 3) that the authors employed in their studies.

Independent Variables. Most studies varied the parameters of the physical interaction. Some compared direct physical contact with the robot to no contact or to interacting through an interface such as a joystick ( \(\textit{n }=10\) ) while others varied who touched the other ( \(\textit{n}=4\) ). Variations in sensation or motion parameters included the type of touch and robot forces [9, 18, 19, 39, 79, 123, 125], body materials [124], temperature [19], the existence of clothes [10], as well as robot’s position or trajectory of motion [8, 25, 45, 69, 88, 98] for the physical interaction. Other studies examined timing variables such as the touch duration [18, 19, 45] or robot’s movement speed [98] or reactivity [39]. Two studies varied the role of the touch interaction [32, 64]. Specifically, in one study, the robot stroked the user’s forearm to either clean it (i.e., completing an action) or to support the user emotionally [32]. In another study, the robot directly touched the user to open a path for its navigation (support completing a task), or the user accidentally collided with the moving robot (unintended contact) in two different conditions [64]. The location of touch was only varied in one study. The focus on studying the impact of physical parameters is not surprising for pHRI studies.

Other studies varied factors such as visual and auditory modalities, the interaction tasks, and the user or robot’s background or intent. Only one study varied the robot form by asking users to rate videos of two hugging robots [18]. All the other studies used a single robot. Similarly, verbal utterances or sounds were also only varied in one study. In this case, Chen et al. manipulated the timing of verbal utterance in relation to touch [32]. The results of these studies confirmed that various contextual parameters can mediate the UX of touch. Finally, a few studies did not have any independent variable; they examined the correlation between different variables and measurements (e.g., number of verbal utterances and data from temperature sensors) without systematically varying them [14] or collected user interaction data and experience for only one physical interaction condition [34, 100].

Study Methods. We analyzed the user study methods based on the time and methods of data collection and the questionnaire types and sources following Bargas-Avila and Hornbæ’s analysis of the HCI literature [15].

The results of coding the time of data collection show clear similarities between the pHRI and the HCI literature (Table 3). Researchers often collected the data after ( \(\textit{n }=37,84\) %) or during ( \(\textit{n}=25,57\) %) the physical interaction. A small set of studies also collected data before the physical interaction ( \(\textit{n }=11,25\) %). The before data provided baselines for physiological recordings ( \(\textit{n }=3\) ) or user impressions of a robot before physical contact ( \(\textit{n }=5\) ). The video recordings helped the researchers code the start time of the physical interactions in two studies. Similar to these trends, the review of UX methods in the HCI literature reported that researchers often collected data after the user interaction (70%), followed by during (58%) and before the interaction (20%).

In contrast, the data collection methods in our review (Table 3) are less diverse and more focused on quantitative data than those reported in the above-mentioned review. Most pHRI studies in our review used a questionnaire with subjective ratings ( \(\textit{n}=38,86\) %), followed by video recordings ( \(\textit{n}=15,34\) %) and datalogs ( \(\textit{n }=7,16\) %). Interviews and other data collection methods, such as measurements of body movements and think-aloud protocol, were rarely used. In contrast, Bargas-Avila and Hornbæk noted that only 33% of their reviewed studies mainly used quantitative methods. Also, they reported a wider range of data collection methods in their sample, such as live observation, diaries, probes, collages or drawings, and photographs.

Many studies ( \(\textit{n}=20\) ) use self-developed questionnaires. In a subset of the studies ( \(\textit{n}=4\) ), the authors do not report the exact statement or questions presented to the participants. Validated questionnaires are often used for evaluating one’s emotions or workload. Similar methodological trends and issues are previously reported in other fields [15] and are known to the community. Such patterns may be inevitable in a developing field and suggest the need to develop questionnaires and best practices for pHRI studies.

To capture the metrics of the pHRI experience, we collated the rated statements and measurements in the reviewed articles, created an affinity diagram, and grouped the resulting metrics into five facets. Table 4 shows the 25 UX metrics from this process. One can note the variety and distribution of the UX metrics in the reviewed studies. No metric was used by more than half of the studies ( \(\textit{n} \gt 22\) ), and some metrics such as autonomy and motivation were included in only one study. Among the UX metrics, accuracy, time, social traits or behavior, and emotions are captured with both rated and measured items. In contrast, the rest of the metrics are only collected via rated statements, and descriptive measures is only among measured items. We further divided the 25 UX metrics into the following five facets:

F1—Overall. These UX metrics provide an overall interaction perspective without focusing on any specific component. The metrics include descriptive measures and overall evaluation ratings.

F2—Usability. This facet provides a task-centered perspective and focuses on the users’ performance and opinion in relation to completing a task. These metrics are known for evaluating the usability of an interface in the HCI literature [55]. Here, the robot is regarded as a computer interface. Ease of use, accuracy of the outcome, user workload, or extent of understanding the task are covered in several studies, whereas learnability and feedback are less common in our sample. Hornbæk reports a wider range of usability metrics for the HCI literature [55], some of which are more detailed subsets of the metrics found in our sample. For example, Hornbæk reports mental effort, communication effort, and information accessed, but we include all of them in workload. Other usability metrics from Hornbæk’s review, such as quality of outcome, binary task completion, and input rate, were rare or absent in our sample.

F3—Sensory. The metrics in this facet evaluate the interaction or robot in relation to the human basic senses such as user perception of visual, auditory, or physical aspects. A few studies ask about attributes that are judged visually, while only 3 studies (out of 44) ask users to rate touch sensory properties. Only one study includes a rating of auditory components [124]. Our sample of studies did not include any metrics related to the sense of smell or taste. We could not find an exact match for this facet in the HCI literature. Metrics related to esthetics or appeal in the HCI literature [15] and the autotelics from recent work on haptic experience [67] seem the most related to this facet.

F4—Personal and Interpersonal. This facet evaluates the robot as an autonomous being. The metrics cover the judgment of the robot’s characteristics, capability, or social attributes and the judgment of the robot’s joint interaction with the user or teamwork. Only 5 studies ask users to rate teamwork when 20 studies have a mutual touch between the user and the robot. Overall, the personal and interpersonal metrics are well represented in our sample with many of the rated statements (142 out of 317). In contrast, these metrics are not present in the HCI literature.

F5—Experiential. These metrics deal with the user’s feelings, are more holistic, and are not focused on a task. The metrics in this facet overlap with the UX metrics reported by Bargas-Avila and Hornbæk, but we note a different distribution in our sample. Among the metrics, enjoyment, emotion, and motivation are present in both our and their HCI sample. In contrast, the UX of safety is the most common metric in our sample, but it is absent from their review. The focus on safety in the pHRI literature is perhaps due to the possibility of user injury in physical interactions with robots compared to the minimal risk of interacting with visual and auditory interfaces common in HCI. Surprisingly, the experience of autonomy is only included in one study in our sample.

The experiential (F5) and personal and interpersonal (F4) metrics are included the most in our sample, followed by the usability (F2) and overall (F1) metrics. The sensory metrics (F3) are evaluated the least in our sample.

Analyzing the distribution of UX metrics employed in the pHRI studies highlights that researchers are often interested in more than a single aspect of UX but have difficulty delineating their UX goals. While some studies focused on a single UX metric such as emotions (e.g., [9, 125]) or social traits or behavior (e.g., [34]), most studies ( \(n\gt30\) ) in our sample collected data on more than one UX metric. In several cases, the studies collect data on UX metrics spanning across both task-centered usability metrics (F2) as well as metrics in the personal and interpersonal or experiential facets (F4, F5). For example, for human–robot collaboration in an assembly task, Gleeson et al. collected data across 11 UX metrics ranging from ease of use, time, and workload of the task to the physical sensation, the robot’s personal and interpersonal traits, and user enjoyment [42]. However, the goal statements are often either broadly defined or are not directly linked to the study’s metrics. For example, some studies aimed to assess if a touch condition was “positive” or “favorable” but collected user responses for several UX metrics (e.g., 16 metrics) across all the facets. Defining the metrics and facets of the pHRI experience can provide researchers with a lexicon to clearly specify and link their goals to data collection.

We propose a conceptual model for pHRI (Figure 1) with three main components: (1) design parameters, (2) interaction timeline, and (3) UX metrics. While this article focuses on the UX of pHRI, the first two components can directly impact UX and thus are included in our conceptual model. These components can help designers and researchers describe an interaction, use it to generate ideas about alternative designs and evaluate the UX of the interaction. Below, we briefly describe each component and discuss its utility for pHRI research.

(1) Design Parameters: The first component of the model depicts the pHRI design parameters around the user, robot, and their interaction. Researchers can manipulate the design parameters to study their impact on UX. For instance, one may design the pHRI for a specific demographic user group or compare the UX of two user groups. For example, Law et al. studied the impact of a user’s gender on their trust in a robot after watching a video of the robot tapping a person on the shoulder [73]. Others could study the impact of user expertise or beliefs (e.g., negative attitude toward robots [63]) on the pHRI experience. Studies of pHRI frequently manipulate physical and overall interaction parameters (Table 2). Finally, several design parameters exist for the robot, such as its multisensory presentation (e.g., robot’s appearance, gaze, and sound) or its social parameters of the robot’s role, intention, or attitude.

Design parameters have both descriptive and generative power by nature. The list of parameters and their values can be used to clearly describe what is manipulated or kept constant in an interaction. The parameters can also help generate ideas for new interactions. For example, how does the robot’s appearance or form impact UX outcomes? As another example, all the studies in our sample focus on the interaction between one human and one robot. Yet, one can examine UX where a team of humans or robots are present [68].

(2) Timeline: The second conceptual component is the interaction timeline, which enables researchers to describe the sequence and timing of physical and overall interactions. The temporal description of pHRI in our sample of studies is surprisingly incomplete. Duration of touch stimulation is a salient parameter for a haptic signal [127]. Yet, many studies in our review do not report the duration of their physical interactions. The interaction sequence is often described qualitatively and sometimes mixed with the description of the study procedure. The timeline component provides descriptive power for pHRI studies and emphasizes the impact of previous interactions, such as verbal utterances, on the pHRI experience.

Second, a timeline view of pHRI has generative power. A few studies in our sample showed the importance of temporal parameters (e.g., motion speed and duration) and the sequence of verbal and touch modalities [32] on UX outcomes. The temporal view is in line with work on robot sensing and planning algorithms that are sequential in nature. In robot sensing, the difference between touch gestures is in the spatiotemporal signature of the touch signals obtained from the sensor [24, 26]. Robot planning frameworks such as partially observable Markov decision processes [72] and reinforcement learning are inherently sequential. Considering interaction as sequential decision-making is also gaining traction in computational models of human interaction with technology [96]. Sequential descriptions of pHRI can facilitate knowledge transfer from pHRI studies to computational simulations of pHRI interactions and results.

(3) UX Metrics: The third component of the model is the UX metrics and their five facets. This component captures the complex and multidimensional aspect of UX in pHRI. The UX metrics are obtained from the statements and measurements in the studies and divided into overarching facets.

In contrast to the design parameters, the relationship between the UX metrics and the user, robot, and interaction entities is complex. Specifically, the overall (F1) metrics can apply to any of the human, robot, or interaction entities, the usability (F2) metrics can apply to the robot or the interaction, the sensory (F3) and personal and interpersonal (F4) metrics mainly apply to the evaluation of the robot or the joint work, and the experiential metrics (F5) can apply to the human or the robot (e.g., evaluating emotion of the robot). Thus, we represent them separately from the user, robot, and interaction entities in the conceptual model. Among these metrics, the first three facets (overall, usability, and sensory) can apply to interactions with computers, personal and interpersonal metrics only apply to robots or virtual agents, and some metrics from the experiential facet such as safety, and autonomy mainly apply to pHRI or interactions with Artificial Intelligence (AI).

The five pHRI facets are meant to provide an initial guide, rather than fixed categories, to evaluate various aspects of UX. Thus, the metrics in the five facets can relate and overlap. For example, ease of use can refer to the evaluation of an interface or task and also convey a subjective feeling of ease, making the metric relevant to both the usability (F2) and the experiential (F5) facets. Similarly, the usability and UX metrics are not strictly separate and overlap in the HCI literature. In the proposed model, we included ease of use in the usability metrics for two reasons. First, the ease of use statements in our sample refer to the evaluation of the task or the robot as a tool when completing a task, in line with our definition of the usability facet. Second, this categorization is aligned with existing HCI literature where ease of use is regarded as a usability rather than an UX metric [55], enabling the development of shared theories across the pHRI and HCI fields.

The UX metrics and their facets provide descriptive and evaluative power for pHRI research. The metrics can help researchers better describe the goal(s) of their pHRI interactions and evaluate the relevant metrics in empirical studies. Also, the metrics and their underlying statements (see supplementary materials) provide a starting point for creating a validated questionnaire for pHRI experience. The design parameters and the interaction timeline influence the UX outcomes. Yet, how these parameters impact UX outcomes cannot be currently established due to the methodological variations and gaps in the pHRI studies.

Using the Conceptual Model of pHRI Experience. We anticipate three ways that future work can use and build on this conceptual model to further highlight nuances of designing and evaluating physical interactions with robots.

First, researchers can use the model to identify gaps and define research directions. For example, an open question is the impact of different design parameters on the pHRI experience. Touch is often described in the literature as a personal and emotional communication channel when compared to the audio and visual modalities [83]. Future work can test whether this assumption holds in pHRI by designing experiments where a robot communicates emotional support through physical contact vs. other channels (e.g., gaze, facial expressions) and comparing user evaluation of personal/interpersonal and experiential metrics (F4, F5). Another important area for future work would be to investigate the impact of temporal parameters such as the duration, repetition, and influence of previous interactions on the UX of pHRI. Also, the combination of the conceptual model and the review of pHRI studies in our sample highlight gaps and open challenges for future research as we discuss in the next section (Section 5.2).

Second, the model can serve as an initial structure for promoting a shared understanding and communication among researchers with various engineering and social experiential research directions in the field. By charting the pHRI space with a set of parameters, the model outlines what and how to report pHRI experiments. The model can also help teaching and education in the field, allowing students to identify and discuss the design considerations in pHRI experiments.

Third, we hope pHRI researchers improve the model based on their empirical results. With improved reporting of design parameters and UX metrics, future research can revise the conceptual model to establish stronger links between different components. For example, our model currently divides the design parameters according to the user, the robot, and their interaction, but future empirical studies may lead to a hierarchical structure for the design parameters that conveys the relative importance of the parameters on UX outcomes. Also, future studies of the temporal parameters of pHRI can help further specify the timeline component. Eventually, these descriptive models can form the basis for predictive and computational models of pHRI experience.

We summarize gaps in the pHRI studies and discuss avenues for further charting the UX of pHRI.

Physical Interaction Parameters. Our review highlights three gaps in studying the physical parameters of pHRI. First, little work exists on the effect of unintended and accidental robot contact on users. Only one study in our sample included unintended contact. As robot hardware and algorithms are susceptible to error, studying the UX of accidental contact and other haptic errors (e.g., force variations) is an important area for future work. Studying accidental contact in an ecologically valid way is challenging and would require a careful study design. We anticipate advances in predicting user movement trajectories [30, 44] and detecting anomalies from robot sensor data [99] can help in conducting such studies in the future.

Second, our sample did not have any studies where robot-initiated touch guides or teaches movement to humans. Such physical interactions are particularly relevant for rehabilitation and skill training scenarios. Robotic devices for physical rehabilitation are often not (perceived as) autonomous systems. Also, rehabilitation studies often focus on hardware design and clinical outcomes [40, 81]. Future work should assess the UX of being guided by robot-initiated touch.

Third, the studies in our sample often involved short physical interaction episodes (e.g., seconds to minutes). We conjecture that longer interactions may be possible depending on the form of the robot. For example, users may hold and touch robotic pets such as PARO for longer (e.g., 30 minutes) and multiple sessions over several weeks or months [102]. Also, autonomous vehicles or automatic beds that simulate the feel and breathing of a human body [51] may be viewed as a form of pHRI; these technologies can sense the environment and “touch” the user, and the duration of physical contact can extend to hours or days. Do users perceive these systems as robots? The relationship between the UX of touching and being touched by these technologies and conventional robots is an open question for future work.

UX of Observed vs. Real Touch. Our sample included five studies of observed touch where participants watched videos of physical interactions between a human and a robot. A few studies in haptics and HRI show that people can infer tactile sensations and their emotional associations through vision [17, 57, 108, 114, 120]. These studies are in line with neuroscience research suggesting that a subset of motor neurons is fired for the observation or execution of actions, offering a neural basis for human empathic response [65]. On the other hand, a recent study showed that observing robot touch in a video vs. receiving robot touch in the lab can result in different emotional reactions in users [70]. Little is known about the extent to which user evaluations of observed and actual pHRI are consistent or different for various facets and metrics of the pHRI experience. Thus, future work should further investigate the relationship between observed vs. felt pHRI.

Generalizing Beyond a Single Robot. The impact of robot form and body materials on the pHRI experience is another open question. For example, users may experience the same tapping gestures differently depending on whether the robot is human or machine-like or whether the robot’s body is made of hard or soft materials. The studies in our sample used a single robot (except for Block et al. [18]) while acknowledging that their results may not generalize to other robots. The variety of robots used in the pHRI studies prevents the community from deriving generalizable findings and formulating guidelines for pHRI design. One obvious solution is to test the same physical interaction(s) with several robots. Yet, robots can vary across many parameters. Researchers often have access to one or a few robots, and adding a new robot can notably increase the time or number of participants for a study. These factors make this problem intractable in a lab setting and for a single research group.

We anticipate two possible solutions to this problem. One solution would be to crowdsource this task across multiple HRI research labs that follow one predefined reproducible interaction protocol. Results of studies from different research labs can be compiled in a database and pulled to assess generalizable trends. An alternative solution can aim to identify archetypal robot forms, create visual proxies of the physical interactions (e.g., with video, virtual reality), and collect data on UX at scale. Recent collections of robots [1, 2, 3, 4, 108] can help HRI researchers formalize archetypal robot forms and their variations. Then, designers create virtual models of archetypal robot forms and program the interactions with a human avatar to collect user responses. Since physical feedback is not present, the collected data can only be an estimate of UX. They must be combined with smaller-scaler in-lab studies to assess the validity of the estimates. Progress in haptic feedback in virtual environments can further help with addressing this problem in the future [116].

Developing and Validating a Questionnaire for pHRI Experience. Our review highlights the need and provides a starting point for developing an UX questionnaire for pHRI. Existing questionnaires are either adapted from other fields or cover a small subset of the long list of metrics that matter to pHRI researchers (Table 4). On the other hand, the relationships between different pHRI metrics are unknown. Which ones correlate? What tradeoffs exist among different UX metrics? Future work on developing a pHRI questionnaire can shed light on these relationships and tradeoffs.

Establishing a valid questionnaire has an extensive process with several rounds of development and validation with many users [94]. Our work provides a list of UX mertics with their relevant statements for developing the questionnaire and a summary of use cases against which the instrument could be validated. The list of rated statements can be expanded through common user research methods (e.g., brainstorming, scenarios) to cover underrepresented dimensions (e.g., autonomy) in the initial phases of questionnaire development. Recently, HRI researchers have successfully used crowdsourcing platforms such as Amazon Mechanical Turk to develop and validate questionnaires such as the RoSAS based on images of robots [29]. Similarly, pHRI researchers may use crowdsourcing with videos of physical interaction use cases to collect responses at scale for the development and initial validation of a pHRI questionnaire. However, given the importance of haptic feedback in pHRI, such a questionnaire must be further validated through in-lab studies.

Going Beyond Ratings of UX. The pHRI literature can further develop thick qualitative descriptions and objective measures of UX. Most pHRI studies in our sample use subjective ratings to capture UX of their physical interactions. While valid, this focus is limiting. This approach falls between the quantitative measurement-based view common in robotics and the qualitative social science view that provides in-depth descriptions of a complex phenomenon. We argue that more pHRI research is needed on the two ends of the spectrum of the qualitative-to-quantitative data collection methods.

On the qualitative end of the spectrum, in-depth interviews and contextual inquiry can provide thick subjective descriptions of the UX of pHRI and shed light on design parameters and qualities that researchers may overlook. For example, the sense of autonomy, teamwork, and symbolic experience of pHRI are hardly captured in the studies. Qualitative studies can further highlight the nuances of these dimensions to inform future work. Interestingly, the review by Bargas-Avila suggests that such qualitative studies are prevalent in the HCI literature [15], and they have been used to highlight nuances of UX with other technological artifacts [71].

On the quantitative end of the spectrum, we need more research on identifying objective behavioral measures of UX. User ratings are hard to collect and potentially disruptive during the interaction, limiting the data to before and after the interaction. As such, the temporal view of physical interactions with robots is still poorly understood. A few studies in our review incorporated measures of user gaze [39, 63] or contact force [34], yet objective metrics of pHRI experience are sparse and hard to link to subjective ratings. For instance, user ratings of safety are often collected in pHRI studies. Yet, the relationship between user perception of safety and existing definitions and safety standards in the robotic literature [43] is underexplored. Devising objective or behavioral pHRI metrics and linking them to subjective UX metrics remains an open challenge for future work.

Adapting Robot Behavior to UX. Metrics of UX are mainly used as an outcome measure to compare predefined interaction conditions. How can UX metrics guide robot planning and learning during the interaction? How can robots adapt their touch actions to the user’s state and decide when the touch could be useful or conversely recognize if touching is undesirable by the user? Past pHRI studies have adapted the robot motion to the user body (e.g., size) or movement (e.g., user speed) [18, 58] with positive outcomes. Yet, the extension of this approach to UX metrics is not apparent. As mentioned above, a possible direction is establishing a link between behavioral measures (e.g., user gaze) and subjective ratings and incorporating them as an optimization parameter in robot learning and planning algorithms. Another approach may use advances in conversational agents to obtain an estimate of UX from natural dialogues during the interaction. Finally, a third approach can build on the pioneering work of Card and Moran [27, 28] and its extensions [75, 96] to predict human behavior in a task. These studies attempt to model the human interaction partner as an agent with perceptual [66], cognitive [56], and motor constraints [82] to predict the human’s actions in a task. Extensions to model human emotion are also being developed [89]. While these computational models are still in their infancy and cannot capture the full range of UX (e.g., motivation), they provide a path for adapting pHRI algorithms to user behavior and experience.

In the last 2 years, 11 other pHRI studies were published in our 3 target venues, showing similar trends to studies covered in our review. Specifically, several studies focused on communicating emotions with robots [13, 20, 128] or predicting user perception of robots before and after touch [108]. In other studies, the physical interaction aimed at completing a task, including two studies on object handover between a human and a robot [36, 60] and a co-manipulation scenario where the user and robot hold and move an object together [85]. Finally, four studies focused on teaching or guiding movement [61, 77, 101, 122]. Interestingly, in three of these studies, the robot was guiding a human [61, 101, 122] each with a different robot form: a drone that corrects user motion during an exercise [122], a robotic cane that can guide a visually impaired user [101], and a humanoid robot that guides the user while walking hand-in-hand [61]. These studies target one of the gaps in the literature where a robot guides or teaches movement to humans. The UX methods and metrics in these 11 pHRI studies were also similar to trends reported in our review. Most of the studies ( \(\textit{n }=7\) ) used custom questionnaires, followed by NASA TLX questionnaire for workload, and a few studies reported measures of task accuracy, gaze or blinking behavior, or physiological recordings.

The UX metrics used in pHRI vs. HCI suggest key differences and shared characteristics for user interactions with robots vs. other technologies.

Most UX metrics in our sample fall in the personal/interpersonal facet (F4) while these metrics were absent from prior surveys of UX in HCI. Perhaps the frequent inclusion of personal/interpersonal metrics in pHRI reflects the common belief that robots are social agents. Previous work has shown that people evaluate robots in the same way they evaluate other humans according to competence, warmth, and comfort [103]. This social view of robots is also reinforced by popular fiction [37]. Relatedly, the design of robots’ actions is often inspired by or modeled after human-human or human-animal interactions. Another important factor in pHRI experience is the robot’s physicality and autonomy. Young et al. argued that the robot’s physical presence and autonomy to act in the personal and social spaces of humans can convey a sense of agency and intentionality often attributed to living creatures [129]. These factors are emphasized in pHRI since physical modifications of the world can signal competence, and touching is often associated with personal or social behavior.

Other key differences are related to the UX of safety (from F5), frustration, and sensory (F3) metrics in the pHRI and HCI literature. The focus on safety in our sample is aligned with the technical research in pHRI that aims to quantify and reduce the potential for harm and injury to users [43]. These safety considerations are less relevant in HCI when the user interacts with software-based user interfaces. In contrast, a common UX in HCI is frustration during computer use [53], but this aspect of UX is absent in our sample of pHRI literature. HRI studies have started to investigate user frustration in HRI [90, 118], which can help develop relevant measures for pHRI research as well. Also, the sensory facet has been a focus in the multisensory HCI and haptics research communities for decades but is rarely present in our sample of the pHRI literature. This facet presents an opportunity for leveraging HCI literature to inform pHRI design and research.

The overall (F1), usability (F2), and experiential (F5) metrics were frequently captured in both the pHRI and HCI literature. The inclusion of usability metrics suggests that robots can also be perceived and evaluated as a computer interface. Thus, the social vs. device appraisal of robots may depend on the interaction context and applications. In the HCI literature, the experiential factors are described as the “third wave in HCI” focusing on aesthetics, affective interaction, and the embodied and contextual aspects of human activity and experience [21, 37, 47]. These considerations overlap with the mixing of technical and social considerations when designing pHRI.

Our work has three main limitations that can be addressed in future work. First, we scoped our sample to publications in three top-tier HRI venues. This decision was pragmatic. Given no prior systematic review of pHRI, we found it important for our analysis to cover various aspects of the user studies (e.g., goals, data collection methods), parameters of physical interactions, and UX metrics. Thus, we narrowed our sample to these three venues that focus on human-subject studies in robotics. Yet, pHRI studies with UX metrics are sometimes published in other robotics, engineering (e.g., rehabilitation, haptics), or interaction design venues (e.g., HCI, affective computing). Future systematic reviews can complement our work by analyzing one aspect (e.g., physical interaction parameters) in a larger sample to assess whether our reported trends can generalize across venues. Second, we obtained the UX metrics and their facets through affinity diagraming and discussion. Affinity diagraming is an established method for qualitative data analysis [46]. We acknowledge the active role of the researchers in generating the clusters from the data and the inherent subjectivity of our analysis method [23]. To support future work, we provide all the rated and measured items annotated with their UX metrics from our analysis as supplementary material. Third, our review focused on interactions with robots that are autonomous or perceived as autonomous by users. Yet, robot autonomy is a spectrum from user-controlled (e.g., in grounded force-feedback haptic devices or teleoperation) to semi-autonomous (e.g., shared-control robots) to fully autonomous robots. Future work can extend the proposed conceptual model to capture interactions with robots with varying levels of autonomy.