Evaluating the use of human aware navigation in industrial robot arms

1 Introduction

“A robot may not injure a human being, or, through inaction, allow a human being to come to harm,” the First Law of Robotics put forward by Asimov [1] is a guiding principle within human–robot interaction (HRI). The interpretation of harm has primarily been interpreted as physical harm despite Asimov not limiting the definition to that within his short stories. For instance, in pp. 101–122, a robot gives people false information to prevent them from feeling upset to not break the first law. Furthermore, the short stories serve more as a warning to the pitfalls of setting such laws without flexibility for the inherent unpredictability of human behavior. Currently, technology does not allow for a robot to be able to read someone’s thoughts, but there is the technology and research available to understand how a robot’s actions may affect a person’s psychological well-being (discussed in further detail in Section 3). This is becoming more relevant as not only the number of robots within the industrial environment increases but also as the number of collaborative robots increases [2,3,4]. Initially designed to be used as tools for completing highly repetitive tasks [5], the range of applications for robots has expanded significantly from providing meals, laundry, and basic patient care in hospitals [6], to tour guides in museums [7,8]. High speed, high levels of repeatability, and continuous programming are all advantages that robots have over human workers. This is offset by the difficulty robots have adapted to dynamic changes in the working environment, which human workers can generally take in their stride. In ref. [9], it was highlighted that there are two elements of safety to be considered during HRI: physical safety (the action does not result in an injury for the human) and psychological safety (the action does not result in fear or surprise for the human). Research into navigation while maintaining psychological safety has been more limited in relation to industrial robot arms (stationary and typically high payload capabilities) than with mobile robotics (a robot capable with no fixed base, capable of moving in the environment). Although mobile robots have been deployed for use in HRI since the late 1990s [10] and the psychological effects of robotic behavior were becoming prominent in the early 2000s [11], it was not until the mid-2000s that these were combined to design a path planner that would incorporate them both. A Human Aware path planner is one that incorporates a person’s psychology into the path finding calculations [12]. Therefore, to assess the current state of Human Aware Navigation (HAN) in industrial robot arms, the following questions will be addressed:

• How can the current HRI safety guidelines be optimized for maintaining the physical and psychological safety of the operator?

• How can the approaches for HAN in mobile robotics be applied to an industrial robot arm?

To achieve this, first, a review of the current guidance for physical safety in the industry for robot arms was conducted, with the aim of highlighting the lack of psychological safety considerations. This was followed by a systematic literature review for HAN, where papers were searched on Google Scholar under the search terms “Human Aware Navigation” and “Human–Robot Interaction,” with the aim of providing answers to the questions posed earlier. The review included over 50 papers between 1998 and 2020, with papers only included if they involve HRI and the measure of a psychological variable as a result of the HRI.

2 Safety with barriers – the current state of safety in industrial robot arms

With an increasing prevalence of HRI in an industrial environment, more attention has been paid recently to dynamic obstacle avoidance in static base robot arms. This research differs from mobile robot navigation in that the dynamic obstacles are generally assumed to be a person, and therefore, the path planning and obstacle avoidance are designed with physical safety as the main priority and come with a more conservative approach [13]. The traditional method was the use of physical barriers to completely enclose the robot when it is in operation. However, this is beginning to change as shown by the guidance set in ISO 10218-2:2011 and ISO/TS 15066:2016 [14,15]. ISO 10218-2:2011 is the second part of the standards encompassed by ISO 10218, where the first part covers the “design and application of the particular robot integration” and the second part “provides guidelines for the safeguarding of personnel during integration, installation, functional testing, programming, operation, maintenance and repair.” Although the scope of ISO 10218 is industrial robots, ISO/TS 15066:2016 focuses on collaborative robotics by providing “guidance for collaborative robot operation where a robot system and people share the same workspace.” Following the guidance set in ISO/TS 15066:2016, when a human enters the workspace of the robot arm, one of the three following measures must take place for safe collaborative operation:

• Safety-rated monitored stop, which involves the robot ceasing motion before the operator enters a preset collaborative workspace. If the robot is in motion and within the workspace when the operator enters, then the robot ceases motion, only continuing again once the operator has left the workspace.

• Speed and separation monitoring, which involves the robot maintaining a protective separation distance, which can be reduced with reduced robot speed and/or by the robot executing a different path. If the distance is reduced to below a set value, then the robot completes a safety-rated monitored stop.

• Power and force limiting, which involves reducing the levels of impact should a physical contact between the robot and the operator occur. The reduced impact can be achieved by increasing the contact surface area, mechanisms, and/or materials for absorbing the energy, extending energy transfer time, and limiting movement masses.

These can be categorized into two distinct strategies: precollision and postcollision. A precollision strategy aims to prevent a collision from happening (safety-rated monitored stop, speed, and separation monitoring), while a postcollision strategy aims to minimize the potential damage when a collision occurs (power and force limiting). Although not a collision avoidance/mitigation strategy, ISO/TS 15066:2016 also mentions hand guiding within the HRI guidelines, which involves the robot performing a safety-rated monitored stop followed by the operator maneuvering the end effector.

The precollision strategies operate as guidelines for path finding and obstacle avoidance algorithms although these have been an area of research well before the introduction of the guidelines. Algorithms for navigating the world have been studied and developed for nearly 80 years, with the A* [16,17] and the artificial potential field (APF) [18] algorithms coming to prominence in robotics and HAN (see Section 3). A* algorithm generates a cost of traveling to a point primarily on the distance (although other variables can be added based on the use such as obstacles), an APF algorithm generates repulsive and attractive fields around obstacles and goals, respectively. By reducing the calculation cost, APF algorithms are able to operate more efficiently in a 3D space and provide real-time obstacle avoidance [19,20,21]. Although the original APF algorithm suffered from local minima and Goal-Not Reachable Obstacle Nearby issues, iterations have shown that these issues can be overcome without completely rewriting the algorithm [22,23,24,25,26]. Both of these algorithms benefit from a relatively low complexity, allowing for further variables to be added to the cost function/repulsive fields.

The algorithm, however, is only one element required for an effective precollision strategy. For a robot to avoid an obstacle, it must also have a means to detect the obstacle. While an algorithm may be efficient with a high avoidance success rate in simulation, if the detection system is not adequate for the task, then the avoidance success rate will decrease [20]. Detection systems have seen significant technological advances, such as the development of increasingly complex on-board systems, especially machine vision systems. However, early robotic systems were reliant on laser distance scanners for interpreting the world around them, and many are still used, and more modern systems can use depth and color camera systems capable of relaying a significantly greater amount of information [27,28]. A laser scanner can provide a highly reactive and detailed image of the distance of an object from the robot, but it cannot be used to interpret human features, gestures, or emotions in the same way as an RGB-D camera. Furthermore, the increase of processing power allows for the detailed analysis of what a vision system is receiving. Machine learning has allowed for sophisticated algorithms to detect, track, and determine the pose a person is taking in real time. Early systems were struck with requiring constant calibration, a static environment, and markers for the person to be detected and track [29]. Not only were early systems unreliable but also costly. Recent additions, such as Microsoft’s Kinect, have made real-time, full-body tracking more feasible and have received attention for use in human–robot collaboration (HRC) [30,31,32,33].

Although the precollision strategies significantly reduce the physical harm, they do not fully mitigate the likelihood of an injury [34]. A person can still collide with inanimate objects, which means that the robot stopping when a person enters the workspace is not a guarantee of physical safety, and even with the robot’s speed and force being reduced the person can still come to harm as a result of their speed and force. As the maximum allowed speed for a robot during HRC is 250 mm/s, well below the speed a person can achieve, the robot’s ability to avoid a collision can be significantly influenced by the actions of the person it is avoiding. Furthermore, a robot that would either collide with or ceases working when a person enters the workspace is not ideal for HRC.

The strategies also do not consider how changes in the robot’s proximity or speed may affect the person’s psychological well-being. This is despite the growing research that shows there is a link between them, predominantly in HRI and social robotics (which is discussed in further detail in Section 3). By taking the factors mentioned earlier into consideration, one can argue that the current guidance in ISO/TS 15066:2016 can be improved to maintain the physical and psychological safety of the operator. HAN [12], a field of mobile robotics, does take both physical and mental safety into consideration. The lessons learned from studies into this recent field may provide a means to inform and improve the current guidelines.

3 Human aware navigation

To operate and be accepted in the same environment as people, a robot must not only be able to avoid collisions with them but also recognize and act accordingly to the social behavior of humans [35]. Path finding and avoidance algorithms that take this factor into account are now finding increased relevance in robotics [36]. Alami et al. [37] argued that for navigation to be considered human-aware, the robot should be able to convey in an understandable manner its current state, current goal, and imminent move. Since this definition in 2000, the criteria for HAN have become more sophisticated as the understanding of the relationships between a robot’s attributes and the person’s psychological well-being has improved. The improved understanding comes because of advancing technology, where it has not only increased the types of interactions people can have with robots but also the means of assessing a person’s reaction during the interaction.

Although a robot may not have anthropomorphic features, concepts such as personality and intent will still be applied by people onto the robot [38,39,40]. This understanding has led many of the social psychological concepts for human–human interaction to form the base for the psychological concepts in HRI; however, the direct nature of the application has been the source of debate [41]. Some of the social psychological concepts, such as trust and workload, have been previously researched in workplaces with increasing automation [42,43], which can aid in providing a foundation and comparison. In this sense, trust is determined as the ability of the machine to complete the task without harming those around it, which has become a key area of research with the development of self-driving vehicles [44]. Just as the passenger must be able to trust the vehicle to take them to their destination without incident, so must the operator be able to trust the robot they are collaborating with be able to complete the job without incident. As the operator gains more trust in the robot with the task, the efficiency of the HRC increases up to a point [45,46,47]. Therefore, it is essential that for HRC to become accepted by workers, a deeper understanding of how the robot can affect trust needs to be developed. A potential solution to this is to bring operators into the early design stages, with an extended study to determine how such factors scale over time. In experimental environments, the robots may perform differently than in the industrial environment. Therefore, expectations will be more accurately set for the workers and they will be more familiar with the robot’s capabilities and their role within the task. Furthermore, it sets more realistic expectations of what the robot can do, reducing the potential dissatisfaction when the robot is not as adaptable as initially perceived [48]. Takayama and Pantofaru [49] showed that participants would allow a robot to approach closer during initial interactions when they had at least 1 year or more experience. However, as the person gains more experience with a particular robot, this becomes less dependent on experience and more dependent on the robot being used as the task is completed [50]. Experience can also play a key role in the efficiency of the interaction; therefore, it is a key to introduce the human collaborators at the earliest possible stage. Although the human always having priority is generally considered an important aspect for social path planning [51], people unfamiliar with a robot and its capabilities will usually opt to give way [52]. Therefore, if operators gain experience early in the development stage for how the robot will react based on their reactions, it may alleviate this confusion. Furthermore, there is a lack of research into the long-term effects of HRI on psychological factors and how they change over this time.

Research into human factors in human–robot collaboration is still a relatively new field, but current studies are establishing relationships between robot attributes and a person’s psychological attributes. These relationships will be discussed in the following two sections. The first will focus on mobile robotics, which has received more focus on the psychological impacts of HRI due to the roles these robots are envisioned to have, for example, social robotics. The proceeding section will then focus on robot arms, which have received increased attention of late as advances in technology have allowed physical barriers to be removed.

3.1 Mobile robots

Even though mobile robots have been deployed for use in HRI since the late 1990s [10], and the psychological effects of robotic behavior were becoming prominent in the early 2000s [11], it was not until the mid-2000s that these were combined to design a path planner that would incorporate them both into what was coined as HAN. To achieve HAN, Sisbot et al. [12,53,54] set the following criteria:

• The motion must not result in physical harm to the person.

• The motion must be able to complete the task reliably and sufficiently.

• The motion considers the preferences and requirements of the person.

While both the first and second criteria are achievable with traditional path finding methods, the last criterion requires a more thorough understanding of how a robot’s motion can affect the person. Mateus et al. [55] further defined the last criteria by stating the goals to achieve this: comfort, respect for social rules, and naturalness. One of the attributes of a robot’s motion that has been considered is its proximity. Human–human interaction already has a well-established model for socially acceptable proximity, which can serve as a template for HRI. As described by Hall [56], proxemics provide the fundamental outline for a socially acceptable distance for people, which can be utilized as a reference for socially acceptable distances for robots. These socially acceptable distances can be used to designate comfort zones, with the closer the zone, the higher levels of stress the person would experience should a stranger enter it. However, there is no consensus on the most accurate social spacing model. The most prominent models for social spacing around an individual are shown in Figure 1. Each of these shapes can be applied depending on the context of the interaction, but they all suggest that the social sensitivity of the person decreases with increasing distance away from the person. The closest distance a person will allow a robot to approach is highly subjective and has been shown to be linked to personality traits [57], where the robot is “looking” [58], the size of the robot [11,59], and whether the person or the robot is approaching [60].

Figure 1

The model displays four social spacing shapes around a person, which dictate the different comfort zones: (a) concentric circles, (b) egg shaped, (c) elliptical, and (d) elliptical, which is skewed on the dominant side. Adapted from ref. [61].

Pandey and Alami [62,63] used such a model to develop an algorithm, leading the robot to avoid an elliptical space (compared with Figure 1), deemed too close for comfort, with socially acceptable zones based on one’s field of view. The robot’s path would then be planned using a combination of an A* algorithm and Voronoi diagrams. When compared to a static obstacle avoidance algorithm, which does not apply social distances, the robot was considered to have performed in a less uncomfortable manner. Sisbot et al. [12,53] developed a multilevel motion planner that generates a cost grid around a detected person in an environment. The associated costs of the grid are determined first by the physical distance to the person and then by the person’s perceived vision. The more effort required to see the robot on the path the higher the cost, with the highest costs being behind the person or behind an obstacle. Therefore, the robot plans to be as visible for as long as possible and only enters the social proxemic zone when necessary. The use of a cost grid allows for an existing algorithm (in this case A* algorithm) to be iterated on for navigating in a socially acceptable manner. Sun et al. [64] also identified the sudden appearance of a robot around a corner as socially unacceptable, generating a higher cost around corners and blind spots. Vega-Magro et al. [65] took an applied approach by generating cost maps around items in the environment based on the way a person would use the item, e.g., a trapezoidal area in front of a TV. When evaluated, either in simulation or real-life, all of these algorithms were capable of maintaining the set socially acceptable distances, even with multiple people included in the calculations.

Ferrer et al. [66,67] also applied a proxemics-based model when developing their mobile robots, Tibi and Dabo, which reduced the social work caused to a person as a result of the robot navigating a crowd. The model used was based on the social force model (SFM) developed by Helbing and Molnár [68], which was designed as a means to describe the self-organization of pedestrians in a crowd. Similar to APFs, the SFM generates repulsive forces around obstacles (in this case other pedestrians) and attractive forces toward the goal. Ferrer and Sanfeliu [69] iterated on their design further by adding the capability of predicting the person’s reaction to the robot’s possible actions and taking the course with the lowest social work impact, again reducing the stress further. Shiomi et al. [70] expanded on this model to develop a socially acceptable, human-like collision avoidance system for a robot moving among pedestrians. According to ref. [69], the model was first calibrated by the robot moving toward the person without collision avoidance to determine the socially acceptable distance instead of using proxemics. The system also operated on a collaborative avoidance basis, where both parties move to avoid the collision, as is the case in most human–human collision avoidance situations [52]. Surveys taken by participants reported that the robot with the updated model gave the perception of being safer, as well as the results showing the avoidance system performed objectively safer. Although SFM has been shown to perform well outdoors, it has been shown to perform less well indoors, where the repulsive vectors can result in the robot taking unnecessary detours to avoid collisions [71].

The aforementioned algorithms show a key limitation in the analysis of HAN, which is the metrics used for evaluation. Despite the algorithms being designed to improve the psychological well-being of the person, the psychological factors were not assessed. Instead, they rely on physical distance data and interpreted socially acceptable distances based on proxemics. Robust studies of the algorithms with multiple participants, where factors such as comfort, workload, and trust may prove beneficially in further understanding the variables to be added to existing algorithms to make the path planning “Human Aware.” While they show the promise of adapting existing algorithms to incorporate psychological factors, they only consider proximity. This is significant as studies have previously highlighted mobile robot attributes other than proximity, which contribute to a person’s psychological well-being. Predictable movement is where the movement is the same as the movement expected by the person [72]. Motion that is more predictable than human motion has been shown to be preferential when collaborating with a robot [73]. Another attribute that has been shown to influence a person’s comfort is the speed of the robot. Butler and Agah [11] used a Nomadic Scout II at varying speeds when approaching a person, after which the person was asked to complete a 5-point Likert scale survey ranging from Very Uncomfortable to Very Comfortable. The speeds that scored higher on this scale were between 0.25 and 0.4 m/s, but a significant change to uncomfortable was not reported until 1.0 m/s, while a decrease in comfort and an increase in frustration were suggested to be possible at speeds below 0.25 m/s. Sardar et al. [74] used multiple scales (Negative Attitude Towards Robots, Source Credibility, Perceived Human-Likeness, and Interpersonal Attraction Scale) as well as physiological measures to assess participants’ compensatory behaviors when a robot approaches them at two different speed settings. They found that at the higher speed, participants reacted with more “pleasant” facial expressions and that the robot was more trustworthy (which may be attributed to the greater noise generated by the robot at higher speeds, leading to increased awareness of the robot’s location). Despite the potential for relationships between a robot’s speed and a person’s psychological well-being to exist, the number of studies into this is quite limited. This may be due to the speed being limited in the perceived roles for the robot during HRI, whether by environment or the nature of the task. Furthermore, the studies have little cohesion as the metrics used are not consistent and tend to be tailor made to the experiment rather than using a universal method for measurement. The lack of consensus on psychological concepts is challenging to overcome due to the inherent subjective nature and has been problematic for workload for nearly 40 years [42].

From this review into HAN in mobile robotics, the path planner should utilize a model based on proxemics spacing. However, proxemics should not be considered the only attribute, which contributes to a path planner to make it “Human Aware.” A model incorporating an understanding of a person’s available field of view has also been shown to help improve the interaction [12]. The robot’s speed and predictability when interacting with a person are also important factors, which require further study [11,73]. It is of note, however, that mobile robots used in the aforementioned speed and proximity studies are smaller or equal to the height of an average person. This will have to be taken into consideration when assessing the application of HAN to industrial robot arms, as they are often larger with a higher payload capacity. As some of the studies have shown there to be a relationship between robot size and acceptable proximity, it is essential to understand the effects of proximity and speed for robots larger than a person. There is also a persistent lack of consensus in psychological analysis tools within robotics that should be addressed. Many of the studies within this review use different methods for assessing a participant’s “comfort,” without a formal or agreed upon definition. This leads to difficulty in comparing studies as the metric is consistently vague. Therefore, it would be of importance in this field for a formal definition of psychological concepts such as comfort and a standardized scale for assessment.

Considering these studies into mobile robotics, the following criteria can be a considering key for HAN:

• The robot must avoid collision with persons and obstacles during navigation.

• The movements of the robot must be predictable and smooth.

• The path planner should be informed by the psychological needs of the people it is intended to interact with.

Section 3.2 reviews the few studies that have included psychological safety factors when designing a path finding and obstacle avoidance algorithm on robot arms.

4 Conclusion

At the beginning of this review, a question was posed: How can the current Human–Robot Interaction safety guidance be optimized for maintaining the safety of the operator? The safety guidance set in the technical specifications of ISO/TS 15066:2016 presents methods of reducing the likelihood of physical injury as a result of a robot’s actions but, as highlighted in Section 1 of this article, that is considering only the potential physical impact in HRC. Therefore, they can be improved. To improve the quality of the human element of HRC, it is important to develop a better understanding of how the robot’s action (or inaction) can influence the operator. By failing to consider the psychological element of the interaction, which would be experienced by the operator during HRC, then the efficiency of the team can be reduced, as well as acceptance and job satisfaction. One of the primary solutions for this could be the inclusion of operators at the early design stages of a collaborative work cell. In doing this, the parameters for the robot can be set more appropriately for the task. These parameters can then be implemented in another potential solution: using the guidance that informs HAN to supplement existing guidance. Therefore, a second question was posed: How can the approaches for HAN in mobile robotics be applied to an industrial robot arms? The reviews in Sections 3.1 and 3.2 show the potential crossover between human aware planners in mobile robotics and in robot arms, with the following areas identified as readily transferrable:

• Interpretation of a person’s intent through machine vision and learning

• Robot motion based upon a person’s field of view

  However, other areas that have been identified but require further development include:

• Legible and predictable robot motion

• How do the robot’s speed and proximity affect the person’s comfort

• Link between the robot’s size and shape with person’s comfort

• The appropriate spacing model for an industrial robot arm

• Psychological attributes of a person which are influenced by a robot’s attributes

The transfer of skills is also limited by a lack of research into the different perceptions of safety that occur between a mobile robot and an industrial robot arm. Although the robot arm will have an extra dimension of movement, the base is fixed, and there is a limit to its reach, which is not possible in mobile robots. Although physical only safety operates on the principles that a person can be treated as an obstacle in the same way as any other object, HAN recognizes that the cognitive abilities of a person require special treatment. Although an obstacle will not be affected by the robot’s speed, size, proximity, or gaze, studies have established relationships between these elements and a person’s psychological well-being. Without considering the human aspect of HRC, there is a risk for the robots not to be fully accepted and the efficiency of the tasks to be reduced. Furthermore, although the results from studies into mobile robotics can inform approaches for HAN in robot arms, the differences between the two types of robotics should be acknowledged. Therefore, there is a need for studies to better understand and develop the relationships between a robot arm’s attributes and the relationships they have with a person’s psychology. A further developed understanding of these can allow for better evaluation of the algorithms with respect to the person’s psychological well-being. While proximity to the person can prove a useful and easily measurable metric, it cannot be considered the only one for determining whether a robot is “Human Aware.”

Despite the limitations of studies into HAN, it is clear that speed and proximity of a robot arm can affect a person’s comfort, trust, and workload. This can lead to an objective improvement when using HAN in HRC: the efficiency of both the human and the robot is increased. Even with this improved efficiency, HRC is not widespread in the industry. One of the main reasons for the relatively few occurrences of such interaction can be attributed to safety regulations being behind the advances in technology. Nevertheless, with further studies and research into HAN, which highlight the advantages mentioned in Section 3.2, as well as the significant improvements in robotics safety without the requirement of physical barriers, this is due for a change. A key limitation of studies into HAN, and Human Factors in robotics, is the lack of an agreed upon formal definition for many of the social psychological concepts. This is further hindered by the lack of a universal measurement tool for the concepts within HRC. However, there are some tools that are seeing more prominence in the measurement of increasing automation and may prove beneficial to the evaluation of HAN.

The future proposed by Asimov envisioned a successful shared working environment between robots and humans based on the understanding that the robot interprets the human as that: a human and not just a dynamic obstacle. There may be less vacuum tubes and mining stations on Mercury, but by furthering our knowledge of this key aspect in HRI, the acceptance of industrial robot arms in a shared workspace can be considered that much closer.

This article is an extended version of the published manuscript from ICRES 2019: M. Story, C. Jaksic, S. R. Fletcher, P. Webb, and J. Carberry, “Evaluating the use of HAN in Industrial Robot Arms” in ICRES 2019: International Conference on Robot Ethics and Standards, 2019, no. July, pp. 79–86.