Teleoperation of the industrial robot

Teleoperation of the Industrial Robot: Augmented reality application Chung Xue Er (Shamaine) Department of Computer & Software Engineering The Technological University of the Shannon: Midlands Midwest Athlone, Ireland xechung@research.ait.ie John Henry Robotics and Drives Mullingar Business Park, Mullingar, Ireland jhenry@rdservices.ie Yuansong Qiao Department of Computer & Software Engineering The Technological University of the Shannon: Midlands Midwest Athlone, Ireland ysqiao@research.ait.ie Ken McNevin Robotics and Drives Mullingar Business Park, Mullingar, Ireland ken@rdservices.ie Vladimir Kuts† Department of Computer & Software Engineering The Technological University of the Shannon: Midlands Midwest Athlone, Ireland vkuts@ait.ie Niall Murray Department of Computer & Software Engineering The Technological University of the Shannon: Midlands Midwest Athlone, Ireland nmurray@research.ait.ie Augmented reality application. In Proceedings of ACM MMSys conference (MMSys2022). ACM, Athlone, Ireland, 5 pages. ABSTRACT Industry 4.0 is aimed at the full manufacturing domain automatization and digitalization. Humans and robots working together are being discussed widely both in the academic and industrial sectors. As being discussed, there is a need for a more novel type of interaction method between humans and robots. This demonstrational paper is presenting the technical advancement and prototype of remote control and reprogramming of the Industrial robot. This development is safe and efficient and allows the operator to focus rather on the final task than the programming process. 1 Introduction The fourth industrial revolution often referred to as Industry 4.0, is a general term for a new industrial model [1]. Industry 4.0 aims at building cyber-physical production systems (CPPS) (which comprise intelligent, real-time-capable, networked sensors and actuators) that unite both digital and physical worlds to make manufacturing increasingly smart by utilising the internet of things (IoT)[1][2]. Industry 4.0 [3] includes a range of digital technologies that includes but is not limited to Augmented Reality (AR), Virtual Reality (VR), Digital Twins (DT), predictive maintenance, cloud computing, Internet of Things (IoT), Artificial Intelligence (AI), and big data. This stack is essential in achieving smart factories of the future (FoF). However, little research has been conducted to develop a usercentred human-robot interaction (HRI) in the context of industry 4.0 environment requirements. HRI is the study of how humans communicate with robotic systems. It informs us on how to best design, evaluate, understand and implement robotic systems that are competent enough for carrying out collaborative tasks with the human stakeholder [4][5]. One of the major challenges encountered CCS CONCEPTS • Human-centered computing • Human-computer interaction (HCI) • Interactive systems and tools • User interface management systems KEYWORDS Augmented Reality, Human-Robot Interaction, Industrial Robot ACM Reference format: Chung Xue Er Shamaine, Yuansong Qiao, Vladimir Kuts, John Henry, Ken McNevin and Niall Murray. 2022. Teleoperation of the Industrial Robot: Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. MMSys '22, June 14–17, 2022, Athlone, Ireland © 2022 Association for Computing Machinery. ACM ISBN 978-1-4503-9283-9/22/06…$15.00 https://doi.org/10.1145/3524273.3532901 299 ACM MMSys, June, 2022, Athlone, Ireland Chung Xue Er (Shamaine) et al. by HRI is that both humans and robots use extremely distinct methods of communication and data representation [6]. Current work in HRI seeks to discover a less training-intensive, more direct, safe, effective, natural and stress-free interaction [7]. Teleoperation or often defined as telerobotic [8] when controlling robotics systems, is commonly used in human-robot interaction (HRI) research. It is an approach that allows human operators to control machines or robotic systems from a distance. This approach has been intensively developed in the last decade. Teleoperated robots are mostly used in telesurgery [9], military search and rescue tasks [10], mining [11] as well as space exploration [12]. According to [13], teleoperated robots helps to reduce the risk and costs associated with human exposure to life-threatening conditions. The capabilities spectrum for telerobotic systems has been expanded from traditional 2D computer interfaces to immersive media-based [14] VR and AR controlled teleoperated robots and has attracted considerable attention in recent years [15][16]. AR provides more spatial awareness for the users if compared with the more immersive VR experience which may cause them to lose connection with the real-world surroundings. The novel approach of using Augmented Reality (AR) creates a new design space and an opportunity for facilitating more naturalistic human-robotic interaction for applications such as teleoperation in real-time [17]. The key inspiration is that AR has the potential of displaying visual objects directly (from the field of view of the robotic system) into the field of view of the user without being disconnected from the physical world. A clear communication orientation and information visualisation are needed to mediate robot teleoperation [18][19]. On account of the ability of AR to augment the operator view and visualise the robotic corresponding task space, this allows the operator to retrieve the robot’s sensor state information (temperature level, position, joint angle etc) via graphic overlays result in AR. This provides great potential as an interface for robotic teleoperation [20]. To the best of this demo paper authors knowledge, no studies have been performed to thoroughly analyse AR interfaces’ effectiveness and accuracy in more complex path following tasks. 2 Figure 1: Developed system architecture On the right is our physical ABB robot arm connected to the ROS server over an ethernet cable. The integration of ROS and Unity 3D is enabled via ROS-Sharp, through a WebSocket connection on port 9090. All development tools used in this project are summarized in Table 1. On the HoloLens 2 client-side, the Mixed Reality Toolkit (MRTK) library [21] was imported to control the user hand gestures on a HoloLens 2. The Final IK asset attached to the holographic robot arm consisted of a powerful Inverse Kinematics solution (FABRIK algorithm is chosen) for advanced robot animation. Vuforia Image Target is used to correctly overlay the holographic robot arm on top of the real robot arm. On the server-side, two virtual switches were created and connected to the 31 Hyper V machine at the same time. A file server will be the first to launch opening port 9090, waiting for the HoloLens Client to connect over the mobile Wi-Fi hotspot. Secondly, both MoveIT and Rviz are launch together. MoveIT is mainly for Motion Planning. Basically, it creates a sequence of movements that all joints have to perform in order for the end effector to reach the desired position. While Rviz is for visualization purposes. On the ABB client-side, a WAN port is chosen to connect the ABB controller over the ROS server. Table 1: Tools used for the solution development System Architecture and Technology Stack The proposed HRI system architecture is presented in Fig.1. On the left is a Microsoft HoloLens 2 user interface with Unity 2019.4.21f1 (64-bit) game engine. The Microsoft HoloLens 2 AR HMD is employed because of its built-in localization and mapping (SLAM) abilities in addition to its self-supporting nature. It also currently represents state of the art in AR HMD technology. In the Centre, is a ROS system running on a virtual Ubuntu16.04 LTS ROS Kinetic virtual machine. The integration of ROS and Unity 3D is enabled by implementing the open-source ROS-Sharp library [22]. A WebSocket connection is established on port 9090 between Unity and the ROS arm simulator. It allows ROS-Sharp to serialize a topic. A pose stamped publisher script in Unity will create a topic. The topic is activated 300 Teleoperation of the Industrial Robot: Augmented reality application ACM MMSys, June, 2022, Athlone, Ireland each time when Unity tries to send end-effector pose information as JSON [23] to ROS. Within the script, the Unity end effector Euler rotation x, y and z will automatically be converted into quaternion x, y, z and w before sending. In this case, Unity will act as a publisher, while the ROS simulator is the subscriber. Unity converts C# data structures into JSON messages, and the pose data is passed across the bridge. The rosbridge library which contains the core rosbridge package then retrieves the pose data through the same topic by converting the JSON message back into geometry messages. 2.1 ROS is a versatile, multilingual middleware. It provides a collection of tools and libraries for robotic developers to simplify their effort of designing complicated and robust robot actions [26]. ROS is open-source and is language agnostic. It has become the common automation language for robotics and research focus in recent years. It is widely used in various robotic applications. Gazebo, as shown in Fig. 3(a). serves as a robotic simulator while ROS is the interface to the robot. The key benefit of using Gazebo is its ready integration with ROS. Similar to ROS, Gazebo is also open source. It is one of the most popular robotic simulators [68]. RViz, as shown in Fig. 3(b), is not a simulator but a 3D visualisation tool for ROS applications. It offers real-time sensor information from robot sensors and a view of the robot model. It can be used to display 2D and 3D point clouds as well as camera or lasers data [27]. Client-side stack The Client-Side stack mainly responsible for handling the user’s request and interaction. This section covers the client-side technology such as the Unity game engine, the Mixed Reality Toolkit (MRTK) and finally, the Vuforia SDK used to build the application on a HoloLens 2. The Unity3D game engine is deployed to create the AR application. A suite of development tools is provided to create different experiences, and in this case, the Mixed Reality Toolkit (MRTK) [24] was used. MRTK is a Microsoft- motivated open-source development kit that offers a set of components and features used to accelerate application development for the Microsoft HoloLens 2 [25]. The MRTK ineditor hand input simulation allows the developer to test a scene in the Unity game view. The MRTK supports rapid AR application development. MRTK automatically sets up a default scene for a new project when the “Add to Scene and Configure” is selected. The HoloLens 2 implemented the style buttons provided by the MRTK example package to create a set of tag-along buttons as shown on figure 2. A Radial View script is attached to the parent button. This solver script allows the buttons to follow the user's head movement. Figure 3. (a)ABB robot arm in Gazebo simulator (b) ABB robot arm in RViz with transform frames. 3 Experiment This chapter is a description of the measures taken to evaluate the effectiveness of AR application, those will be applied in the future steps within the user test experiments. The main design of the experiment as illustrated in Fig. 4 consists of a computer running Ubuntu16, HMD Microsoft HoloLens 2, and an ABB robot arm version IRB 1200-5 /0.9. We will place an image target marker on a table 15cm in front of the physical robot arm. The data collection flowchart of the RoSTAR system is shown in Fig. 4-2. The evaluation of the proposed RoSTAR system was achieved by running two experiments. It is the same person that performs the testing sequences throughout the first and second experiments. A series of objective measures for the ABB robot arm was conducted to test the robot path accuracy, end-effector’s position accuracy and path completion time. 1. Path Accuracy: Path accuracy was calculated as the number of correct trajectories executed by the robot divided by the total number of trials (10 trials). We treated the robot arm movement as timeout when the robot arm does not move as predicted/ spinning 180 degrees along the pre-specified trajectories Figure2: (a) Virtual Tag-Along Buttons Created with MRTK Examples package (b) Draggable 3D waypoint with MRTK articulated hand tracking input. 2.1 Server-side stack The Server-Side stack is mainly responsible for receiving the user’s request and feedback from the Client-Side. This section covers the server-side technology such as the Robot Operating System (ROS), MoveIT, ROSBridge and finally, the Robot Model used to teleoperate the physical ABB robot arm. 301 ACM MMSys, June, 2022, Athlone, Ireland 2. 3. Chung Xue Er (Shamaine) et al. 5 (Singularities). The path accuracy results will be plotted in the form of graphs. Absolute Position Accuracy: The position measurements across four different complexities are stored only after the end-effector arrived at each pose (irrespective of the path taken by the robot to reach the goal position). The robot arm will execute the same trajectories 10 times after receiving the pose data. Average Path Completion Time: Each trial began when the robot moves from start to goal position. The time measure was calculated as the average time of the interaction across all 10 trials. Conclusion In this technical demonstration overview, we present a novel opensource HRI system based on the Robot Operating System (ROS) and Augmented Reality (AR). It is demonstrated that the HoloLens 2 is capable of visualizing the robot movement and safety zone within a cutting-edge AR device. We propose a prototype application that successfully displayed key parameters of userdefined robot trajectory within the Microsoft HoloLens 2. As a result, it is possible to control and re-program robot in real-time from remote distances. The safety aspect is also considered, as the safety gate near the robot is always prioritized. REFERENCES [1] A. C. Pereira and F. Romero, ‘A review of the meanings and the implications of the Industry 4.0 concept’, Procedia Manuf., vol. 13, pp. 1206–1214, Jan. 2017, doi: 10.1016/j.promfg.2017.09.032. [2] D. Kerpen, M. Löhrer, M. Saggiomo, M. Kemper, J. Lemm, and Y.-S. Gloy, ‘Effects of cyber-physical production systems on human factors in a weaving mill: Implementation of digital working environments based on augmented reality’, in 2016 IEEE International Conference on Industrial Technology (ICIT), Mar. 2016, pp. 2094–2098. doi: 10.1109/ICIT.2016.7475092. [3] J. Egger and T. Masood, ‘Augmented reality in support of intelligent manufacturing – A systematic literature review’, Comput. Ind. Eng., vol. 140, p. 106195, Feb. 2020, doi: 10.1016/j.cie.2019.106195. [4] D. Feil-Seifer and M. Mataric, ‘Human-Robot Interaction’, in Encyclopedia of Complexity and Systems Science, 2009, pp. 4643–4659. doi: 10.1007/978-0387-30440-3_274.[5] M. Goodrich and A. Schultz, ‘Human-Robot Interaction: A Survey’, Found. Trends Hum.-Comput. Interact., vol. 1, pp. 203–275, Jan. 2007, doi: 10.1561/1100000005. [6] T. Williams, D. Szafir, T. Chakraborti, and E. Phillips, ‘Virtual, Augmented, and Mixed Reality for Human-Robot Interaction (VAM-HRI)’, in 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Mar. 2019, pp. 671–672. doi: 10.1109/HRI.2019.8673207. [7] L. H. Manring et al., ‘Augmented Reality for Interactive Robot Control’, 2020. doi: 10.1007/978-3-030-12243-0_2.[8] G. Niemeyer, C. Preusche, and G. Hirzinger, ‘Telerobotics’, in Automatica, vol. 25, 2008, pp. 741–757. doi: 10.1007/978-3540-30301-5_32. [9] T. Haidegger, J. Sándor, and Z. Benyó, ‘Surgery in space: the future of robotic telesurgery’, Surg. Endosc., vol. 25, no. 3, pp. 681–690, Mar. 2011, doi: 10.1007/s00464-010-1243-3. [10] T. Kot and P. Novák, ‘Application of virtual reality in teleoperation of the military mobile robotic system TAROS’, 2018, doi: 10.1177/1729881417751545. [11] D. W. Hainsworth, ‘Teleoperation User Interfaces for Mining Robotics’, Auton. Robots, vol. 11, pp. 19–28, Jul. 2001, doi: 10.1023/A:1011299910904. [12] G. Liu, X. Geng, L. Liu, and Y. Wang, ‘Haptic based teleoperation with masterslave motion mapping and haptic rendering for space exploration’, Chin. J. Aeronaut., vol. 32, no. 3, pp. 723–736, Mar. 2019, doi: 10.1016/j.cja.2018.07.009. [13] N. Small, K. Lee, and G. Mann, ‘An assigned responsibility system for robotic teleoperation control’, Int. J. Intell. Robot. Appl., vol. 2, no. 1, pp. 81–97, Mar. 2018, doi: 10.1007/s41315-018-0043-0.58 [14] A. Perkis et al., QUALINET White Paper on Definitions of Immersive Media Experience (IMEx). 2020. [15] D. Krupke, F. Steinicke, P. Lubos, Y. Jonetzko, M. Görner, and J. Zhang, Comparison of Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction. 2018. doi: 10.1109/IROS.2018.8594043. [16] L. Peppoloni, F. Brizzi, C. A. Avizzano, and E. Ruffaldi, ‘Immersive ROSintegrated framework for robot teleoperation’, in 2015 IEEE Symposium on 3D User Interfaces (3DUI), Mar. 2015, pp. 177–178. doi: 10.1109/3DUI.2015.7131758. [17] E. Dima et al., ‘View Position Impact on QoE in an Immersive Telepresence System for Remote Operation’, in 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), Jun. 2019, pp. 1–3. doi: 10.1109/QoMEX.2019.8743147. [18] S. Kortekamp, S. Werning, I. Ingmar, and O. Thomas, THE FUTURE OF DIGITAL WORK - USE CASES FOR AUGMENTED REALITY GLASSES. 2019. [19] A. Monferrer and D. Bonyuet, Cooperative robot teleoperation through virtual reality interfaces. 2002, p. 248. doi: 10.1109/IV.2002.1028783. Figure 4: Experiment setup The performance of the HoloLens 2 in the task was evaluated by monitoring the frame rate at four different stages mainly: Active Tracking Stage, Fixed Position Stage, Trajectory Execution Stage, End Trajectory Stage. The objective measures for the HoloLens 2 are frame rate and path completion time. 1. Frame Rate: The frame rate is calculated by averaging the total frame rates received every second at each stage. The HoloLens 2 frame rate are collected every second at the start up and classified into four main stages: The tracking stage, planning stage, execution stage and reset the stage. 2. Average Path Completion Time: Each trial began when the virtual robot arm moves from start to goal position. The time measure was calculated as the average time of the interaction across all 10 trials. The average path completion time across 4 levels of trajectory complexity for both HoloLens 2 virtual robot arm and real ABB robot arm based on the number of waypoints are compared via a 3D-clustered column chart. A shorter path completion time indicates fast inverse kinematics solve times. 4 Demonstration For the demonstration in the ACM MMSys2022, the visitors will be able to program and control in real-time the ABB industrial robot remotely and follow in the PC monitor how the physical robot is being moved in the university (TUS: Athlone campus). The aim of the demonstration is to show a safe and efficient method for industrial hardware teleoperation from remote distances. 302 Teleoperation of the Industrial Robot: Augmented reality application ACM MMSys, June, 2022, Athlone, Ireland [20] S. Green, M. Billinghurst, X. Chen, and J. Chase, ‘Human Robot Collaboration: An Augmented Reality Approach A Literature Review And Analysis’, Jan. 2007, doi: 10.1115/DETC2007-34227 [21] polar-kev, ‘MRTK-Unity Developer Documentation - Mixed Reality Toolkit’. https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/ (accessed Sep. 22, 2021). [22] ‘Home · siemens/ros-sharp Wiki’, GitHub. https://github.com/siemens/ros-sharp (accessed Sep. 22, 2021) [23] ‘JSON’. https://www.json.org/json-en.html (accessed Sep. 12, 2020) [24] polar-kev, ‘MRTK-Unity Developer Documentation - Mixed Reality Toolkit’. https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/ (accessed Sep. 22, 2021). [25] ‘HoloLens 2—Pricing and Options | Microsoft HoloLens’. https://www.microsoft.com/en-us/hololens/buy (accessed Sep. 02, 2019). [26] ‘ROS.org | About ROS’. https://www.ros.org/about-ros/ (accessed May. 21, 2020). [27] ‘rviz - AWS RoboMaker’. https://docs.aws.amazon.com/robomaker/latest/dg/simulation-tools-rviz.html (accessed Aug. 22, 2019). 303