Path and task planning for multiple mobile robots in smart warehouse

MARMARA UNIVERSITY INSTITUTE FOR GRADUATE STUDIES IN PURE AND APPLIED SCIENCES PATH AND TASK PLANNING FOR MULTIPLE MOBILE ROBOTS IN SMART WAREHOUSE ALİ BOLU M AS TER THESIS Department of Computer Engineering Thesis Supervisor Dr.Öğr.Üyesi ÖMER KORÇAK ISTANBUL, 2021 MARMARA UNIVERSITY INSTITUTE FOR GRADUATE STUDIES IN PURE AND APPLIED SCIENCES PATH AND TASK PLANNING FOR MULTIPLE MOBILE ROBOTS IN SMART WAREHOUSE ALİ BOLU (524116047) M AS TER THESIS Department of Computer Engineering Thesis Supervisor Dr.Öğr.Üyesi ÖMER KORÇAK ISTANBUL, 2021 ACKNOWLEDGEMENTS I would like to thank with my special appreciation and deepest gratitude to my supervisor Asst. Prof. Ömer Korçak for his support in my Master thesis. I am grateful to him for guiding me with his experiences. I also would like to thank my company for encouraging me to work on such a great project. Finally, I would like to thank my parents for their continuous support and understanding. January, 2021 Ali Bolu i TABLE OF CONTENTS ACKNOWLEDGEMENTS .......................................................................... TABLE OF CONTENTS ............................................................................. ÖZET ...................................................................................................... ABSTRACT.............................................................................................. ABBREVIATIONS..................................................................................... LIST OF FIGURES .................................................................................... LIST OF TABLES...................................................................................... 1. INTRODUCTION ................................................................................. 1.1. Contributions of the Thesis .............................................................. 1.2. Outline of the Thesis ...................................................................... 2. ROBOTIC MOBILE FULFILLMENT SYSTEM (RMFS) ENVIRONMENT .... 2.1. System Model............................................................................... 2.2. Communication System Design ........................................................ 2.2.1. Literature Review................................................................ 2.2.2. Communication Requirements ............................................... 2.2.2.1. Low Latency ......................................................... 2.2.2.2. High Reliability ..................................................... 2.2.2.3. Full Coverage........................................................ 2.2.3. Communication System Architecture....................................... 2.2.4. Communication Design Decision............................................ 2.2.5. Wi-Fi Network Suggestions................................................... 3. PATH PLANNING FOR ROBOTS ........................................................... 3.1. Related Work................................................................................ 3.2. Path Planning Problem ................................................................... 3.3. Warehouse Design Decisions ........................................................... 3.4. Solution Approaches ...................................................................... 3.4.1. Collision-free Path Planning Algorithm.................................... 3.4.2. Warehouse Design Decisions ................................................. 3.5. Experimental Study ....................................................................... 3.5.1. Number of Robots ............................................................... 3.5.2. Effect of the Traffic Rules ..................................................... 3.5.3. Pod Alignment ................................................................... 4. TASK PLANNING FOR ROBOTS .......................................................... 4.1. Related Work................................................................................ 4.1.1. Main Contributions.............................................................. 4.2. Model and Methodology ................................................................. 4.2.0.1. Order-to-Task Conversion ........................................ 4.2.1. Order Task Selection Method................................................. 4.2.2. Order Task Execution........................................................... 4.2.3. Replenishment Task............................................................. 4.2.4. Robot Life-cycle ................................................................. ii i ii iv vi viii ix x 1 3 4 5 5 7 7 9 9 10 10 10 12 14 17 17 18 20 21 21 24 25 26 26 28 31 31 33 34 34 37 40 42 45 4.3. Simulation Study and Numerical Results ............................................ 4.3.1. Simulation Environment ....................................................... 4.3.2. Numerical Results ............................................................... 4.3.2.1. Order Batch Size.................................................... 4.3.2.2. Robot Count ......................................................... 4.3.2.3. Robots’ Charging and Workload Balance .................... 4.3.2.4. Order Count.......................................................... 4.3.2.5. SKU Effects on Pile-on and Time .............................. 4.3.2.6. Effect of Stock-Waiting Orders ................................. 5. CONCLUSION AND FUTURE WORK .................................................... REFERENCES .......................................................................................... iii 47 47 49 50 51 53 54 54 56 58 60 ÖZET AKILLI DEPODA ÇOKLU ROBOTLAR IÇIN YOL VE GÖREV PLANLAMA Depo yönetiminde, insanlardan daha fazla iş yapabilen, zaman ve enerji açısından verimliliği artırabilen otonom robotların kullanılması çok avantajlıdır. Dahası, manuel işlemler hatalara açıktır bu da büyük kayıplara neden olabilir. Bu nedenle, akıllı depo yönetimi araçlarının kullanımı, mevcut işletmeler için önemli bir ihtiyaç haline gelmiştir. Hareketli robotların çalıştığı akıllı depolardaki operasyonların yürütülmesinde ürünlerin depolanması, yol planlaması, görev planlaması ve kaynak tahsisi önemli problemlerdir. Zaman ve enerjiden tasarruf etmek için bu problemler en optimal şekilde çözülmelidir, ancak sistemin dinamikleri nedeniyle bu çok zordur. Robot, depolama alanı, istasyon ve sepetler gibi sınırlı kaynakların verimli bir şekilde yönetilmesi gerekmektedir. Bu çalışmada, çok robotlu depo sistemlerinde yol ve görev planlama problemine odaklandık. Robotların çakışmadan en optimal şekilde hareket etmesi ve görevleri sorunsuz bir şekilde tamamlaması karmaşık bir problemdir ve genel geçer optimal bir çözümü yoktur. Çünkü depoya özgü parametrelerin yanı sıra robotların hareketi ile ortam parametreleri sürekli değişmektedir. Tez çalışmasında ilk aşamada; yol planlaması için önerdiğimiz depo karakteristiğine ve çoklu robotlara uygun değiştirilmiş A* algoritmasının uyguladık ve farklı senaryolarda sonuçları analiz ettik. Tüm testlerimizi algoritmalarımızı uyguladığımız ve kendi gerçekçi simulasyon ortamını oluşturduğumuz Depo Yürütme Sistemi yazılımımız ile gerçekleştirdik. Farklı depo tasarımlarında ve robot sayılarında algoritmamızın nasıl sonuçlar verdiğini inceledik. İkinci aşamada bulduğumuz yol planlama algoritmasını kullanarak görev planlama algoritmamızı geliştirdik. Görev planlama algoritmamızda en önemli unsur olan ihtiyaç duyulacak robot sayısını azaltmaya yönelik optimizasyon etkisini analiz ettik. Siparişleri tekil olarak robot için sipariş görevine çevirmek yerine birden çok sayıda siparişi tek siparişmiş gibi alarak robotların ürün toplama verimini artıran algoritmamızı sunduk. Sonrasında, oluşturduğumuz bu robot sipariş görevlerinin öncelik sıralamasını hesaplayıp en optimal görevi robota atayan görev seçim metodumuzu önerdik. Önerdiğimiz iv algoritmaları simülasyon yazılım ortamımızda farklı senaryolar altında test ettik. Test sonuçları neticesinde görev planlama algoritmamızın ürün çeşitliliğinin çok olduğu depolarda bile iyi sonuç verdiğini ispat etmiş olduk. Bu sayede robotlar daha kısa sürede daha çok ürün toplayabildi. Testlerimizi farklı sipariş, robot, sepet ve ürün çeşitliliğinde test ettik. Ve algoritmamızın farklı senaryo ve tasarımlarda kararlı çalıştığını gösterdik. Ocak, 2021 Ali Bolu v ABSTRACT PATH AND TASK PLANNING FOR MULTIPLE MOBILE ROBOTS IN SMART WAREHOUSE Using autonomous mobile robots is now a necessity for today’s large e-commerce warehouses to save time and energy, and to prevent human-based errors. Robotic Mobile Fulfillment System (RMFS) controls these robots as well as all other resources and tasks in a warehouse. In warehouse management, there are many problems regarding the optimization of warehouse processes that are related to the storage of products, path planning, task planning, and resource allocation. These problems should be solved in most optimal way in order to save time and energy, but this is very challenging due to the dynamics of the system. Limited resources such as robots, stations, totes and item spaces should be managed efficiently after tracking their status continuously. In this study, we will focus on the path and task-planning problem in multi-robot warehouse systems. The path-planning problem is especially challenging, since finding best and collision-free paths are affected by many warehouse specific parameters as well as dynamically changing environment due to movement of the robots in the system. We applied our modified A* collision-free path planning for multiple mobile robots in the warehouse. We did all of the tests with a realistic simulation environment in our Warehouse Execution System (WES) that we implement our algorithms. We analyzed our algorithm under different robot count and warehouse design decisions. In the second stage of thesis, we implement our task planning algorithms with using our modified A* path planning algorithm. We analyzed the optimization effect of our algorithms on decreasing robot count under different scenario and design decisions with lots of test in our simulation environment. We describe a novel task conversion algorithm which generates tasks from a batch of orders and provides high pile-on value. Then we propose an adaptive heuristic approach to assign generated tasks to robots, considering system dynamics such as location of robots and pods, utilization of totes and age of the tasks. To evaluate the proposed algorithms, we perform extensive set of simulations in a highly realistic environment including robot charging, replenishment process and path planning algorithms. We show that the proposed task planning approach significantly vi reduces order completion time even for high number of stock keeping units (SKU). It also provides balanced workload among robots. We analyze optimal value of order batch size and effect of important system parameters such as robot count, order count and SKU. The obtained results shade light on how to design a smart warehouse system with high efficiency. January, 2021 Ali Bolu vii ABBREVIATIONS AHA : Adaptive Highways Algorithm AMR : Autonomous Mobile Robots ARTS : Adaptive Robotic Task Selection FHA : Fixed Highways Algorithm FIFO : First In First Out OBC : Order Batch Count OBRTC : Order Batch to Robot Task Conversion ODT : Order Departing Time OTC : Order to Task Conversion POA : Pick Order Assignment PPS : Pick Pod Selection PSA : Pod Storage Assignment RBA : Reservation Based Algorithm RMS : Robot Management System ROA : Replenishment Order Assignment RMFS : Robotic Mobile Fulfillment Systems RPS : Replenishment Pod Selection SKU : Stock Keeping Units WES : Warehouse Execution System WMS : Warehouse Management System viii LIST OF FIGURES Figure 2.1 - Warehouse Execution Area. ......................................................... Figure 2.2 - Pick Station [10]. ....................................................................... Figure 2.3 - Warehouse Traffic Rule. .............................................................. Figure 2.4 - WES Model.............................................................................. Figure 2.5 - WES Communication Architecture. ............................................... Figure 2.6 - WLAN Topology of WES AMRs .................................................. Figure 3.1 - Cell Reservation of Robots .......................................................... Figure 3.2 - Head-to-head Encounter.............................................................. Figure 3.3 - An Example No-path Scenario...................................................... Figure 3.4 - Flowchart of collision-free path planning algorithm .......................... Figure 3.5 - Horizontal 2x6 Pod alignment ...................................................... Figure 3.6 - Order completion time vs robot count ............................................ Figure 3.7 - Head-to-head encounter vs robot count........................................... Figure 3.8 - Effect of the traffic rules .............................................................. Figure 4.1 - Order Task Process..................................................................... Figure 4.2 - Tote Reservation Scenario............................................................ Figure 4.3 - Order Task Execution Flowchart. .................................................. Figure 4.4 - Replenishment Task Process......................................................... Figure 4.5 - Replenishment Task Execution Flowchart. ...................................... Figure 4.6 - Replenishment station [10]. ......................................................... Figure 4.7 - Robot Life-cycle........................................................................ Figure 4.8 - The physical robot (Robee of Makhina Robotics [10]) used in 6 7 8 8 11 15 19 20 20 23 25 27 27 28 35 39 41 42 43 43 46 parameter setting................................................................................. Figure 4.9 - Order Batch Size (nob ) vs Orders Completion Time .......................... Figure 4.10 -Order Batch Size (nob ) vs Average Order Departure Time .................. Figure 4.11 -Robot Count vs Orders Completion Time ....................................... Figure 4.12 -Robot Count vs Pile-on .............................................................. 48 51 52 53 54 ix LIST OF TABLES Table 2.1 Table 2.2 Table 3.1 Table 3.2 Table 4.1 Table 4.2 Table 4.3 Table 4.4 - Robot Commands / Requests ......................................................... WES Communication Analyze / Command Count.............................. Test Results for Various Pod Alignments .......................................... Efficiency Results for Various Pod Alignments .................................. Robots’ Charging And Workload Balance ........................................ Effect of Order Count .................................................................. Effect of SKU ........................................................................... Effect of Stock Waiting Orders ...................................................... x 13 14 29 30 55 55 56 56 1. INTRODUCTION The Robotic Mobile Fulfillment System (RMFS) is a new trend warehouse system that employs autonomous mobile robots (AMR). Today, many e-commerce warehouses use this system, while many more are investigating for future use. The RMFS-based warehouses in which robots lift pods and bring to pick stations were first introduced by KIVA Systems, Wurman et al. [1]. Amazon bought Kiva systems in 2012 and renamed it to Amazon Robotics [2]. Since then, many companies have entered the market with their robots such as Swisslog CarryPick, GreyOrange Butler, Fetch Robotics Freight (and Fetch), Scallog System, Hitachi Racrew, etc. [3]. Moreover, major retailers, such as Amazon and Alibaba, use RMFS [2]. However, investment in RMFS to warehouses will cost more than a million-dollar generally [4]. The most expensive part of the RMFS investment is the robots. Hence, optimization on RMFS to complete more order with less robots is crucial point of efficiency. Warehouse operations contain four main processes: (i) Receiving new stock/product, (ii) Placing products to the warehouse (pod storage), (iii) Carrying pods to pick station in case of a new order, (iv) Picking and packing an order. In the modern smart warehouse, most of these processes are performed by robots. Only last process has some human intervention. There are many problems regarding the optimization of these processes that are related to storage of the products, path planning, task planning, and resource allocation. These problems should be solved in most optimal way in order to save time and energy, but this is very challenging due to the dynamics of the system. In this study, we first focused on path planning problem in multi-robot warehouse systems. This problem is especially challenging, since finding best and collision-free paths are affected by many warehouse specific parameters as well as dynamically changing environment due to movement of the robots in the system. Finding optimal solution of multi-agent path finding problem is proven to be NP-Hard even in its simple form [5]. The second part of the thesis, we focused on task planning to optimize warehouse process. The order picking process is the most important part in a warehouse system. 1 Traditional warehouses use picker-to-part method, i.e. workers travel in the warehouse and collect order lines. The new RMFS approach is part-to-picker, such that robots carry the mobile shelves (named as pod) that contain items to the workers who are waiting at the pick stations. RMFS increase picking rate significantly compared to traditional methods especially for large e-commerce warehouses with many SKUs (Stock Keeping Units) [6]. It is crucial to optimize picking order, i.e. to collect more order items with less effort. The order batch method is used for picker-to-parts picking methods. In this method, a set of orders are splitted into several subsets of order items and assigned to pickers in an efficient way in order to reduce travel and picking time [7]. However, in an RMFS, it is not feasible for robots to carry all items in a single pod. Therefore, the optimization objective is to collect more items per pod at the pick station. The average number of items picked from one pod, named as pile-on, is the most important metric to reduce order picking time [6, 8]. Another important RMFS task is the replenishment to pods, in other words to place stocks in the warehouse. There are two types of stations; picking and replenishment stations. Human workers work in these stations for collecting orders (picking) or placing items to the pods (replenishment). In all tasks, the pods are carried to and from the station by robots. The picking process consumes more energy and time than replenishment, it is up to 80 % according to [6]. It is obvious that AMRs need energy and charging to complete their tasks. When a robot completes the assigned job and ready to another one, RMFS will decide to assign one of the three tasks: order picking, replenishment, or charging. In this study, we mainly focus on order picking optimization, while we also consider and do not neglect replenishment and charging tasks. We propose novel order-to-task conversion and task selection methods aiming to increase pile-on and total efficiency of RMFS by considering order throughput, time priority, and utilization of resources. We also test our methods and algorithm with different warehouse parameters such as robots, pick stations, totes, SKU, time, and size of the warehouse. Management of RMFS requires knowledge of all existing and moving objects in 2 the warehouse. For this purpose, we use a web-based central software called Warehouse Execution System (WES) to manage warehouse automation. WES manages all resources such as robots, stations, totes, pods etc. We simulate the RMFS on WES in almost real environment to analyze our approaches. The details of WES and its simulation parameters will be explained in the Simulation section. 1.1. Contributions of the Thesis In this thesis, we mainly focus on challenging problem of RMFS based warehouse such as collision free path planning and task planning. We introduced modified A* algorithm that considers turning costs of robots and grid-base warehouse path environment characteristics. We give details of the modified collision-free A* algorithm and its implementation. In this study we propose a novel Order to Task Conversion (OTC) algorithm which tries to find minimum number of pods that includes items of multiple orders. In other words, contrary to existing studies, the proposed OTC algorithm handles a number of orders as a single order instead of handling orders one by one. We calculates priority of all tasks with our mathematical model according to distance to robot, time and totes usage. • We will give details of the modified collision-free A* algorithm and its implementation. We will analyze no-path and deadlock scenarios and propose a solution to them. • We determine no-path scenarios for path planning algorithm that are not mentioned in the previous studies and propose solution to them. • The effect of the warehouse design in path planning is analyzed. Task completion time and head-to-head encounter count (re-planning) is also analyzed according to robot count and warehouse design decisions. • We provide a task planning approach with high pile-on value even when SKU is high and replenishment is not optimized. Optimization of replenishment need extra efforts such as equipment and process delay. • We propose a dynamic and parametric heuristic model for order task selection process. New task selection is performed according to a novel mathematical priority 3 model which depends on various criteria such as distance to robot, totes usage and time. • Contrary to the most of the studies in the literature, we develop and use a fully realistic simulation environment that includes robot collisions, waiting time on path and stations, and effects of the loaded pod weight. Moreover, our simulation includes charging process by considering a realistic energy consumption model (depending on movements and loads) and real charging times. • Some e-commerce companies may sell items even if they are not already stored in their warehouses and these items are replenished on demand. Therefore there could be many already ordered items on the pods loaded at the replenishment stations. Management of this case is mostly not addressed in existing studies. In our study, we also test such a scenario and analyze its effects on the system performance. 1.2. Outline of the Thesis The rest of the thesis is organized as follows. In Chapter 2, we define the Robotic Mobile Fulfillment System (RMFS) environment. We describe the properties and components of such a smart warehouse to understand its problems. In Chapter 3 we analyze the path planning problem of RMFS based warehouse. Moreover, we offer our solution for collision-free path planning for multiple robots in RMFS. In Chapter 4, we analyze the task planning of RMFS based warehouses. The most important part of the optimization is reducing the number of robot count. Because the robot is the most expensive component of the RMFS. Hence, we offer our Order Batch to Robot Task Conversion (OBRTC) algorithm and order task selection methods to optimize RMFS. We define Order tasks, replenishment tasks, charging needs, and the well-design robot life-cycle. Finally, we conclude our thesis and presents future works in chapter 5. 4 2. ROBOTIC MOBILE FULFILLMENT SYSTEM (RMFS) ENVIRONMENT In this chapter we describe RMFS based warehouse environment. We use a web-based central software called Warehouse Execution System (WES) to manage warehouse automation. WES should communicate with other components such as robots and human workers. The communication system is also explained in this chapter. 2.1. System Model Figure 2.1 illustrates an RMFS based warehouse system. The components of this system are the following: • Pods: All the warehouse items are stored in the pods. When an order or replenishment arrives to the system, it should be converted to the task by assigning appropriate pod or pods. These pods that are reserved by the system are shown in yellow color in Figure 2.1 (such as 7). When a pod is lifted by a robot, its cell becomes free (5 in Figure 2.1). • Robots: Robots carry pods to the stations for order collection or replenishment (6 in Figure 2.1). • Pick station (2 in Figure 2.1): There can be multiple pick stations in a warehouse, where human workers collect the order items carried by the robots. Figure 2.2 shows a pick station. There exists limited number of totes, and each tote is assigned for a single order. When an order is collected, that tote is replaced with an empty one. • Replenishment station (4 in Figure 2.1): Human workers place replenishment items to the shelves of pods that are carried by the robots. In a the warehouse, human workers work only in pick and replenishment stations, and the rest is no-human zone. • Turning cells (3 in Figure 2.1): A pod may have multiple faces and an item can be reached only through a single face [9]. There are two types of pods according to the number of faces: two-directional and four-directional (pods that can be seen in Figure 2.2 are two-directional). Robots can rotate pods only on turning cells. • Charging cells (1 in Figure 2.1): Robots go to the charging cells when their batteries are low. 5 Figure 2.1: Warehouse Execution Area. The cells that are symbolized with upward arrows are picking line cells. Robots should follow these line cells in order to go to a picking cell. Path cells are the other cells that robots move on. Some traffic rules are applied on these path cells. Figure 2.3 illustrates a sample direction rule on every path cell. On the cells with right arrow, robots are not allowed to go left, while they can go towards up or down if there is no obstacle. Similarly, on the cells with up arrow, robots cannot go down, while they can turn to left or right. On the cells with no direction arrow, robots can go towards any direction. In the considered system, path planning and control issues are performed in a WES which has both hardware and software components. Robots just execute atomic orders such as going to the next cell, lifting the pod, etc. WES and robots communicate with each other via wireless links, however, there is no communication link between robots. This reduces communication cost on the network as mentioned in [5]. There are three main factors that should be well designed to succeed an efficient life-long task planning in an RMFS based warehouse: incoming of stock (replenishment), outgoing of stock (collecting orders) and necessity (such as charging of robots). WES manages all processes to handle these factors. An order process starts with customer order via online shopping, and ends with packaging order in a box. Steps of an order task can be summarized as follows. After receiving an order, WES checks stock and selects a pod 6 Figure 2.2: Pick Station [10]. that stores order items (pod selection). Then it assigns the pod to a robot (robot selection) and assigns the order to a pick station (pick station selection). After that, selected robot moves, takes and carries pod to the pick station (path selection). In the pick station, an employee picks product to basket. Finally, the robot reallocates the pod to the warehouse (cell selection) and returns to its initial location. Figure 2.4 illustrates main components of WES. All orders are managed in Order Management unit and passed to Robot Management System (RMS). RMS handles orders by managing robots in the warehouse and makes them ready for shipping management. 2.2. Communication System Design 2.2.1. Literature Review In this section, we will focus on communication design for smart warehouses with AMR. Nowadays, using internet-connected mobile devices is increasing in the industry, especially in Industry 4. Most of the mobile devices connect to the system via Wi-Fi. The characteristics of these devices and systems will change the demands 7 Figure 2.3: Warehouse Traffic Rule. Figure 2.4: WES Model. 8 of the network and especially the TCP algorithm. Zhang et al. [11] worked on wireless communication, Zigbee and Wi-Fi (IEEE 802.11), for Automated Guided Vehicles (AGV) as a survey. They analyzed characteristics of wireless communications for AGVs with needed high reliability, low latency, short packets. Another Zigbee working is for smart warehouses that done by Liu et al. [12]. They analyzed controlling sensors network such as temperature, light, smoke and infrared for smart warehouses. Zigbee network is a good solution for such an environment. Mostly, mobile robots have upper communication level application over microcontroller to connect with Wi-Fi, Bhuyia et al. also developed a Wi-Fi communication module for ATmega Microcontroller [13]. Mobile robots-based systems need low latency to manage action control. Kumagai et al. worked on controlling robots over WLAN with the estimation of RTT [14]. In other words, they made a calculation to estimate delay to adaptive controls of robots. Mobile robot teleoperation in the WLAN is worked in [15]. They analyzed package delay effects of extra network traffic caused by other users for tracking robots. Both, UPD and TCP protocols were analyzed in this work. Haxhibeqiri et al. worked on flexible Wi-Fi communication for mobile robots especially AGVs [16]. They analyzed AGVs issues on Wi-Fi such as handover latency effect and proposed their solutions. Small mobile devices may have limited CPU and memory resources so it may hard to run complete TCP/IP protocol. Salleh et al. proposed a minimal TCP/IP model in terms of code size and using resources [17]. 2.2.2. Communication Requirements 2.2.2.1. Low Latency. WES coordinates AMR with commands over Wi-Fi. Most of the time, AMRs will move on the warehouse with 1,5 m/s velocity. WES will send the going command to the robot step by step to avoid collision with the reservation system. The robot will send a location with capturing QR code on the cell floor as a response for the going command. This process is time critic. A latency caused by processing computation or network-based will reduce manageability. The upper bound of latency is 50 ms (30 ms for processing, 20 ms for communication) is calculated for AGVs 100 mm path accuracy as mentioned in [16]. Our environment and localization methods are different from AGVs that mentioned in [16]. Therefore, these values will be calculated for our demands at the next steps. 9 2.2.2.2. High Reliability. There are continuous communications between WES components, especially between Logic Service to AMRs. When the Logic Service gives a command to the robots, it must ensure that the robot received this command. Moreover, the robot also sends to the response of the command whether it succeeds or not. These responses are a must for the robot to get another command. The WES works request-response base so both requests and responses are crucial to continuing the warehouse process safely. WES communication needs high reliability because failure will decrease the performance of the system. Increasing the failure of communication will cause corruption in the system. 2.2.2.3. Full Coverage. Warehouses may have a huge area. Wi-Fi network coverage is crucial for the system. The range of the Wi-Fi should be calculated well to accomplish full coverage. Any dead point that does not access Wi-Fi will cause a problem for the system. Moreover, the Wi-Fi network should handle static and dynamic obstacles for Wi-Fi communications such as walls, pods, pod loaded robots. Robot’s signal quality must be enough to communicate healthily even when the robot is under the pod or at the corners. 2.2.3. Communication System Architecture There are two types of communication methods on mobile devices in smart warehouses. The first one is the centralized method that the central control mechanism manages the robots and gives an atomic command to complete tasks. It makes global optimization simple and efficient. The second method is the distributed method that each robot performs their localization, path planning, and obstacle avoidance tasks via communication with other robots instead of the central controller [11]. Increasing the robot number will make the management of the system harder in a centralized method. Managing the robot individually reduces the complexity of the system. On the other hand, it increases communication costs because each robot should inform other robots while changing the location or action. WES and robots communicate with each other via Wi-Fi links, while there is no communication link between robots. This reduced communication costs on the network as mentioned in [5]. Figure 2.5 illustrates the main components of WES. Logic (Core) Service is a restful 10 Figure 2.5: WES Communication Architecture. 11 service so other components connect with it via HTTP. HTTP is an application layer that is transferred over TCP. Logistic Service, Pick Station, Manager Dashboard, Replenishment Station and Layer 3 Robot Api is .Net Core web application. The Robot includes 3 layers. The Layer 3 - Robot Api is .Net Core Web Application. The robot communicates with Logic Service over it via HTTP. It makes easier to control the robot from the outside by another application. Moreover, it isolates the outside communication interface from the robot applications. Layer 2, the communication coordinator of the robot, is the C++ application. Cam1 and Cam2 are also C++ applications these read QR code from the camera and make localization process. Cam1 is for cell QR code on the floor. Cam2 is for the pod QR code under the pod. Layer 1, robot motion controller, is the low-level robot coordinator written in C. Layer 2 communicate with Layer 3 via Nng (a modified version of Nanomsg socket library) over TCP socket. Layer 2 communicate with Layer 1 via RS232 serial port. All WES system components work on LAN. We chose Wi-Fi instead of Zigbee that has limited performance mentioned in [10]. Although Zigbee is cheaper and consumes a low battery, we need more capacity. Most of the communication work with TCP. Therefore, the communication issue is crucial to manage and optimize the smart warehouse process with AMR. C1, C2, C3 and C4 connections in Figure 2.5 are wired connected to reduce BER. C5 and C6 must be wireless connected. 2.2.4. Communication Design Decision We consider a 38 x 43 m2 warehouse that has 1634 cells. We have implemented a WES as mentioned before in Figure 2.5. For each robot, we run a separate .Net Core web application for simulation on the server. In the simulations, the parameters of robot actions and movements are realistically set according to the values obtained from a real physical robot. The communication between the management system and robots are done via HTTP over Wi-Fi. There are eight methods (command) for robots as we can see in Table 2.1. These commands are sent to L3 Layer Communication Api via HTTP by Logic Service in Figure 2.5. The response of these commands is a generic robot response message model to standardize implementation. It includes isSuccess (bool), ErrorCode (int), ErrorMesage 12 Table 2.1: Robot Commands / Requests (string), Data (object). We run 12 robots on the warehouses to analyze the robot communications (request/response) count between WES. We saved for 2375 working seconds while robots run for tasks. The results can be seen in Table 2.2. The mean total command count for one robot is 1607 in 2375 seconds. In other words, the robot gets 0,68 command on average, in one second. However, it is an average, hence the robot may get more than one command in a second. Waiting on stations with pod decrease this average. The picking process took approximately 1200 seconds in our simulation. When we subtract this time from working duration, it will give us to average command number in one second as 1,37. This value is critical for us to design communication system. The total command number for all robots is 19282. It meant that the WES gives 8,12 commands to the robot on average in one second. Each robot takes care of its commands in the centralized methods. There is no need to inform other robots to their changes. The central mechanism handles the operations to coordinate them. On the other hand, the Distributed method will increase the network traffic as robot count times. In the distributed system robot should inform just localization change to the other robots. Therefore, the network communication count will approximately 194964 (16247 x 12) without optimization. It will cause huge traffic for WES. There are new solutions as mentioned in [18]. They divide large warehouse to sectors and coordinate motion and tasks with decentralized methods. Robots will inform robots which are just in their sectors. 13 Table 2.2: WES Communication Analyze / Command Count 2.2.5. Wi-Fi Network Suggestions Figure 2.6 shows the sample of WLAN topology of WES. Wi-Fi sharing over robots can be useful for some critical situations. Let assume that R8 in Figure 2.6 can’t access any APs in a point, maybe under the pod or near the obstacles, so it can’t receive any command. If elapsed time without connection exceeds a threshold value, it should stop to avoid a collision or an accident. While the robot in such a situation, stop or moving, it tries to access network over another robot that has network device to share Wi-Fi over itself. These types of the robot can be used as a relay node if needed. The WES system will forward a Wi-Fi sharable robot if there is no one near the R8. The WES data packages sent to robots are very short because it just includes some parameter for commands. The highest data package is the response of the SetRobotPath method on L3 Layer. This response includes cell list, in other words, path information of a robot. The size of the data package will not exceed 3 kB. Since the packages are short, this would reduce the collision possibilities in the Wi-Fi network. The Wi-Fi has a limited range to access with enough signal power. We need multiple Access Point (AP) to cover warehouse with desired signal power. Only robots can use the Wi-Fi to reduce bandwidth usage and noise of the channels. Other components are 14 Figure 2.6: WLAN Topology of WES AMRs connected to the LAN with wired network interfaces. The APs use a queue for outgoing packages. When the queue of the AP becomes full, it will cause packet losses [13]. Therefore, management of the queue size of APs is critical point to achieve efficient network communication. Moreover, each robot has buffer to receive TCP packages. These buffers, window size, is crucial for efficient Wi-Fi communication. The AP receiver buffer gets responses from robots. Increasing the robot count will increase the buffer load on the AP receiver. The WES system gets approximately 8 commands in a second on average for 12 robots. This load will be shared by APs. The load of the AP receiver buffer will change according to AP count and robot traffic near AP. One command will send just one package without division because robot response in our WES is well for Wi-Fi TCP package size. There are two types of robot requests; command and information. The priority of the command request is higher than the information request. Hereby, using priority scheduling on APs receiving buffer will reduce the loss of critical packages. The count of the AP in Figure 2.6 will change according to the power range of APs. The WES must be well tested in the real environment to ensure full network coverage. 15 The control message should be sent to a robot in a short time interval continuously while the robot is moving to all possible paths. 16 3. PATH PLANNING FOR ROBOTS In this chapter we first define the path planning problem for multiple mobile robots in warehouse. Moreover, the design decision rules for path planning is explained. Then, we provide solution approaches for collusion free path planning. In the last section, we test our path planning algorithm under different number of robot and design decisions in our WES simulation environment. 3.1. Related Work Autonomous warehouse robots that lift pods and bring to pick stations were first introduced by Wurman et al. [1] in 2006 and patented by KIVA Systems Inc (which was later acquired by Amazon in 2012) [2]. Since then, optimizing warehouse operations has been attracted much attention from both the academic community and industrial sector. Currently, other than Amazon KIVA, there exists many other warehouse robots, such as Knapp Open Shuttle (2012), Locus Robotics System, Swisslog CarryPick, GreyOrange Butler, Fetch Robotics Freight (and Fetch), Scallog System, Hitachi Racrew [3]. Recently, Boysen et al. [2] provide a thorough survey that discusses warehouse systems especially suited for e-commerce retailers. Roozbehani et al. [19] explored three algorithms for trajectory optimization: Reservation Based Algorithm (RBA), Fixed Highways Algorithm (FHA), and Adaptive Highways Algorithm (AHA). RBA tries to choose a route that does not pass through the space reserved by other vehicles at any given time. FHA is a rule-based approach that includes specific movement rules for different scenarios and usually do not include massive search. [19, 5] . On each lane, robots can only move in one direction, hence they never encounter head-to-head situation. The drawback is that a large portion of the search space is omitted. On the other hand, AHA uses time estimations and atomic decisions to avoid collisions. There are some hybrid approaches as mentioned in [5], which include some specific traffic rules to critical lines and include reservation-based search. However, these approaches are not complete and error-free, as deadlocks may occur in bottlenecks. Liu et al. [18] introduce a hierarchical path planning and motion coordination of the large-scale system with thousands of robots. They divide large warehouse to sectors and coordinate motion and tasks with decentralized methods. This work is applicable for large scale 17 systems that have thousands of robots. Recently, several studies are conducted related to applying machine learning techniques in path planning. Deep reinforcement learning method is introduced in [20], with a small area experiment. Genetic Scheduling and Reinforcement Learning path planning approaches are mentioned in [21]. Results of these works are promising but there is so much space to improve. In this study we use a reservation system that is similar to the one proposed in [19]. Instead of applying fixed highway to the whole warehouse as in FHA algorithm, we only apply traffic rules to particular critical areas. We also apply time dependent path planning similar to AHA algorithm on 3x3 neighboring cells of the starting cell. Therefore, it can be said that this study merges several ideas proposed in [19, 5]. Compared to existing studies, this paper has some novel contributions. We introduced modified A* algorithm that considers turning costs of robots. We give details of the modified collision-free A* algorithm and its implementation. We determine no-path scenarios that are not mentioned in the previous studies and propose solution to them. The effect of the warehouse design in path planning is analyzed. Task completion time and head-to-head encounter count (re-planning) is also analyzed according to robot count and warehouse design decisions. 3.2. Path Planning Problem Path planning is an extremely important and challenging problem in warehouse system with multiple robots. For every order, most appropriate path should be selected for the assigned robot, while avoiding collision with other robots and handling deadlock conditions. The main objective is to minimize path lengths in order to increase efficiency of the system and to reduce both energy and time costs. There are several constraints of the problem. Robots can move in both forward and backward directions and can rotate to right and left. On each cell, some traffic rules apply as described in the previous subsection. For every robot, there are some fixed obstacles (pod cells) and dynamic obstacles (other robots) in the warehouse. Location and direction of the robots can be tracked by QR codes on the cells. Movement and rotating of robots have time and energy costs. We assume that moving one cell ahead has a cost of 1 unit. For rotating, there are additional costs for deceleration, turning and acceleration. In real robots, we measure time consumption 18 of these three actions as 0.5 units, 1.5 units and 0.5 units respectively. With an additional energy consumption cost of ce , total cost of rotating, CR , is given as: CR = 2.5 + ce (3.1) ce depends on energy consumption parameters and importance of the energy usage. Figure 3.1: Cell Reservation of Robots For every order, a path should be calculated for a robot from the initial location to the pod, from the pod to the pick station and from the pick station back to the initial location. In order to avoid collisions with other robots, some additional mechanisms are needed. Reservation table is a common method for collision avoidance. In this approach, robots that are moving on their paths reserve next two cells and they stop and wait if a cell is already reserved for another robot. (Note that when the second cell cannot be reserved, robot decelerates and stops in the first reserved cell.) Figure 3.1 illustrates this reservation approach. Green circle is a robot with identification number 146. Blue squares are pods. Robot 146 has reserved next two cells on its path as shown in the figure. Right arrows on the cells indicate that the robot cannot move on left direction on these cells. Reserving next two cells for all robots avoids collisions. However, it does not prevent deadlock conditions. Figure 3.2 illustrates a head-to-head encounter condition. The robot 151 wants to reserve next two cells, but it could not reserve second cell because of robot 152. Robot 152 too cannot proceed, since it waits robot 151. Hence, both robots are waiting each other, and deadlock occurs. In an optimal path planning, these deadlock conditions should be minimized and should be resolved in an efficient way. In case of a head-to-head encounter, a path re-planning is needed. However, in some rare scenarios no path may be found to the target. Figure 3.3 illustrates an example to such scenarios. Robot 150 (green circle) tries to go to cell 150 (red square). At the same time, robot 151 tries to go to cell 151, and 19 Figure 3.2: Head-to-head Encounter head-to-head encounter occurs. When robot 150 tries to re-plan the path, it fails because the robot 151 blocks the target. On the other hand, robot 150 blocks the target of robot 151. Therefore, both robots cannot re-plan their paths and they are blocked. Figure 3.3: An Example No-path Scenario 3.3. Warehouse Design Decisions The warehouse design plan is a deep and complex topic that affects workflow directly. In this paper, we especially focus on two design issues: (i) pod alignment plan, and (ii) warehouse traffic rules. Both design considerations directly affect movement of robots, and they depend on warehouse characteristics and needs. While designing pod layout in a warehouse, one should consider many important factors such as physical properties of warehouse and pods, warehouse occupancy rate and sales rates of products. There are three main objectives of designing pod alignment: • Space gain: Number of pods placed in the warehouse should be maximized to store as much product as possible; • Fast order picking: Pods should be placed in such a way that the collection of the products should be done in an efficient and fast way; • Robot path suitability: Pods should be placed well to achieve collision-free efficient 20 path planning. However, there is a trade-off between these objectives, and it may not be possible to achieve all three objectives perfectly. For example, if we increase the space gain, robot path suitability decreases. The priority of these objectives could be different for different warehouse owners. Considering all these issues, most suitable design of pod alignment should be investigated. The warehouse traffic rule is another important design issue. As we mentioned above, robots may encounter some deadlock scenarios. In order to resolve a deadlock scenario, path re-planning would be needed for at least one robot. This re-planning causes extra energy and time cost. Therefore, assigning traffic rules to some cells would reduce the frequency of deadlock occurrences significantly. 3.4. Solution Approaches In this section, we provide solution approaches for path planning and warehouse design problems. We first present collision-free path planning algorithm and then we discuss warehouse design decisions. 3.4.1. Collision-free Path Planning Algorithm Path planning for robots on a static grid is a well-studied problem. Most of the proposed algorithms are based on the well-known A* algorithm. A* algorithm is based on heuristic search on grid cells. It maintains a tree of paths and extends those paths one edge at a time based on the cost of the paths and estimates of the cost required to extend the paths all the way to the goal. Specifically, A* algorithm selects the path that minimizes f(n) defined in f (n) = g(n) + h(n) (3.2) where h(n) is the heuristic distance between the nth cell and the target cell, and g(n) is the distance between the starting cell and the nth cell [22]. Due to high dynamics of the warehouse environment, traditional A* should be revised appropriately. A sensor-equipped mobile robot can encounter an obstacle (or deadlock conditions) and the remaining path should be revised. Rapid path re-planning is very important because the robot should not wait redundantly for the computation of new path, or it should not move to a wrong direction. 21 Although the A* algorithm finds the shortest path, it does not deal with the number of turns. However, the turning process has a significant cost, CR defined in 3.1, that cannot be neglected. Therefore, turning cost should be added to g(n) as follows: g(n) = |xn − xs | + |yn + ys | + k ∗CR (3.3) where k is the turning count, (xn ,yn ) is the x-y coordinate of nth cell and (xs ,ys ) is the x-y coordinate of the starting cell. The heuristic function h(n) is defined as usual: h(n) = q (x f − xn )2 + (y f − yn )2 (3.4) where (x f ,y f ) is the coordinates of the final (target) cell. Finding the shortest path is not the best and enough solution for mobile robots in the warehouse [18]. Because statuses of the cells change dynamically. For example, another robot can be on a cell or will be on a cell in a short time. Therefore, availability of the path should be checked continuously. For this purpose, after path planning is done for a robot, WMS gives orders to the robot to proceed cells on the path, step by step. It checks status of the next cell which can be free or reserved by another robot. If it is free, it reserves the cell and proceeds, otherwise it stops and waits. To achieve this process, a reservation table is used, as described in [19]. Collision-free path planning algorithms consist of two main parts: (i) Path planning, and (ii) step by step movement of robot. In Figure 3.4, the steps above the dashed line are related to path planning, while the steps below the dashed line are related to step by step movement of the robot. In step 1, the whole layout of the static warehouse cells is obtained. Locations of the fixed obstacles are stored in the memory. In step 2, 3x3 neighboring cells of the starting cell is checked in the reservation table. All the nearby cells that are occupied or reserved are treated as obstacles. This is to avoid any contention between the nearby robots. The reservation statuses of faraway cells are ignored while finding a path. In step 3, modified A* algorithm (with turning cost as described above) is applied considering all the obstacles in the grid space. The modified A* algorithm finds a path to the target 22 Figure 3.4: Flowchart of collision-free path planning algorithm 23 cell, if exists. However, sometimes a path may not exist as described in the previous section. In this case, the robot waits one second and retries to find a path, hoping that some dynamic obstacles blocking the path get out of the way. However, in some cases, two robots continuously block their paths as in Figure 3.3. To cope with such cases, a timer is employed for every robot, and once timeout occurs before finding any path, robot proceeds n cells towards the target without following a particular path (step 15). This step aims to remove obstruction of robots to each other by moving a robot arbitrarily towards its target and expectantly making way for the other robots. Once a path is set for a robot, it proceeds step by step by reserving cells. In order to move, a robot needs to reserve two cells ahead. If any cell is already reserved or occupied by another robot, it waits and retries after one second. If both cells are available, it proceeds one cell and tries to reserve the next cell. Robot moves continuously as long as its path is available. However, if the cell after next is not available, then it stops and waits for one second. Surely it cannot stop immediately and while decelerating and stopping, it will occupy the next cell ahead. This is why two cells should be reserved in any case, before starting to move. In case of unavailability, after waiting a second, the robot retries to reserve next cell. However, if a deadlock or timeout event occur, then the path planning algorithm goes back to step 2 in order to re-plan the path. The algorithm stops when the robot reaches the target. 3.4.2. Warehouse Design Decisions Now, we will describe several warehouse design decisions that affect the efficiency of the system significantly. As we mentioned in the previous section, there are three main (but potentially conflicting) objectives: space gain, fast order picking and robot path suitability. As illustrated in Figure 2.1, there are several types of cells in a warehouse. Some of the cells should be reserved for charging, pod turning, picking, etc. A large rectangular area is reserved for pod placement. However, there should be some free cells for movement of robots. If the number of free cells increase, robots can move more easily and can complete their paths in a shorter time, however number of pods that can be stored in the warehouse would decrease. If number of pods increase, this 24 would complicate movement of robots. In order to analyze trade off between alternatives, we would consider several pod alignment decisions. Figure 3.5 8 illustrates part of a warehouse with horizontal 2x6 pod alignment. Figure 3.5: Horizontal 2x6 Pod alignment In this study, we will analyze 6 different pod alignment design: • Horizontal 2x4 pod alignment (H 2x4) • Horizontal 2x6 pod alignment (H 2x6) • Horizontal 2x8 pod alignment (H 2x8) • Vertical 4x2 pod alignment (V 4x2) • Vertical 6x2 pod alignment (V 6x2) • Vertical 8x2 pod alignment (V 8x2) 3.5. Experimental Study We consider a 27 x 34 m2 warehouse that has 918 cells. We have implemented a robot management system, which is a C# .net core web application that coordinates the robots and gives orders to them. Robots are also simulated using C# .net core web application. For each, we run a separate simulation application on the server. Warehouse and robot motions are 3D visualized by using Threejs. In the simulations, the parameters of robot actions and movements are realistically set according to the values obtained from a real physical robot. In modified A* algorithm, energy cost (ce ) in (1) is picked as 0.5. The communication between the management system and robots are done via HTTP over Wi-Fi. The only human activity in the overall process is picking and order item from pod. 25 The time elapsed for this task is calculated as uniformly distributed random value between 3-5 seconds for a single item. Each order consists of random number of different items (between 1 and 5), and from each item, single, two or three units are ordered randomly. Therefore, maximum number of items for any order is 15. 3.5.1. Number of Robots One important test parameter of our study is the number of robots. We run the warehouse simulations with 6 to 13 robots for horizontal 2x6 pod alignment. We processed 500 orders that include 1677 items in total. Robots completed orders in 795 order tasks. In a single order task, the robot carries one pod to the pick station, and put back the pod to its cell after items are picked. One order can be completed in multiple order tasks. Moreover, in a single order task, robot can carry different order items at once. Figure 3.6 illustrates the change in the order completion time when number of robot increases. Although completion time reduces (as expected), the reduction is not linear. This is because the movement area is fixed, and as the number of robots increases in the warehouse, possibility of robot encounter and waiting increases. Increase in the number of head-to-head encounter occurrences with respect to the robot count can be clearly seen in Figure 3.7. Total route elapsed by the robots range between 70 km and 72 km when we increase number of robots from 6 to 13. We observe that increasing number of robots do not increase total elapsed route significantly. 3.5.2. Effect of the Traffic Rules The restriction on the movement directions on cells reduces head-to-head encounter significantly. On the other hand, traffic rules increase the path length of the robots in general. We have tested the system with and without traffic rules for 10, 12, 14, 16, and 18 robots, 500 order count and horizontal 2x6 pod alignment. Although the number of head-to-head encounters is approximately 10 times greater for the case with no traffic rule, difference in order completion times is close to each other. The results can be seen in Figure 3.8. Removing the traffic rules give better results for small robot count. However, while increasing the number of robots, slope of the decrease in order completion time is higher and applying traffic rules gives better results (after 16 robots in our scenario). In order to get better results, traffic rules can be relaxed and collisions may be 26 Figure 3.6: Order completion time vs robot count Figure 3.7: Head-to-head encounter vs robot count 27 Figure 3.8: Effect of the traffic rules avoided by time estimations as proposed in AHA algorithm [19]. However, robot motion delays increase complexity, and traffic jams and deadlocks cannot be avoided completely [18]. In this work we did not implement time estimation, instead, we use atomic decisions and some algorithmic improvements as mentioned in Figure 3.4. As future work, we will work on learning-based approaches to avoid collisions in warehouses without traffic flow rules, with an aim to reduce total order completion time. 3.5.3. Pod Alignment For pod alignment tests, we fixed the number of robots to 10. Robots processed 500 order for each test. For different pod alignments, the number and location of pods are different. Therefore, in our simulations we selected order items randomly at each test. Although order counts are equal, locations of order items are different. Therefore, we ran each test scenario 4 times and take their mean values in order to reduce the effect of this location change. 23 x 27 m2 rectangular area of the warehouse can be used for placing pods. As an example, for horizontal 2x6 pod alignment, there are 4x8=32 pod sets in the whole warehouse. Therefore, the total number of pod cells and pods in H 2x6 pod alignment is 32x12=384. Some pod alignment designs do not fit this area (for example there would be a large empty area if we only use horizontal 2x8 pod sets). In that case, we align shorter pod sets in the middle of the warehouse. 28 Test results for various pod alignments are shown in Table 3.1. For vertical 4x2 and horizontal 2x4 pod alignments, robots complete order tasks in a shorter time and move less kilometer than the other cases. However, pod count is less than others. One interesting result is that vertical 8x2 pod alignment which enables placement of highest number of (401) pods, yields less picking time per item compared to other pod alignments like H 2x6 and H 2x8. Table 3.1: Test Results for Various Pod Alignments Let us define A as the set of all possible pod alignments. In order to compare different pod alignments, we define an efficiency metric for each pod alignment a ∈ A, namely Ea as follows. minti na i∈A Ea = pS + pT max ni ta (3.5) i∈A where na is the number of pods and ta is the average picking time elapsed per item, for pod alignment a ∈ A. pS ∈ (0, 1) is the space priority and pT ∈ (0, 1) is the time priority of the warehouse system, where pS + pT = 1. Hence, if space and time have the same priority for a warehouse system, both pS and pT would be set to 0.5. If space is a more important constraint and it is very important to store more pods in the warehouse, then pS would be higher than pT . If picking time is more important than pod count, then pT would be higher. The maximum possible value of Ea is 1. We show Ea values of each pod alignment, for three different cases (with pS equal to 0.5, 0.3 and 0.7 respectively) in Table 3.2. It is observed that vertical 8x2 is the best pod alignment when space has at least as much priority than time. When (pS , pT ) = (0.3, 0.7), vertical 4x2 is slightly better than vertical 8x2. 29 Table 3.2: Efficiency Results for Various Pod Alignments 30 4. TASK PLANNING FOR ROBOTS In this chapter, we first define the task planning and optimization problem for multiple mobile robots in the warehouse. Moreover, we propose our Order Batch to Robot Task Conversion (OBRTC) algorithm and order task selection method to optimize RMFS based warehouses. In the last section, we test our algorithms under different parameters such as the number of robots, order, pick station, tote count, and different design rules in our realistic WES simulation environment. 4.1. Related Work RMFS is a new approach with an increasing popularity in the last decade. Cooperation of multiple robots and coordination of many other resources such as pods, totes, stations etc. increase the complexity. Enright and Wurman (2011) present general concepts of optimization and coordination of an RMFS in [8]. They mention importance of pile-on optimization to reduce order picking time. Lamballais et al. [4] analyze performance of an RMFS under four models such as single-line and multi-line orders and with or without storage zones. They develop queuing network model that includes storage zoning and multi-line orders in order to estimate performance according to order throughput, average order cycle time, and utilization of robots and workstations. The assignment of pick orders to pick stations (Pick Order Assignment - POA) and the pod selection in the pick process (Pick Pod Selection - PPS) are two important issues for order picking optimization in an RMFS. First POA, then PPS is applied [6, 1]. Moreover, [23] performs an approach for integrating POA and PPS. Merschformann et al. [6] analyze optimization performance by studying important decision rules such as POA, PPS, Replenishment Order Assignment (ROA), Replenishment Pod Selection (RPS) and Pod Storage Assignment (PSA). They compare multiple rules and find correlation between them to increase order item throughput. They point out importance of pile-on and robot travelling distance for RMFS performance. Boysen et al. [24] study on processing the orders at the pick station. They focus on batching and sequencing of picking orders in order to decrease robot needs. The decision problem they formulate is NP-Hard and they provide several heuristic algorithms to converge to the optimal. They show that their algorithms decrease the number of pod visits, but they study only for small instances (up 31 to 100 orders). Their results imply that the provided heuristic approaches would take significant amount of time for larger instances and may not be feasible for real time applications. [23] integrate assignment of pods to station and orders to station for order picking process, instead of calculating separately. They also propose to split orders in order to improve RMFS efficiency. In other words, parts of an order are allowed to be processed in different stations and combined later at the packaging stations. While splitting orders improve the efficiency, it cause additional processing load. Evaluation of extra effort for splitting orders and combining later requires real experiments beyond simulation and analytical models. Zou et al. [25] focus on rule based robot to task assignment with handling speeds of workstations and propose a neighbourhood search algorithm to find a near optimal solution. Moreover, they analyze shelf block size effect to the RMFS throughput. [26] identify the performance characteristics of an RMFS by providing a literature review. They mention the relation between performance and the design decisions of RMFS for operation as a preliminary finding from an ongoing study. Other important optimization approach in order picking process is to use efficient replenishment and pod allocation methods. These methods proactively increase picking order throughput since they provide more efficient options to select pods. [27] focus on efficient pod alignment on the warehouse storage area. [28] prove that spreading inventory across many pods significantly decreases the time of collecting orders. They also analyze the optimization of variables such as the number of items per pod, replenishment level per pod, and pick station to replenishment station ratio. Robots can be dedicated specifically to order picking task or replenishment task. This approach is called dedicated robot assignment. On the other hand, in pooled robot assignment, both tasks are performed by a single pool of robots. Merschformann et al. [6] adopts dedicated robot assignment and assigns two-third of the robots for order picking tasks, and others to replenishment tasks. Roy et al. [29] analyze RMFS for single and multiple zone with both dedicated and pooled robot assignment. They notice that using pooled robot assignment reduces order picking time, while it increases the replenishment time. Yuan et al. [30] perform their order picking analysis with the pooled and dedicated 32 robot system. They also analyze optimal robot count and speed for their test environment. Zhou et al. [31] focus on balancing robot workload while optimizing total robot travel time. They propose a heuristic balance mechanism to assign tasks to robots. In other words they select robots for tasks with an aim of minimizing total travel cost and balancing robot workload. However, this approach will reduce robot utilization since robots will need to wait for the others to become free. In a cost-efficient RMFS, number of robots are optimized and they are busy most of the time. In a warehouse with several hundreds of robots, 3 to 4 robots may become idle concurrently [8] and it can be suitable to choose only among those robots. 4.1.1. Main Contributions In this study we propose a novel Order Batch to Robot Task Conversion (OBRTC) algorithm which tries to find minimum number of pods that includes items of multiple orders. In other words, contrary to existing studies, OBRTC algorithm handles a number of orders as a single order instead of handling orders one by one. The number of orders to be handled as a batch has a crucial role to increase pile-on value even for high SKU. After OBRTC is executed, WES selects available totes at the pick stations for items in pod when the order task is assigned to the robot. The pod of order task will consist of multiple order’s items, and it can be assigned to different stations if necessary. Furthermore, we propose Adaptive Robot Task Selection (ARTS) method to select a new task for a robot that become available. New task selection is performed according to a novel priority based heuristic model which depends on various criteria such as distance to robot, totes usage and time. In brief, we first select pods for multiple orders without assigning stations, totes, or robots to reach maximal pile-on value. Then we adaptively manage other resources considering the system dynamics. Contrary to the most of the studies in the literature, we develop and use a fully realistic simulation environment that includes robot collisions, waiting time on path and stations, and effects of the loaded pod weight. Moreover, our simulation includes charging process by considering a realistic energy consumption model (depending on movements and loads) and real charging times. Some e-commerce companies may sell items even if they are not already stored 33 in their warehouses and these items are replenished on demand. Therefore there could be many already ordered items on the pods loaded at the replenishment stations. Management of this case is mostly not addressed in existing studies. In our study, we also test such a scenario and analyze its effects on the system performance. In summary, our work has three main contributions; (i) We provide a task planning approach with high pile-on value even for high number of SKUs. (ii) We propose a novel dynamic and parametric heuristic model for order task selection process. (iii) We develop a highly realistic simulation environment and provide test results under various decision rules which shade light on how to design a smart warehouse system with high efficiency. 4.2. Model and Methodology There are three main factors that should be handled in a best way to succeed an efficient life-long task planning in an RMFS based warehouse: incoming of stock (replenishment), outgoing of stock (collecting orders) and necessity (such as charging of robots). In this section, we explain our methodology, algorithms and design decisions to handle these factors in an efficient way. 4.2.0.1. Order-to-Task Conversion. Order task is the process of collecting orders. We consider multi-line orders as a usual behavior of e-commerce customers. In the basic approach, each order is converted to order task(s) individually. Let assume that we have an order with three items. WES tries to find pods that includes these items. This order will be completed with at least one pod and at most three pods. In the basic approach, most of the time robots carry one pod for just one item of an order. Collecting a multi-line order will mostly need more than one order task. In order to increase the pile-on value, we handle order batches, instead of handling orders one by one. Order batch method is usually applied in traditional warehouses which do not employ robots [7]. Human workers collect orders after applying some order batch optimization. In this study, we aim to adapt "order batch" concept to RMFS-based 34 Figure 4.1: Order Task Process. smart warehouse in most effective manner. Figure 4.1 shows high level flow of order task process. When an order arrives to the WES, it is stored in an Order Pool. WES chooses a batch of orders from this pool and converts these orders to order tasks (Step 1 in Figure 4.1). Then these tasks are assigned to the available robots (Step 2). Then the assigned robot goes under the pod, lifts it, and carries to the pick station (step 3). Picker puts order items to the totes. Each order is collected in the tote that is reserved for that order. When an order is completed, its tote is departed from the pick station and a new tote is put in its place. Now we will describe Step 1 of Figure 4.1, i.e. the order to task conversion method in more detail. To accomplish high pile-on value and provide high utilization of robots, we propose a novel Order Batch to Robot Task Conversion (OBRTC) algorithm as shown 35 in Algorithm 1. Algorithm 1: Order Batch to Robot Task Conversion OrderList ← n orders from the set of all orders; OrderItemList ← all items of orders in OrderList; PodList ← set of all pods in the warehouse; while OrderItemList 6= 0/ do 1. Podmax ← Pod that contains highest number of items in OrderItemList; 2. Create OrderTask with Podmax ; 3. OrderItems ← All the items of OrderItemList included in Pod1; 4. Add OrderItems to the OrderTask; 5. Remove OrderItems from OrderItemList; 6. Remove Podmax from PodList; end In OBRTC algorithm, firstly we select nob orders from order list for order to task conversion. This method considers nob orders as a single order to increase item pod matching possibilities. Thus, more order items can be collected with less number of pods. All the items of an order should exist in the warehouse stock. If there is a missing item for an order, this order is marked as "stock-waiting" order and not included in the task conversion process until the required stock is provided through replenishment process. At each iteration, OBRTC algorithm creates an order task with the pod that includes maximum number of items. Note that if an order task already exists for this pod (as a result of previous run of OBRTC algorithm), then new items would be included in this order task, instead of creating a new one. Then this pod and the items it includes are removed from the list and the algorithm continues with the next iteration for converting remaining items to tasks with the remaining pods. The algorithm terminates when all the items in these nob orders are converted to tasks. In the OBRTC algorithm, there are some important design parameters and restrictions to be considered. The value of nob has an important role for optimization. Increasing the value of nob will increase the number of items per task. In other words, more items will be collected at the pick station when the robot brings the pod. On the other hand tote count at the pick station is the important restriction for executing nob orders 36 during the same time interval. For single order to task conversion, it is simple to handle tote reservation. Because all the items of an order task will be collected consecutively in a short time. However, when the orders are handled as a batch, items of an order may be collected in an extended time period. Because a number of order tasks are created by OBRTC simultaneously and items of an order may be spread over these order tasks and there could be some gaps between their collection times. This results in high number of reserved totes that include a portion of order items. We analyzed the best value of nob in different scenarios in Section 4.3. OBRTC algorithm should be executed and the order task pool should be refilled in appropriate intervals. This is important to avoid idle waiting of robots (due to lack of order tasks). On the other hand, if task pool includes too many tasks, then this can increase the execution time of the task selection process described in the next subsection. Therefore, OBRTC algorithm is executed when the number of order tasks in the order task pool becomes less than the number of working robots. 4.2.1. Order Task Selection Method After order tasks are created, an order task selection method is applied to manage resources efficiently and adaptively. For an available robot, WES assigns a task from the order task pool. In this study, we propose a heuristic model for task selection, namely Adaptive Robot Task Selection (ARTS) which adapts to the dynamics of the resources. WES selects a task for an idle robot from the task pool according to three important parameters: 1. Distance of the order task’s pod to the robot; 2. Time elapsed after the creation of the order task; 3. Completion rate of the tote(s) assigned for the order task. When a robot finishes its previous task and becomes available, ARTS calculates a priority value for all order tasks in the pool, and assign the task with the highest priority. The priority value of an order task i for a robot r (prir ) consists of three components (that correspond to three parameters mentioned above): Spatial priority prirs , temporal priority prit and tote priority priτ . Each of these components are calculated as follows. i. Spatial Priority: 37 prirs = α 1 + log2 (distir + 1) (4.1) where α is a normalization constant and distri is the Manhattan distance between robot r and pod pi of task i, which is calculated as in (4.2). distir = x(r) − x(pi ) + y(r) − y(pi ) (4.2) Note that x(·) and y(·) are x and y coordinates of the cells that robots and pods stay on. Spatial priority is modeled using a logarithmic function. This indicates that if the distance is too low, high priority is given for that robot in order to decrease path traveling distance, and consequently to reduce the traffic. As the distance increases, there would be diminishing marginal effect of the distance to the task priority. The value of α describes maximum possible value of prirs , which is chosen to be 10 in our warehouse. ii. Temporal Priority: prit = β · agei (4.3) where β is a normalization constant and agei is the elapsed time (in terms of minutes) after the creation of task i. Temporal priority is modeled using a linear function that is not bounded above. This is to guarantee that all the tasks are handled before a maximum waiting time. The value of β is important to define maximum task age. In our study, we chose β = 0, 2. iii. Tote priority: Let us define O as set of all orders and Oi ∈ O as set of all orders that has some items to be handled by task i. Tote priority of task i is defined as priτ = ∑ (Boi)1+ρ (4.4) o∈Oi where ρ ∈ [0, 1] is the utilization ratio of the totes. In other words, it is the ratio of the number of reserved totes to the number of all totes. Boi is the base value for order-task pair 38 Figure 4.2: Tote Reservation Scenario. (o, i), which is related to the effect of task i on completing order o. We use the following values for Boi : • (Reserve) Boi = 1 if task i reserves new tote for order o, but not includes all items of the order. • (ReserveAndComplete) Boi = 2 if task i reserves new tote for order o, and includes all items of the order (hence completes the order). • (Fill) Boi = 3 if there already exists a partially filled tote for order o, and task i puts some more item(s) to the tote without completing the order. • (Complete) Boi = 4 if there already exists a partially filled tote for order o, and task i puts some more item(s) to the tote and completes the order. Let us describe tote priority by an example scenario illustrated in Figure 4.2. We consider a pick station with five totes (T1 - T5) and consider collection of four orders (O1 - O4). The items of the orders are shown in the figure. In a specific time, tote 3 (T3) is reserved for order 2 (O2) and tote 4 is reserved for order 4, and these totes are partially filled. Now we will calculate tote priority of the order task 1 (Pod1). Pod1 includes A and B items of order 1, but doesn’t include H. Therefore this task 1 reserves a tote for order 1 without completing the order, so B11 = 1. On the other hand, order 2 has two items, E and D. E is already put in tote 3, and D is included in Pod1. Therefore task 1 completes order 2, and B21 = 4. Moreover, order 3 includes single item (C) which is included in Pod1. So task1 reserves and completes order 3, i.e. B31 = 2. Task1 doesn’t include any missing item of order 4, so order 4 will not be included in the calculation of tote priority. Since two of 39 the five totes are already reserved, ρ = 2 5 = 0.4. Tote priority for task 1 is calculated as 11.4 + 41.4 + 21.4 = 10.6. The rationale behind the proposed tote priority model can be described as follows. In order to efficiently utilize the totes, partially filled totes should be completed as soon as possible. Therefore, if a task fills and completes a tote, it would have highest priority and the base value is set to 4. If it fills a tote without completing, again it has a high priority and the base value is 3. Tasks that reserve new totes are given lower priority, especially if they only partially fill a tote. Tote priority becomes much more critical when only a few unreserved totes remain. Therefore the base values are powered by 1+ ρ , which converges to 2 as the number of idle totes gets closer to zero. After finding spatial, temporal and tote priority values, overall priority of task i with respect to robot r can be simply defined as the sum of these values. Pr(i, r) = prirs + prit + priτ (4.5) While priority of a task is effected by all three parameters, it is dominated by temporal priority (prit ) if the task is too old, and dominated by tote priority (priτ ) if the tote utilization is too high. All the orders created from the same run of the OBRTC algorithm have almost same temporal priority. For these order tasks, other priority metrics effect the order of selection. But if there exists tasks that are generated from an older order-to-task conversion process, a newer task is favored only if its pod is too close to the robot. This is provided by the logaritmic function used in prirs . The proposed heuristic model is designed to dynamically achieve several performance goals such as reducing robot travel distance and order completion time, and increasing system utilization. Here it should be noted that although the task selection method provides efficient utilization of totes, it is still possible to not have an available tote for any of the items in a selected task. In that case this task is not assigned to the robot, and WES tries to select the next task in the calculated priority order. 4.2.2. Order Task Execution Order task execution starts just after WES assigns an order task to the robot by the order task selection method described above. Figure 4.3 illustrates the order task execution 40 Figure 4.3: Order Task Execution Flowchart. flow. The robot goes to the pod and lifts it. Since existence of tote is checked before task assignment (as described in the previous part), it is guaranteed that there is at least one available pick station. If tote assignment is required for any items in the task, WES tries to assign all totes in a single pick station in order to decrease robot moving time. WES also tries to balance workload at the stations while selecting the totes. The robot carries the pod to the selected pick station. When the robot is close to the pick station, WES checks whether the pod’s current angle is appropriate for the pick station. If it is not as desired, the robot goes to the turning cell and the pod is rotated. After desired angle of the pod is provided, the robot takes the pod to the picker. Picker collects items and insert into the tote(s). After the last item of the current face of the pod is taken, WES checks whether another face of the pod is also required for this station. When the picking task is finished for all pod faces in this pick station, WES checks whether another pick station is waiting for that pod. If so, the same processes are handled until all required items at that pod are collected. Afterwards, WES selects a free cell for the pod to locate. When the robot goes to the pod cell and release it, order task is completed. Now the robot is ready for another order task, and task selection process is executed again. If there is no 41 Figure 4.4: Replenishment Task Process. available task at that moment, the robot waits until a new task is created. 4.2.3. Replenishment Task Replenishment of goods is the other important task of WES that is performed by autonomous robots. Figure 4.4 illustrates overall replenishment task process and Figure 4.5 illustrates execution of replenishment task. Replenishment to task conversion is simpler than order to task conversion. It starts with the request of the worker in the replenishment station. Each station has a single worker. The worker requests a pod to fill with the replenishment goods. WES selects a pod according to the free space, i.e. the pods with more free space are prioritized. After selecting a pod, the system waits for an available robot if all the robots are busy. Replenishment task is assigned to the first robot that becomes available. The robot goes to the pod, lifts it and carries it to the replenishment station. If the pod’s access direction is not appropriate, the robot first goes to the turning cell and rotates the pod as desired. Then the robot carries the pod to the station. A replenishment station is shown in Figure 4.6. The worker fills the pod with the items. Each item’s barcode is first scanned with a barcode reader, and then the item is inserted in a shelf of the pod. The worker should also scan the shelf code in order to assign replenished items to the shelf. After the worker finishes these jobs, he/she informs the system by pressing a button. After that, the robot relocates the pod back to its place. Several parts of the replenishment task execution directly depend on the decision 42 Figure 4.5: Replenishment Task Execution Flowchart. Figure 4.6: Replenishment station [10]. 43 of the worker. Replenishment task starts and ends upon request of the worker. Moreover, the worker decides which items to locate in the pod, and also in which shelf. The system may limit number of replenishment items even though the pod has enough space. Several studies (such as [27, 28]) analyze effects of pod selection and inventory allocation in the replenishment process. These works show that classification of items and spreading the inventory across multiple pods will increase the pod selection options for order tasks and improve the order throughput performance. However, inventory classification significantly increases the complexity of the replenishment task. Inventory items should be known, classified and ordered. Although these additional efforts are mentioned in the related work, cost analysis is not performed and the trade-off between the extra effort and performance gain is not clear. In addition, the extra effort also requires extra resources in terms of equipment and workers, and extra rules. Optimization of the replenishment task by inventory allocation decisions is not in the main scope of this study. In order to increase algorithm efficiency in the picking process, we adopt replenishment rules that are easy to install with low cost of pre-processing. In our model, WES does not have to know replenishment items and their counts. Items on the replenishment station may be held on pallets or anywhere. The worker can choose any item that is easy to take. Pod selection rule is emptiest-first as mentioned in [6]. Item selection and shelf selection decisions are left to the worker. Most e-commerce companies sell some items although they are not stored in their warehouse. They should supply such items in a short time. In such warehouses, the replenished items typically include lots of already ordered items (of "stock-waiting" orders). This fact causes some challenges. Let us assume that there exists items of 100 stock-waiting orders in a replenishment task. The first question is how to locate these items to the pod. The second question is when to start collecting these orders. As mentioned above, WES chooses the pod with most available free space for replenishment task. However, allocating many items of stock-waiting orders in a single pod will cause some unwanted situations. WES will want to reserve 100 totes for 100 stock-waiting orders’ items stored in this pod. Nevertheless, pick stations have limited totes to collect orders simultaneously. Therefore managing such a highly desired pod will cause moving the pod to the pick station several times. In order to avoid this situation and 44 increase manageability of the pods, we limit replenishment count for each pod. This way, the already ordered items will be allocated to more pods and pod selection options will be increased during the order task selection process. At the first glance, it seems to be an effective solution to take the incoming pod directly to the pick station without placing it to the warehouse. If this pod contains all items of an order, this order will be completed quickly. However, in most cases this pod will only partially contain items of stock-waiting orders and it will cause creation of lots of order tasks. Managing large number of order tasks with a small number of totes is very difficult. Consequently, it is observed that taking replenishment items directly to the pick station reduces global system efficiency, although it may bring a local gain. To sum up, we limit number of items to be stored in a single pod in the replenishment stations with n p items. Furthermore, we relocate the pod to the warehouse without visiting the pick station even if it includes already ordered items. This pod will be converted to order task in the next OBRTC execution in the usual way. 4.2.4. Robot Life-cycle Automated mobile robots are physical components of WES. When a configured robot turns on, it sends a request to the WES by introducing itself. After handshaking, WES creates an entry for the robot in its database, if not exist. Robot life-cycle is shown in Figure 4.7. Robots wait in the robot pool until a new task is assigned to them. There are mainly three types of tasks: order, replenishment and charging tasks. WES will assign one of these three tasks to the robot according to the real-time dynamics of the warehouse. WES checks the battery level of the robot before assigning a task, If it is low, robot will go to the charging cell. It is charged up to a saturation level (which is defined as 85% of the full capacity in this study) if there is a waiting task in the system. Otherwise, it can wait until fully charging its battery. The replenishment task starts with the pod request of the worker in the station. However, there should be a limit for concurrent pod requests of a worker to avoid needless waiting of robots in the replenishment station. In this study, we have limited the maximum number of concurrently running replenishment tasks with the replenishment station count. If there are replenishment items but there is no robot in the replenishment station (i.e. 45 Figure 4.7: Robot Life-cycle. replenishment station is free), then WES creates a replenishment task and assigns it to the robot. Otherwise, WES assigns an order task to the robot with the order task selection method. The robot executes the order task as mentioned before. When the robot finishes its task, it informs the WES and asks for a new task. There is a separate thread running asynchronously for each robot. This avoids needless waiting of robots. A robot thread only waits if task selection process is running for another robot in order to avoid race conditions. Task assignment to a robot is done in less than a second, including all waiting time. Task assignment is one directional. WES assigns tasks, but a robot cannot select its task, it can just request from the system. WES evaluates this request and handles task selection according to real time dynamics. If there is no task in the pool, it sends the robots to charging even if their battery levels are not low. WES checks new orders and replenishment for new task every minute. This system guarantees completion of order and replenishment processes with high utilization of resources. 46 4.3. Simulation Study and Numerical Results 4.3.1. Simulation Environment Figure 2.5 illustrates the main components of WES. The database functions (insert, update, delete, select) and business logic (task planning, resource allocation, etc.) are performed by the Logic (Core) Service. It is a RESTful service so other components connect with it via HTTP. Logistic Service, Pick Station and Replenishment station screens, Manager Dashboard and Layer 3 Robot API are .Net Core C# web applications. Pick Station screen is used by pick worker. When the robot arrives at the station with the pod, this screen shows the commands to the pick worker for collecting the order items. Replenishment station screen is used by the replenishing worker for inventory allocation. The Manager Dashboard is for management of warehouse process and components. Reports and results are also shown in the Manager Dashboard. The 3D Viewer shows robot movements in the warehouse in 3 dimensions and it is coded with Tree.js 3D Javascript library and Vue.js. For each robot, we run a separate simulation application on the server as Robot Layer 3 API. Low level communication and process on the robot is simulated with realistic parameters. All the delays encountered while processing commands such as "go", "turn" or "lift" are considered in the simulations according to real measurements. Robot cannot access new command until finishing its last job. Robot charge management is also simulated. Robot charge decreases according to its state. For example, a robot will consume more energy when it picks a pod. All the parameters are set according to real measurements obtained from Robee of Makhina Robotics [10] shown in Figure 4.8. Robot path planning is implemented according to the modified collision free A* algorithm proposed in our previous work [9]. However in this study, we implement some modifications as follows. In the previous algorithm, there exists some directed paths such that robots are allowed to go only in single direction. We relaxed this restriction and allow robots to go in opposite direction by assigning 5 times more cost for this movement. Increasing the cost per one cell movement will result in the selection of such paths very rarely, only if it has significant gain. Allowing the opposite direction will increase possibility of head to head encountering but it decreases the total robot traveling time by 47 Figure 4.8: The physical robot (Robee of Makhina Robotics [10]) used in parameter setting. providing shorter options. Another modification is that robots are also allowed to go under the pods. Moving under the pods is not preferred because there is a restricted movement area and small deviations may cause stopping of the robots. Therefore we set the same cost as the opposite direction movement for moving under the pods. Path planning is out of the scope of this paper so we skip the details to [9]. We analyzed orders of a book e-commerce warehouse in a two months span. The mean of order line count for an order was 3,28. Approximately half of the daily orders included only a single item. We calculate the selling rate of the items. We create our simulation orders randomly using statistical characteristics of these orders, so that we accomplish realistic order line counts and items selling rate. Orders arrival to the WES is simulated with appropriate time intervals to analyze utilization of robots and pick stations with changing order density in time. We track and record state of all dynamic WES processes including robots, pods, shelves, items, orders, stations, and totes while simulation tests are running. Main assumptions of our simulation model are the following. (i) A pod can include up to 500 items. In general, we used approximately half of this 48 capacity. (ii) Orders can be single-line or multi-line. (iii) Robot’s velocity, acceleration/deceleration, and energy consumption will change according to the weight of the robot load (picked pod). We have simulated the weight effect according to whether the robot is loaded a pod or not. Loaded pod weight is set to 300 kg which is the half of the robot’s maximum load capacity. (iv) After an order completed in the tote at the pick station, a new free tote is located instead of the completed one. This process can be done separately without blocking picking processes. Hence, time of tote relocating is ignored. (v) Charging cell count is equal to the robot count so there is no queue for charging. (vi) Each picking and replenishment station has a single worker. (vii) 80% of our pods are two-sided and the rest are four-sided. The robot should rotate the pod to the desired side before the picking or replenishment process. (viii) Time elapsed for picking one item at the pick station is set to 8 seconds. Time elapsed for locating an SKU to a pod at the replenishment station is set to 6 seconds. Process time per single item for worker at the replenishment station will be less than the pick station because typically replenishment task is easier. Replenishment workers mostly allocate same SKU items at once with less effort and time. These times may change according to the physical environment, worker capabilities, etc. (ix) Totes are of the same size and they have enough space to contain items. (x) 10% of the orders arrive the system at the beginning. Remaining orders arrive the WES gradually in different time periods. We want to realize e-commerce orders such that there are different number of orders in the pool at different time periods. However, there will be always enough orders in the pool to keep robots busy until all orders are completed. (xi) Order arrival duration is divided into four time periods (for 2000 orders) and replenishment arrives as a bulk at the end of each period, if not otherwise stated. For more orders, number of time periods increase proportionally. 4.3.2. Numerical Results We simulate our algorithms on WES under various design decisions and parameters. We have analyzed our tests under 6 subtitles: (1) Effect of order batch size (nob ); (2) 49 Effect of the number of robots; (3) Robots’ charging time and workload balance; (4) Effect of the order count; (5) SKU effects on pile-on and time; (6) Effect of stock-waiting orders. All tests are performed for a warehouse where 624 pods are located in an area of 36 × 43 = 1548m2 . If not otherwise stated, there are 48272 SKU and 169343 items stored in the warehouse. The mean number of items per SKU is 3,5. 4.3.2.1. Order Batch Size. In the first set of experiments, we analyze effect of the proposed algorithm for various values of nob , which corresponds to order batch size. Tests are performed for 12 robots, 2000 orders, 6560 order items and 6204 replenished items. There exists 2 or 3 pick stations, and each pick station has 20 totes. We first simulated the system without applying OBRTC and ARTS methods. We set nob to one, in other words, each order is converted to a task individually and WES assigns these tasks to the robots randomly. In a warehouse with 2 pick stations (and 40 totes), 12 robots completed orders with 6560 items in 1637 minutes. The pile-on number is 1,3. For 3 pick stations (with 60 totes), same number of orders are completed in 1677 minutes. Then we applied the proposed OBRTC and ARTS methods by handling orders in batches. Initially we set nob to 10, and we increase it ten by ten in each new test in order to find the best value of nob . The results of tests can be seen in Figure 4.9. For 2 pick stations, the minimum time for collecting items (981 minutes) is obtained when nob is 70 for 40 totes. The pile-on values is 2,6. As mentioned before, achieving high pile-on value is very important to reduce total time for collecting orders. The results show that the proposed methodology (with an optimal nob selection) reduces the process time by 40% for a warehouse system with 2 pick stations. For a warehouse with 3 pick stations, the minimum completion time is found as 910 minutes when nob is set to 120. The pile-on value is 2,99. For this case, reduction in total time is 45,7%, which is even more than the case with 2 pick stations. The results shown in Figure 4.9 indicate that after an optimal value, further increasing the nob value would start increasing the order completion time. This is mainly because of the tote limitation. When there is no available tote in the pick station, pods will be returned back to the warehouse without all the items collected. Then they will be carried to the pick station again after totes become available. 50 Figure 4.9: Order Batch Size (nob ) vs Orders Completion Time nob value has also an important effect on the order departure time (ODT), i.e. the waiting time of an order until departing from the warehouse. The relation between nob and mean ODT is shown in Figure 4.10. It is observed that the best mean ODT value is obtained at the optimal nob value. Although the ROB algorithm changes FIFO order and items of an order may be spread over multiple order tasks, mean ODT value decreases with the batch size until the optimal nob value. Temporal priority in the ARTS method provides a balance to avoid long waiting for an order in order task pool. The maximum ODT, in other words, the longest waiting time of an order in the pool, is approximately double the mean ODT value. The mean and maximum values of ODT are 166 minutes and 305 minutes respectively when nob is 70 and tote count is 40. These values are 134 minutes (mean) and 260 minutes (max) for 60 totes. 4.3.2.2. Robot Count. We analyze the effect of robot count on the order completion time and pile-on. We test for 2 pick stations (with 8-16 robots), 3 pick stations (with 51 Figure 4.10: Order Batch Size (nob ) vs Average Order Departure Time 8-20 robots) and 4 pick stations (with 15-20 robots). nob is set to 70 for 2 pick stations and 120 for 3 and 4 pick stations. Figure 4.11 illustrates the effect of the robot count on the order completion time. Firstly, it is observed that completion time does not decrease linearly with the number of robots. The contribution of extra robot decreases as number of robots increase. This is because of extra waiting of robots on path and picking line. As an example result, a robot can carry 37 items per hour (on average) if 8 robots are employed, while 32 items can be carried per hour when 20 robots are employed (for 3 pick stations). Another result is related to the increment in the number of pick stations. Increasing number of totes increase optimization effect as mentioned before. However, there is a limit for number of totes in a pick station. Furthermore, opening new pick station has a cost of employing new worker. Hence, pick station count should be well optimized. Number of pick stations should be proportional to number of robots in order to increase utilization of pick workers. For example, using 3 or 4 pick stations is unnecessary for 8 robots, because robots will not be able to provide enough items to the pickers. Figure 4.11 shows that the difference in completion times for 40 and 60 totes (for the same number of robots) increases with the robot count. When 8 robots are employed, adding new 20 totes 52 Figure 4.11: Robot Count vs Orders Completion Time yields a 4% decrease in the completion time, while this ratio is (10%) when 16 robots are employed. Similar result is obtained when we compare the results for 60 and 80 totes. Decrease in the completion time is 6% for 16 robots and 9% for 20 robots. The effect of the robot count on the pile-on value is shown in Figure 4.12. It is observed that while the number of robots is increased, the pile-on value has been preserved with minor changes. 4.3.2.3. Robots’ Charging and Workload Balance. We analyze the balance in the workload and charging time of robots in the warehouse setting with 10 robots and 2 pick stations. All robots work for 42 hours (including charging times) and collect 13073 items. The results are shown in Table 4.1. The total covered distances and total charging times for all robots are very close to each other. Any robot spends approximately 14.5% of the time on charging. This indicates that all the robots consume almost same amount of energy. It can be realized that the balance in the workload and energy consumption of the robots comes with the nature of RMFS, where robots are employed anonymously and are busy most of the time. This result suggests that there is no need for extra load balancing effort (such as [31]) in the proposed system. 53 Figure 4.12: Robot Count vs Pile-on 4.3.2.4. Order Count. We increase number of orders up to 5000, and the number of replenished items in parallel. We test for 20 robots and 4 pick stations (80 totes). Table 4.2 illustrates the obtained results. It is observed that there is an approximately linear relation between order count and completion time. Furthermore, the pile-on value is preserved while the order count increases. For each case, a robot carries around 35 items per hour. 4.3.2.5. SKU Effects on Pile-on and Time. SKU and stock count have important effect on pile-on value. Increasing SKU decreases the pile-on value. Merschforman et al. [6] describe the negative effect of high SKU by performing tests for up to 10000 SKU. However, in the above tests, we analyze our algorithms for 48272 SKU which is approximately 5 times more than the maximum SKU value considered in [6]. We perform another set of tests after decreasing SKU to 22603 for approximately same amount of stock, and increasing stock per SKU from 3,51 to 7,53. The tests are performed for 2000 and 4000 orders, 12 robots and 3 pick stations. The results are shown in Table 4.3. When SKU is decreased, the pile-on value increases from 2,99 to 3,7 (for 2000 orders) and 3,02 to 3,78 (for 4000 orders). This corresponds to an increase of approximately 25%. As a result, 4000 orders (12868) are collected in 1815 − 1601 = 214 minutes less time after decreasing the SKU value. 54 Table 4.1: Robots’ Charging And Workload Balance Robot No Distance Covered (km) Charging Time / Total Time Robot 1 56,61 0,1459 Robot 2 56,38 0,1432 Robot 3 56,98 0,1472 Robot 4 56,56 0,1450 Robot 5 56,70 0,1455 Robot 6 56,10 0,1437 Robot 7 56,53 0,1453 Robot 8 56,48 0,1460 Robot 9 56,84 0,1456 Robot 10 55,96 0,1448 Average 56,51 0,1452 Table 4.2: Effect of Order Count Items/ Order Order Task Repl. Time Dist. Pile- Count Items Count Items (min) (km) on 2000 6560 2082 6204 551 259 3,15 35,72 3000 9757 3059 9491 837 390 3,19 34,97 4000 13073 4100 12511 1116 519 3,19 35,14 5000 16305 5144 15552 1402 650 3,17 34,89 55 Robot (hourly) Table 4.3: Effect of SKU Stock/ Order Order Task Repl. Time Dist. Pile- Items SKU Count Items Count Items (min) (km) on (hourly) 3,51 2000 6560 2197 6204 913 267 2,99 431,11 7,53 2000 6434 1741 5510 791 222 3,70 488,04 3,51 4000 13073 4330 12511 1815 526 3,02 432,17 7,53 4000 12868 3404 11059 1601 446 3,78 482,25 Table 4.4: Effect of Stock Waiting Orders Waiting Waiting Repl. Repl. Time Pile- Distance Orders Items Part. Items (min.) on (km) 0 0 4 6204 913 2,99 267 208 410 4 6204 983 2,64 290 281 619 2 6204 992 2,57 294 346 862 1 6204 1050 2,50 300 4.3.2.6. Effect of Stock-Waiting Orders. In the replenishment station, the maximum number of items that can be located in a pod is limited with n p in order to manage stock-waiting orders, as mentioned in Section 4.2.3. In all our tests, we set n p to 40. In order to test the system with more stock-waiting orders, we remove some items arbitrarily from the stock at the beginning. Tests are performed for 12 robots, 3 pick stations and 2000 orders (6560 items). We consider 3 different replenishment scenarios which result in different number of stock-waiting orders. Although all 3 scenarios have the same number of items to replenish as 6204, they differ in the time intervals that the replenishment arrives to the warehouse. In the first scenario, all the replenishment arrives as a bulk after arrival of all orders. In the second scenario, replenishment is partitioned into two parts. The first part arrives to the system in the middle and the second part arrives at the end. In the third scenario, replenishment is partitioned into four parts and each part arrives in equal time intervals. The effect of stock-waiting orders can be observed in Table 4.4. When stock-waiting 56 orders increase, total collection time of all orders also increase. This increase is aligned with expectations, because it causes lower utilization of the system while orders are waiting for stock, compared to the case where these order tasks are distributed throughout the day. Hence, this fact causes a decrease in the pile-on value. 57 5. CONCLUSION AND FUTURE WORK In this work, we propose a task planning approach for RMFS-based smart warehouses, with the aim of utilizing system resources efficiently and adaptively. We describe a novel algorithm for conversion of order batches to tasks. We also propose a heuristic model for assignment of tasks to robots. We consider various system dynamics such as location and battery level of robots, replenishment of new stock in various pods, utilization level of totes and time elapsed after generation of order tasks. We develop a highly realistic simulation environment which can be considered as a digital twin of the real system. We take into account all realistic considerations such as routing, queuing, turning, energy consumption and charging of the robots. We analyzed the total elapsed time, pile-on value, mean order waiting time and total distance covered by the robots. We realize crucial relations between all of these outputs, especially between pile-on value and total elapsed time. It is shown that the proposed algorithms preserve the pile-on value for various number of robots and orders, and work well under different situations even in high SKU. The obtained results show that the reduction in total completion time of orders is about 40% in a warehouse with 2 pick stations and 40 totes, and about 46% in a warehouse with 3 pick stations and 60 totes. We provide a collision-free path planning algorithm for mobile robots on a warehouse grid. We introduce a modified A* algorithm which considers turning costs of the robots. We also consider deadlock and no-path scenarios. We test the proposed solutions in various scenarios with different number of robots and various pod alignment plans. Test results provide valuable information about smart warehouse design decisions under different warehouse operation priorities. Through extensive set of simulations, we observed that the provided collision-free path planning algorithm operates without any error in acceptable time in various warehouse design cases and different number of robots. Although this work gives significant insight into how to design an efficient RMFS-based warehouse system, there are always open directions for research. Using learning techniques for path planning to avoid collisions and deadlocks in a warehouse with less or no restriction rules can be performed. Moreover, in the order-to-task conversion step, orders are handled with the order they arrive in this study. As a future work, an additional optimization study can be performed while selecting orders into 58 the order batch. Another idea is to allow allocation of temporary totes in pick stations when there is no available tote for some of the order items in a coming pod. This may increase efficiency but also increase complexity due to additional process for workers. Performance effects and induced costs of these further improvements remain to be discovered. 59 REFERENCES [1] Peter R Wurman, Raffaello D’Andrea, and Mick Mountz. “Coordinating hundreds of cooperative, autonomous vehicles in warehouses”. In: AI magazine 29.1 (2008), pp. 9–9. [2] Nils Boysen, René de Koster, and Felix Weidinger. “Warehousing in the e-commerce era: A survey”. In: European Journal of Operational Research 277.2 (2019), pp. 396–411. [3] Steve Banker. Robots In The Warehouse: It’s Not Just Amazon. 2016. URL: https : / / www . forbes . com / sites / stevebanker / 2016 / 01 / 11 / robots - in - the - warehouse - its - not - just - amazon / ?sh = d7f4dcc40b8f (visited on 07/09/2020). [4] Tim Lamballais, Debjit Roy, and MBM De Koster. “Estimating performance in a robotic mobile fulfillment system”. In: European Journal of Operational Research 256.3 (2017), pp. 976–990. [5] Guni Sharon et al. “Conflict-based search for optimal multi-agent pathfinding”. In: Artificial Intelligence 219 (2015), pp. 40–66. [6] Marius Merschformann et al. “Decision rules for robotic mobile fulfillment systems”. In: Operations Research Perspectives 6 (2019), p. 100128. [7] Jun Zhang, Xuping Wang, and Kai Huang. “Integrated on-line scheduling of order batching and delivery under B2C e-commerce”. In: Computers & Industrial Engineering 94 (2016), pp. 280–289. [8] John J Enright and Peter R Wurman. “Optimization and coordinated autonomy in mobile fulfillment systems”. In: Workshops at the twenty-fifth AAAI conference on artificial intelligence. 2011. [9] Ali Bolu and Ömer Korçak. “Path Planning for Multiple Mobile Robots in Smart Warehouse”. In: 2019 7th International Conference on Control, Mechatronics and Automation (ICCMA). IEEE. 2019, pp. 144–150. [10] Boost your warehouse with Robee. https://www.makhina.com/product. Accessed: 2020-12-25. 60 [11] Ming Zhang and Kan Yu. “Wireless communication technologies in automated guided vehicles: Survey and analysis”. In: IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society. IEEE. 2018, pp. 4155–4161. [12] Tongjuan Liu, Jun Liu, and Bingwu Liu. “Design of intelligent warehouse measure and control system based on Zigbee WSN”. In: 2010 IEEE International Conference on Mechatronics and Automation. IEEE. 2010, pp. 888–893. [13] Aritra Bhuiya, Aditya Mukherjee, and Ranjit Kumar Barai. “Development of wi-fi communication module for atmega microcontroller based mobile robot for cooperative autonomous navigation”. In: 2017 IEEE Calcutta Conference (CALCON). IEEE. 2017, pp. 168–172. [14] Taichi Kumagai, Hiroshi Yoshida, and Kozo Satoda. “Adaptive remote control of a mobile robot system with delay fluctuation”. In: 2018 IEEE International Conference on Consumer Electronics (ICCE). IEEE. 2018, pp. 1–5. [15] Zoltán Szántó et al. “Performance analysis of WLAN based mobile robot teleoperation”. In: 2013 IEEE 9th International Conference on Intelligent Computer Communication and Processing (ICCP). IEEE. 2013, pp. 299–305. [16] Jetmir Haxhibeqiri et al. “Flexible Wi-Fi communication among mobile robots in indoor industrial environments”. In: Mobile Information Systems 2018 (2018). [17] Noor Azurati Ahmad Salleh and Rosbi Mamat. “Implementation of embedded Web server for mobile robot system”. In: 2007 International Conference on Intelligent and Advanced Systems. IEEE. 2007, pp. 1324–1329. [18] Zhe Liu et al. “Coordinating Large-Scale Robot Networks with Motion and Communication Uncertainties for Logistics Applications”. In: arXiv preprint arXiv:1904.01303 (2019). [19] Hajir Roozbehani and Raffaello D’Andrea. “Adaptive highways on a grid”. In: Robotics Research. Springer, 2011, pp. 661–680. [20] Aleksandr I Panov, Konstantin S Yakovlev, and Roman Suvorov. “Grid path planning with deep reinforcement learning: Preliminary results”. In: Procedia computer science 123 (2018), pp. 347–353. 61 [21] Jiajia Dou, Chunlin Chen, and Pei Yang. “Genetic scheduling and reinforcement learning in multirobot systems for intelligent warehouses”. In: Mathematical Problems in Engineering 2015 (2015). [22] František Duchoň et al. “Path planning with modified a star algorithm for a mobile robot”. In: Procedia Engineering 96 (2014), pp. 59–69. [23] Lin Xie et al. “Efficient order picking methods in robotic mobile fulfillment systems”. In: arXiv preprint arXiv:1902.03092 (2019). [24] Nils Boysen, Dirk Briskorn, and Simon Emde. “Parts-to-picker based order processing in a rack-moving mobile robots environment”. In: European Journal of Operational Research 262.2 (2017), pp. 550–562. [25] Bipan Zou et al. “Assignment rules in robotic mobile fulfilment systems for online retailers”. In: International Journal of Production Research 55.20 (2017), pp. 6175–6192. [26] Robin Hanson, Lars Medbo, and Mats I Johansson. “Performance characteristics of robotic mobile fulfilment systems in order picking applications”. In: IFAC-PapersOnLine 51.11 (2018), pp. 1493–1498. [27] Felix Weidinger, Nils Boysen, and Dirk Briskorn. “Storage assignment with rack-moving mobile robots in KIVA warehouses”. In: Transportation Science 52.6 (2018), pp. 1479–1495. [28] Tim Lamballais, Debjit Roy, and MBM De Koster. “Inventory allocation in robotic mobile fulfillment systems”. In: Available at SSRN 2900940 (2017). [29] Debjit Roy et al. “Robot-storage zone assignment strategies in mobile fulfillment systems”. In: Transportation Research Part E: Logistics and Transportation Review 122 (2019), pp. 119–142. [30] Zhe Yuan and Yeming Yale Gong. “Bot-in-time delivery for robotic mobile fulfillment systems”. In: IEEE Transactions on Engineering Management 64.1 (2017), pp. 83–93. 62 [31] Luowei Zhou et al. “A balanced heuristic mechanism for multirobot task allocation of intelligent warehouses”. In: Mathematical Problems in Engineering 2014 (2014). 63 CURRICULUM VITAE Personal Information Surname, Name : BOLU, Ali Nationality : T.C. Date and Place of Birth : 24.10.1990, Bursa Phone number : 0 (537) 880 4572 E-mail : alibolu@marun.edu.tr Biography Ali Bolu received his B.Sc. degree in Telecommunication Engineering from the Istanbul Technical University, Istanbul, Turkey, in 2015. He is currently pursuing an M.Sc. degree in Computer Engineering at Marmara University, Istanbul, Turkey.His research interests include robotics systems and their optimizations. He has been working on robotic software technologies since the beginning of 2018, coping with math, optimization, and algorithmic challenges. Education Degree School/ Program Graduation Date MSc Marmara University / Computer Engineering Ongoing Undergraduate ITU / Telecommunication Engineering 2015 High School Bursa Anatolian High School 2009 Professional Experience Year Place of Work 2020-Ongoing Delivers.ai 2018-2020 Makhina Robotics 2015-2017 Techlife Information Technology. Foreign Language English Position Robotics Software Engineer . Software Leader Software Engineer Publications Ali Bolu and Ömer Korçak. “Path Planning for Multiple Mobile Robots in Smart Warehouse”. In:2019 7th International Conference on Control, Mechatronics and Automation (ICCMA). IEEE. 2019, pp. 144–150. Ali Bolu and Ömer Korçak. “Adaptive Task Planning for Multi-Robot Smart Warehouse”. IEEE Access (2021), Submitted