research-article

Public Access

Cooperative Route Planning Framework for Multiple Distributed Assets in Maritime Applications

Authors:

Sepideh Nikookar,

Paras Sakharkar,

Sathyanarayanan Somasunder,

Senjuti Basu Roy,

Adam Bienkowski,

Matthew Macesker,

Krishna R. Pattipati,

David SidotiAuthors Info & Claims

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

Pages 1518 - 1527

https://doi.org/10.1145/3514221.3526131

Published: 11 June 2022 Publication History

Abstract

This work formalizes the Route Planning Problem (RPP), wherein a set of distributed assets (e.g., ships, submarines, unmanned systems) simultaneously plan routes to optimize a team goal (e.g., find the location of an unknown threat or object in minimum time and/or fuel consumption) while ensuring that the planned routes satisfy certain constraints (e.g., avoiding collisions and obstacles). This problem becomes overwhelmingly complex for multiple distributed assets as the search space grows exponentially to design such plans. The RPP is formalized as a Team Discrete Markov Decision Process (TDMDP) and we propose a Multi-agent Multi-objective Reinforcement Learning (MaMoRL) framework for solving it. We investigate challenges in deploying the solution in real-world settings and study approximation opportunities. We experimentally demonstrate MaMoRL's effectiveness on multiple real-world and synthetic grids, as well as for transfer learning. MaMoRL is deployed for use by the Naval Research Laboratory - Marine Meteorology Division (NRL-MMD), Monterey, CA.

Supplemental Material

PDF File

Read me

Download
19.42 KB

ZIP File

Source Code

Download
10.10 MB

References

[1]

Gopi Vinod Avvari, David Sidoti, Lingyi Zhang, Manisha Mishra, Krishna Pattipati, Charles R Sampson, and James Hansen. 2018. Robust multi-objective asset routing in a dynamic and uncertain environment. In 2018 IEEE Aerospace Conference. IEEE, 1--9.

[2]

Ori Bar El, Tova Milo, and Amit Somech. 2020. Automatically generating data exploration sessions using deep reinforcement learning. In Proceedings of 2020 ACM SIGMOD International Conference on Management of Data. 1527--1537.

[3]

Nicolas Bialystocki and Dimitris Konovessis. 2016. On the estimation of ship's fuel consumption and speed curve: a statistical approach. Journal of Ocean Engineering and Science 1, 2 (2016), 157--166.

[4]

Adam Bienkowski et al. 2018. Path Planning in an Uncertain Environment Using Approximate Dynamic Programming Methods. In FUSION. IEEE.

[5]

Lucian Bu?oniu et al. 2011. Approximate reinforcement learning: An overview. In ADPRL.

[6]

Lucian Bu?oniu, Robert Babu?ka, and Bart De Schutter. 2010. Multi-agent reinforcement learning: An overview. Innovations in multi-agent systems and applications-1 (2010), 183--221.

[7]

Gang Chen, Zhonghua Yang, Hao He, and Kiah Mok Goh. 2005. Coordinating multiple agents via reinforcement learning. Autonomous Agents and Multi-Agent Systems 10, 3 (2005), 273--328.

Digital Library

[8]

Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. 2010. Generalized learning automata for multi-agent reinforcement learning. Ai Communications 23, 4 (2010), 311--324.

Digital Library

[9]

Edsger WDijkstra et al. 1959. A note on two problems in connexion with graphs. Numerische mathematik 1, 1 (1959), 269--271.

[10]

Javier Garc?a and Fernando Fernández. 2015. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research 16, 1 (2015), 1437-- 1480.

[11]

Peter Geibel. 2006. Reinforcement learning for MDPs with constraints. In European Conference on Machine Learning. Springer, 646--653.

Digital Library

[12]

Christophe Geuzaine and Jean-Francois Remacle. 2009. Gmsh: a threedimensional finite element mesh generator with built-in pre- and post-processing facilities. Internat. J. Numer. Methods Engrg. 79, 11 (2009), 1309--1331.

[13]

Henry Hsu and Peter A Lachenbruch. 2014. Paired t test. Wiley StatsRef: statistics reference online (2014), book.

[14]

Yoshinobu Kadota, Masami Kurano, and Masami Yasuda. 2006. Discounted Markov decision processes with utility constraints. Computers & Mathematics with Applications 51, 2 (2006), 279--284.

Digital Library

[15]

Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research 4 (1996), 237--285.

Digital Library

[16]

Francisco S Melo, Sean P Meyn, and M Isabel Ribeiro. 2008. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th international conference on Machine learning. 664--671.

Digital Library

[17]

Manisha Mishra, David Sidoti, Gopi Vinod Avvari, Pujitha Mannaru, Diego Fernando Martínez Ayala, Krishna R. Pattipati, and David L. Kleinman. 2017. A Context-Driven Framework for Proactive Decision Support With Applications. IEEE Access 5, 12475--12495.

[18]

S. Nikookar, P. Sakharkar, B. Smagh, S. Amer-Yahia, and S. Basu Roy. 2022. Guided Task Planning Under Complex Constraints. In 38th IEEE International Conference on Data Engineering. accepted for publication.

[19]

Edwin Pednault et al. 2002. Sequential cost-sensitive decision making with reinforcement learning. In SIGKDD.

[20]

Martin L Puterman. 2014. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.

[21]

Mariia Seleznova, Behrooz Omidvar-Tehrani, Sihem Amer-Yahia, and Eric Simon. 2020. Guided exploration of user groups. Proceedings of the VLDB Endowment (PVLDB) 13, 9 (2020), 1469--1482.

Digital Library

[22]

David Sidoti et al. 2016. A multiobjective path-planning algorithm with time windows for asset routing in a dynamic weather-impacted environment. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2016).

[23]

Da Silva et al. 2018. Autonomously Reusing Knowledge in Multiagent Reinforcement Learning. In IJCAI. 5487--5493.

[24]

Richard S Sutton et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPs.

[25]

HuaWei et al. 2018. Intellilight: A reinforcement learning approach for intelligent traffic light control. In SIGKDD.

[26]

Paul Wessel et al. 1996. A global, self-consistent, hierarchical, high-resolution shoreline database. Journal of Geophysical Research (1996).

[27]

Yang Yu. 2018. Towards Sample Efficient Reinforcement Learning. In IJCAI. 5739--5743.

[28]

Pucheng Zhou et al. 2011. Multi-agent cooperation by reinforcement learning with teammate modeling and reward allotment. In FSKD, Vol. 2. 1316--1319.

[29]

Xinyuan Zhou, Peng Wu, Haifeng Zhang, Weihong Guo, and Yuanchang Liu. 2019. Learn to navigate: cooperative path planning for unmanned surface vehicles using deep reinforcement learning. IEEE Access 7 (2019), 165262--165278.

Cited By

Nikookar S(2023)Human-AI Complex Task Planning2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00382(3923-3927)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00382
Rai RNakashima TNishino N(2023)Port Arrival Reservation System Using Auctions for Fuel Consumption Reduction2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386620(3212-3221)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386620
Altan DMarijan DKholodna T(2023)SafeWay: Improving the safety of autonomous waypoint detection in maritime using transformer and interpolationMaritime Transport Research10.1016/j.martra.2023.1000864(100086)Online publication date: Jun-2023
https://doi.org/10.1016/j.martra.2023.100086

Index Terms

Cooperative Route Planning Framework for Multiple Distributed Assets in Maritime Applications
1. Information systems
  1. Data management systems
    1. Database design and models

Recommendations

Online Route Replanning for Scalable System-Optimal Route Planning
SIGSPATIAL '21: Proceedings of the 29th International Conference on Advances in Geographic Information Systems

Route planning in transportation networks is typically performed as a single optimization at trip departure. In this paper, we consider the impact of within-trip replanning on the performance of the overall network in a fully-algorithmic route selection ...
Multi‐AGV route planning in automated warehouse system based on shortest‐time Q‐learning algorithm
Abstract
Route planning for automated guided vehicles (AGVs) is one of the key factors that affects work efficiency of automated storage and retrieval systems (AS/RSes). Route planning plays an important role in the operation of AGVs. Since the ...
Multiple objective optimisation applied to route planning
GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computation

This paper presents an evaluation of the benefits of multi-objective optimisation algorithms, compared to single objective optimisation algorithms, when applied to the problem of planning a route over an unstructured environment, where a route has a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

June 2022

2597 pages

ISBN:9781450392495

DOI:10.1145/3514221

General Chair:
Zachary Ives
University of Pennsylvania (USA)
,
Program Chairs:
Angela Bonifati
Lyon 1 University (France)
,
Amr El Abbadi
University of California, Santa Barbara (USA)

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SIGMOD/PODS '22

Sponsor:

SIGMOD

SIGMOD/PODS '22: International Conference on Management of Data

June 12 - 17, 2022

PA, Philadelphia, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
401
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)34

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nikookar S(2023)Human-AI Complex Task Planning2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00382(3923-3927)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00382
Rai RNakashima TNishino N(2023)Port Arrival Reservation System Using Auctions for Fuel Consumption Reduction2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386620(3212-3221)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386620
Altan DMarijan DKholodna T(2023)SafeWay: Improving the safety of autonomous waypoint detection in maritime using transformer and interpolationMaritime Transport Research10.1016/j.martra.2023.1000864(100086)Online publication date: Jun-2023
https://doi.org/10.1016/j.martra.2023.100086

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents