research-article

When digital twin meets deep reinforcement learning in multi-UAV path planning

Authors:

Siyuan Li,

Xi Lin,

Jun Wu,

Ali Kashif Bashir,

Raheel NawazAuthors Info & Claims

DroneCom '22: Proceedings of the 5th International ACM Mobicom Workshop on Drone Assisted Wireless Communications for 5G and Beyond

Pages 61 - 66

https://doi.org/10.1145/3555661.3560865

Published: 24 October 2022 Publication History

Get Access

Abstract

Unmanned aerial vehicles (UAVs) path planning is one of the promising technologies in the fifth-generation wireless communications. The gap between simulation and reality limits the application of deep reinforcement learning (DRL) in UAV path planning. Therefore, we propose a digital twin-based deep reinforcement learning training framework. With the help of digital twin, DRL model can be trained more effectively deployed to real UAVs. In this training framework, we propose a deep deterministic policy gradient (DDPG) based multi-UAV path planning algorithm. Based on decomposed actor structure in DRL, we design a pooling-based combined LSTM network to better understand different state information in a multi-UAV path planning task. Moreover, we also establish a digital twin platform for multi-UAV system, which has a high degree of simulation and visualization. The simulation result shows that the proposed algorithm can achieve higher mean rewards, and outperforms DDPG in average arrival rate by more than 30%.

References

[1]

Muhammad Atif, Rizwan Ahmad, Waqas Ahmad, Liang Zhao, and Joel JPC Rodrigues. 2021. UAV-assisted wireless localization for search and rescue. IEEE Systems Journal 15, 3 (2021), 3261--3272.

Crossref

Google Scholar

[2]

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. 2018. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Salt Lake City, 2255--2264.

Crossref

Google Scholar

[3]

Siyi Liao, Jun Wu, Ali Kashif Bashir, Wu Yang, Jianhua Li, and Usman Tariq. 2021. Digital Twin Consensus for Blockchain-Enabled Intelligent Transportation Systems in Smart Cities. IEEE Transactions on Intelligent Transportation Systems (2021), 1--11.

Crossref

Google Scholar

[4]

Xi Lin, Jun Wu, Jianhua Li, Wu Yang, and Mohsen Guizani. 2021. Stochastic Digital-Twin Service Demand with Edge Response: An Incentive-Based Congestion Control Approach. IEEE Transactions on Mobile Computing (2021), 1--1.

Digital Library

Google Scholar

[5]

Q. Luo and H. Duan. 2017. Distributed UAV flocking control based on homing pigeon hierarchical strategies. Aerospace science and technology 70, nov. (2017), 257--264.

Google Scholar

[6]

S. Maliah and G. Shani. 2021. Using POMDPs for learning cost sensitive decision trees. Artificial Intelligence 292 (2021), 103400.

Crossref

Google Scholar

[7]

Qingfei Min, Yangguang Lu, Zhiyong Liu, Chao Su, and Bo Wang. 2019. Machine Learning based Digital Twin Framework for Production Optimization in Petrochemical Industry. International Journal of Information Management 49 (2019), 502--519.

Digital Library

Google Scholar

[8]

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, et al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature 588, 7839 (2020), 604--609.

Google Scholar

[9]

Gaoqing Shen, Lei Lei, Zhilin Li, Shengsuo Cai, Lijuan Zhang, Pan Cao, and Xiaojiao Liu. 2022. Deep Reinforcement Learning for Flocking Motion of Multi-UAV systems: Learn from a Digital Twin. IEEE Internet of Things Journal 9, 13 (2022), 11141--11153.

Crossref

Google Scholar

[10]

H. Su, X. Wang, and Z. Lin. 2009. Flocking of Multi-Agents With a Virtual Leader. IEEE Trans. Automat. Control 54, 2 (2009), 293--307.

Crossref

Google Scholar

[11]

R. Sutton and A. Barto. 2018. Reinforcement Learning:An Introduction. MIT Press.

Google Scholar

[12]

L. Winterer, S. Junges, R. Wimmer, N. Jansen, and B. Becker. 2020. Strategy Synthesis for POMDPs in Robot Planning via Game-Based Abstractions. IEEE Trans. Automat. Control PP, 99 (2020), 1--1.

Google Scholar

Cited By

View all

Li SLin XLiu YLi GLi J(2024)OpticGAI: Generative AI-aided Deep Reinforcement Learning for Optical Networks OptimizationProceedings of the 1st SIGCOMM Workshop on Hot Topics in Optical Technologies and Applications in Networking10.1145/3672201.3674119(1-6)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672201.3674119
Liu XDavid ICombemale BWimmer MChechik MEgyed A(2024)AI Simulation by Digital Twins: Systematic Survey of the State of the Art and a Reference FrameworkProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688253(401-412)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3688253
Sicard BMcCafferty-Leroux AAppuhamy RNewton AKosierb PGadsden S(2024)Aerospace digital twins: examining applications of digital twin technology to unmanned aerial vehicles and satellitesAutonomous Systems: Sensors, Processing, and Security for Ground, Air, Sea, and Space Vehicles and Infrastructure 202410.1117/12.3013115(5)Online publication date: 7-Jun-2024
https://doi.org/10.1117/12.3013115
Show More Cited By

Index Terms

When digital twin meets deep reinforcement learning in multi-UAV path planning
1. Computing methodologies
  1. Artificial intelligence
    1. Planning and scheduling
      1. Multi-agent planning
2. Networks
  1. Network types
    1. Cyber-physical networks

Recommendations

Real-time path planning of controllable UAV by subgoals using goal-conditioned reinforcement learning
Abstract
The conventional path planning problem for an unmanned aerial vehicle (UAV) typically involves a pre-defined environment and mission, with the objective of reaching a single target point. However, in order to perform different missions,...
Highlights
- We propose a real-time autonomous path planning of UAVs in unknown environments.
Deep Reinforcement Learning Based UAV Mission Planning with Charging Module
CNIOT '23: Proceedings of the 2023 4th International Conference on Computing, Networks and Internet of Things

This paper focuses on UAV mission planning in an environment with charging using deep reinforcement learning, firstly proposing an improved actor-critic algorithm and constructing a reward function related to the UAV energy as well as the mission to ...
Path planning in an unknown environment based on deep reinforcement learning with prior knowledge

Path planning in an unknown environment is a basic task for mobile robots to complete tasks. As a typical deep reinforcement learning, deep Q-network (DQN) algorithm has gained wide popularity in path planning tasks due to its self-learning and ...

Comments

Information & Contributors

Information

Published In

DroneCom '22: Proceedings of the 5th International ACM Mobicom Workshop on Drone Assisted Wireless Communications for 5G and Beyond

October 2022

138 pages

ISBN:9781450395144

DOI:10.1145/3555661

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACM MobiCom '22

Sponsor:

SIGMOBILE

ACM MobiCom '22: The 28th Annual International Conference on Mobile Computing and Networking

October 17, 2022

NSW, Sydney, Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
418
Total Downloads

Downloads (Last 12 months)165
Downloads (Last 6 weeks)27

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Li SLin XLiu YLi GLi J(2024)OpticGAI: Generative AI-aided Deep Reinforcement Learning for Optical Networks OptimizationProceedings of the 1st SIGCOMM Workshop on Hot Topics in Optical Technologies and Applications in Networking10.1145/3672201.3674119(1-6)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672201.3674119
Liu XDavid ICombemale BWimmer MChechik MEgyed A(2024)AI Simulation by Digital Twins: Systematic Survey of the State of the Art and a Reference FrameworkProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688253(401-412)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3688253
Sicard BMcCafferty-Leroux AAppuhamy RNewton AKosierb PGadsden S(2024)Aerospace digital twins: examining applications of digital twin technology to unmanned aerial vehicles and satellitesAutonomous Systems: Sensors, Processing, and Security for Ground, Air, Sea, and Space Vehicles and Infrastructure 202410.1117/12.3013115(5)Online publication date: 7-Jun-2024
https://doi.org/10.1117/12.3013115
Zhou LLeng SQuek T(2024)Hierarchical Digital-Twin-Enhanced Cooperative Sensing for UAV SwarmsIEEE Internet of Things Journal10.1109/JIOT.2024.342847611:20(33204-33216)Online publication date: 15-Oct-2024
https://doi.org/10.1109/JIOT.2024.3428476
Sun KWu JBashir ALi JXu HPan QAl-Otaibi Y(2024)Personalized Privacy-Preserving Distributed Artificial Intelligence for Digital-Twin-Driven Vehicle Road CooperationIEEE Internet of Things Journal10.1109/JIOT.2024.338965611:22(35902-35916)Online publication date: 15-Nov-2024
https://doi.org/10.1109/JIOT.2024.3389656
Sarantinoudis NVitzilaios NArampatzis G(2024)Applications of Digital Twins in UAVs2024 International Conference on Unmanned Aircraft Systems (ICUAS)10.1109/ICUAS60882.2024.10556896(450-457)Online publication date: 4-Jun-2024
https://doi.org/10.1109/ICUAS60882.2024.10556896
Yu NZhao HHe X(2024)SAPO: A UAV path optimization method through situation assessment2024 36th Chinese Control and Decision Conference (CCDC)10.1109/CCDC62350.2024.10587411(1161-1166)Online publication date: 25-May-2024
https://doi.org/10.1109/CCDC62350.2024.10587411
Feng SLi QYang YWu FChen G(2024)GIPUT: Maximizing Photo Coverage Efficiency for UAV TrajectoryWeb and Big Data10.1007/978-981-97-7232-2_26(391-406)Online publication date: 28-Aug-2024
https://doi.org/10.1007/978-981-97-7232-2_26
Aboueleneen NAlwarafy AAbdallah M(2023)Deep Reinforcement Learning for Internet of Drones Networks: Issues and Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2023.32518554(671-683)Online publication date: 2023
https://doi.org/10.1109/OJCOMS.2023.3251855
Li SLin XWu JZhang WLi J(2023)Digital Twin and Artificial Intelligence-Empowered Panoramic Video Streaming: Reducing Transmission Latency in the Extended Reality-Assisted Vehicular MetaverseIEEE Vehicular Technology Magazine10.1109/MVT.2023.332117218:4(56-65)Online publication date: Dec-2023
https://doi.org/10.1109/MVT.2023.3321172

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Real-time path planning of controllable UAV by subgoals using goal-conditioned reinforcement learning

Deep Reinforcement Learning Based UAV Mission Planning with Charging Module

Path planning in an unknown environment based on deep reinforcement learning with prior knowledge

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Real-time path planning of controllable UAV by subgoals using goal-conditioned reinforcement learning

Deep Reinforcement Learning Based UAV Mission Planning with Charging Module

Path planning in an unknown environment based on deep reinforcement learning with prior knowledge

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations