Autonomous Obstacle Avoidance and Target Tracking of UAV Based on Deep Reinforcement Learning

https://doi.org/10.1007/s10846-022-01601-8

Journal: Journal of Intelligent & Robotic Systems, 2022, № 4

Publisher: Springer Science and Business Media LLC

Authors: Guoqiang Xu, Weilai Jiang, Zhaolei Wang, Yaonan Wang

Funders

Young Scientists Fund
Key Programme

List of references

Shirani, B., Najafi, M., Izadi, I.: Cooperative load transportation using multiple UAVs. Proc. Aerosp. Sci. Technol. 84, 158–169 (2019). https://doi.org/10.1016/j.ast.2018.10.027
https://doi.org/10.1016/j.ast.2018.10.027
Khan, M.A., Cheema, T.A., Ullah, I., Noor, F., Aziz, M.A.: A dual-mode medium access control mechanism for UAV-enabled intelligent transportation system. Proc. Mob. Inf. Syst. (2021). https://doi.org/10.1155/2021/5578490
https://doi.org/10.1155/2021/5578490
Sung, I., Nielsen, P.: Zoning a service area of unmanned aerial vehicles for package delivery services. Proc. J. Intel. Robot. Syst. 97, 719–731 (2020)
https://doi.org/10.1007/s10846-019-01045-7
Umemoto, K., Endo, T., Matsuno, F.: Dynamic cooperative transportation control using friction forces of n multi-rotor unmanned aerial vehicles. Proc. J. Intell. Robot. Syst. 100, 1085–1095 (2020). https://doi.org/10.1007/s10846-020-01212-1
https://doi.org/10.1007/s10846-020-01212-1
Liu, X., Ansari, N.: Resource allocation in UAV-assisted M2M communications for disaster rescue. Proc. IEEE Wirel. Commun. Lett. 8(2), 580–583 (2018)
https://doi.org/10.1109/LWC.2018.2880467
Wang, Y., Su, Z., Xu, Q., Li, R., Luan, T.H.: Lifesaving with Rescuechain: Energy-Efficient and Partition-Tolerant Blockchain Based Secure Information Sharing for UAV-Aided Disaster Rescue. In: Proceeding of IEEE Conference on Computer Communications (2021)
Dong, J., Ota, K., Dong, M.: UAV-Based Real-Time Survivor Detection System in Post-Disaster Search and Rescue Operations. In: Proceeding of IEEE Journal on Miniaturization for Air and Space Systems (2021)
Stampa, M., Sutorma, A., Jahn, U., Thiem, J., Wolff, C., Röhrig, C.: Maturity levels of public safety applications using unmanned aerial systems: a review. Proc. J. Intel. Robot. Syst. 103(1), 1–15 (2021). https://doi.org/10.1007/s10846-021-01462-7
https://doi.org/10.1007/s10846-021-01462-7
Lyu, J., Zeng, Y., Zhang, R., Lim, T.J.: Placement optimization of UAV-mounted mobile base stations. Proc. IEEE Commun. Lett. 21(3), 604–607 (2016)
https://doi.org/10.1109/LCOMM.2016.2633248
Wu, Y., Yang, W., Guan, X., Wu, Q.: UAV-Enabled Relay Communication under Malicious Jamming: Joint Trajectory and Transmit Power Optimization. Proc. IEEE Trans. Veh. Technol. (2021)
https://doi.org/10.1109/TVT.2021.3089158
Cetin, O., Zagli, I., Yilmaz, G.: Establishing obstacle and collision free communication relay for UAVs with artificial potential fields. Proc. J. Intel. Robot. Syst. 69(1), 361–372 (2013). https://doi.org/10.1007/s10846-012-9761-y
https://doi.org/10.1007/s10846-012-9761-y
Oh, D., Lim, J., Lee, J.K., Baek, H.: Airborne-relay-based algorithm for locating crashed UAVs in GPS-denied environments. In: Proceeding of 2019 IEEE 10th annual ubiquitous computing, Electronics & Mobile Communication Conference (UEMCON). IEEE (2019)
Huang, Z., Zhang, T., Liu, P., Lu, X.: Outdoor independent charging platform system for power patrol UAV. In: Proceeding of 2020 12th IEEE PES Asia-Pacific Power and Energy Engineering Conference (APPEEC), pp. 1–5. IEEE (2020)
Chang, A., Jiang, M., Nan, J., Zhou, W., Li, X., Wang, J., He, X.: Research on the application of computer track planning algorithm in UAV power line patrol system. In: Proceeding of Conference Series (Vol. 1915, No. 3, p. 032030). IOP Publishing (2021)
Pham, H.X., La, H.M., Feil-Seifer, D., Nguyen, L.V.: Autonomous UAV navigation using reinforcement learning. arXiv 2018. arXiv preprint arXiv:1801.05086 (2018)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature. 518(7540), 529–533 (2015)
https://doi.org/10.1038/nature14236
Yan, C., Xiang, X., Wang, C.: Towards real-time path planning through deep reinforcement learning for a UAV in dynamic environments. Proc. J. Intel. Robot. Syst. 98, 297–309 (2020). https://doi.org/10.1007/s10846-019-01073-3
https://doi.org/10.1007/s10846-019-01073-3
Yao, Q., Zheng, Z., Qi, L., Yuan, H., Guo, X., Zhao, M., Liu, Z., Yang, T.: Path planning method with improved artificial potential field—a reinforcement learning perspective. Proc. IEEE Access. 8, 135513–135523 (2020)
https://doi.org/10.1109/ACCESS.2020.3011211
Hausknecht, M., Stone, P.: Deep Recurrent Q-Learning for Partially Observable MDPs. In: Proceeding of AAAI Fall Symposium Series (2015)
Singla, A., Padakandla, S., Bhatnagar, S.: Memory-Based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledge. Proc. IEEE Trans. Intell. Transp. Syst. (2019)
Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., … Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Rodriguez-Ramos, A., Sampedro, C., Bavle, H., De La Puente, P., Campoy, P.: A deep reinforcement learning strategy for UAV autonomous landing on a moving platform. Proc J Intel Robot Syst. 93(1–2), 351–366 (2019). https://doi.org/10.1007/s10846-018-0891-8
https://doi.org/10.1007/s10846-018-0891-8
Li, B., Yang, Z.P., Chen, D.Q., Liang, S.Y., Ma, H.: Maneuvering target tracking of UAV based on MN-DDPG and transfer learning. Proc. Defence Technol. 17(2), 457–466 (2021). https://doi.org/10.1016/j.dt.2020.11.014
https://doi.org/10.1016/j.dt.2020.11.014
Wan, K., Gao, X., Hu, Z., Wu, G.: Robust motion control for UAV in dynamic uncertain environments using deep reinforcement learning. Proc. Remote Sens. 12(4), 640 (2020)
https://doi.org/10.3390/rs12040640
Sampedro, C., Rodriguez-Ramos, A., Bavle, H., Carrio, A., de la Puente, P., Campoy, P.: A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques. Proc. J. Intel. Robot. Syst. 95(2), 601–627 (2019). https://doi.org/10.1007/s10846-018-0898-1
https://doi.org/10.1007/s10846-018-0898-1
Wang, C., Wang, J., Shen, Y., Zhang, X.: Autonomous navigation of UAVs in large-scale complex environments: a deep reinforcement learning approach. Proc. IEEE Trans. Veh. Technol. 68(3), 2124–2136 (2019)
https://doi.org/10.1109/TVT.2018.2890773
Song, D.R., Yang, C., McGreavy, C., Li, Z.: Recurrent deterministic policy gradient method for bipedal locomotion on rough terrain challenge. In: Proceeding of 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 311–318. IEEE (2018)
https://doi.org/10.1109/ICARCV.2018.8581309
Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: Proceeding of International Conference on Machine Learning, pp. 1587–1596. PMLR (2018)
Li, B., Gan, Z., Chen, D., Sergey Aleksandrovich, D.: UAV maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning. Proc. Remote Sens. 12(22), 3789 (2020). https://doi.org/10.3390/rs12223789
https://doi.org/10.3390/rs12223789
Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., Wu, Y.: The surprising effectiveness of MAPPO in cooperative multi-agent games. arXiv preprint arXiv:2103.01955 (2021)