research-article

Deep Reinforcement Learning Approach for V2X Managed Intersections of Connected Vehicles

Authors:

Alexandre Lombard,

Abdeljalil Abbas-Turki,

Stéphane GallandAuthors Info & Claims

IEEE Transactions on Intelligent Transportation Systems, Volume 24, Issue 7

Pages 7178 - 7189

https://doi.org/10.1109/TITS.2023.3253867

Published: 01 July 2023 Publication History

Abstract

Intersections are major bottlenecks for road traffic, as well as the origin of many accidents. Efficient management of traffic at intersections is required to ensure both safety and efficiency. Yet, the traditional solutions (static signs, traffic lights) are limited in their efficiency as they consider the flow of vehicles and not the vehicles at the microscopic level. By using inter-vehicular communication of connected vehicles, recent works have shown the possibility to have a great increase in the number of evacuated vehicles thanks to the possibility to give an individual right-of-way directly to each vehicle. In this context of intersections of cooperative vehicles, the scheduling of this right-of-way in order to maximize the throughput of the intersection is still a challenging task, with regard to the hybrid and dynamic aspects of the problem. In this paper, we propose an approach based on Deep Reinforcement Learning (DRL) to efficiently distribute the right-of-way to each vehicle. A Markov Decision Process model of intersections of cooperative vehicles, enabling the application of DRL, is proposed. The performance of the DRL-based scheduling is then compared with classic traffic lights, and with two state-of-the-art cooperative scheduling policies, showing the benefits of the approach (increase of the flow, reduction of CO2 emissions).

References

[1]

SAE International, “Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles,” Soc. Automot. Eng., Warrendale, PA, USA, Tech. Rep. J3016_202104, 2021.

[2]

Federal Highway Administration. (2019). Intersection Safety. [Online]. Available: https://cms7.fhwa.dot.gov/research/research-programs/safety/intersection-safety

[3]

J. Wu, A. Abbas-Turki, A. Correia, and A. El Moudni, “Discrete intersection signal control,” in Proc. IEEE Int. Conf. Service Oper. Logistics, Informat., Aug. 2007, pp. 1–6.

[4]

J. Wu, F. Perronnet, and A. Abbas-Turki, “Cooperative vehicle-actuator system: A sequence-based framework of cooperative intersections management,” IET Intell. Transp. Syst., vol. 8, no. 4, pp. 352–360, Jun. 2014.

[5]

B. Chachuat, “Mixed-integer linear programming (MILP): Model formulation,” McMaster Univ. Dept. Chem. Eng. Tech. Rep., Jul. 2019, vol. 17, pp. 1–26.

[6]

S. Soleimaniamiri and X. Li, “Scheduling of heterogeneous connected automated vehicles at a general conflict area,” in Proc. Transp. Res. Board 98th Annu. Meeting Transp. Res. Board, 2019.

[7]

Z. Yao, H. Jiang, Y. Cheng, Y. Jiang, and B. Ran, “Integrated schedule and trajectory optimization for connected automated vehicles in a conflict zone,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 3, pp. 1841–1851, Mar. 2020.

[8]

J. Wu, A. Abbas-Turki, and A. El Moudni, “Cooperative driving: An ant colony system for autonomous intersection management,” Appl. Intell., vol. 37, no. 2, pp. 207–222, 2012.

Digital Library

[9]

F. Yan, M. Dridi, and A. E. Moudni, “Autonomous vehicle sequencing problem for a multi-intersection network: A genetic algorithm approach,” in Proc. Int. Conf. Adv. Logistics Transp., May 2013, pp. 215–220.

[10]

T.-H. Nguyen and J. J. Jung, “Ant colony optimization-based traffic routing with intersection negotiation for connected vehicles,” Appl. Soft Comput., vol. 112, Nov. 2021, Art. no.

[11]

L. Cruz-Piris, M. A. Lopez-Carmona, and I. Marsa-Maestre, “Automated optimization of intersections using a genetic algorithm,” IEEE Access, vol. 7, pp. 15452–15468, 2019.

[12]

J. Chodur and K. Ostrowski, “Assessment of traffic conditions at signalized intersections,” Arch. Transp., vol. 18, no. 2, pp. 5–24, 2006.

[13]

F. Perronnet, A. Abbas-Turki, and A. El Moudni, “A sequenced-based protocol to manage autonomous vehicles at isolated intersections,” in Proc. 16th Int. IEEE Conf. Intell. Transp. Syst. (ITSC), Oct. 2013, pp. 1811–1816.

[14]

V. Mnih et al., “Playing Atari with deep reinforcement learning,” in Proc. Conf. Neural Inf. Process. Syst., 2013, pp. 1–9.

[15]

S. S. Mousavi, M. Schukat, and E. Howley, “Deep reinforcement learning: An overview,” in Proc. SAI Intell. Syst. Conf. Cham, Switzerland: Springer, 2016, pp. 426–440.

[16]

Z. Ding and H. Dong, “Challenges of reinforcement learning,” in Deep Reinforcement Learning. Singapore: Springer, 2020, pp. 249–272.

[17]

R. Naumann, R. Rasche, and J. Tacken, “Managing autonomous vehicles at intersections,” IEEE Intell. Syst. Appl., vol. 13, no. 3, pp. 82–86, May 1998.

Digital Library

[18]

K. Dresner and P. Stone, “Multiagent traffic management: An improved intersection control mechanism,” in Proc. 4th Int. Joint Conf. Auto. Agents Multiagent Syst., Jul. 2005, pp. 530–537.

[19]

J. Gregoire, S. Bonnabel, and A. De La Fortelle, “Optimal cooperative motion planning for vehicles at intersections,” in Proc. IEEE 4th Workshop Navigat., Accurate Positioning Mapping Intell. Vehicles, 2013, pp. 1–6.

[20]

A. Lombard, F. Perronnet, A. Abbas-Turki, A. El Moudni, and R. Bouyekhf, “V2X for vehicle speed synchronization at intersections,” in Proc. 22nd Intell. Transp. Syst. World Congr., 2015, pp. 255–262.

[21]

E. Namazi, J. Li, and C. Lu, “Intelligent intersection management systems considering autonomous vehicles: A systematic literature review,” IEEE Access, vol. 7, pp. 91946–91965, 2019.

[22]

Y. Li and Q. Liu, “Intersection management for autonomous vehicles with vehicle-to-infrastructure communication,” PLoS ONE, vol. 15, no. 7, Jul. 2020, Art. no.

[23]

M. Zhang, A. Abbas-Turki, A. Lombard, A. Koukam, and K.-H. Jo, “Autonomous vehicle with communicative driving for pedestrian crossing: Trajectory optimization,” in Proc. IEEE 23rd Int. Conf. Intell. Transp. Syst. (ITSC), Sep. 2020, pp. 1–6.

[24]

R. Chen, J. Hu, M. W. Levin, and D. Rey, “Stability-based analysis of autonomous intersection management with pedestrians,” Transp. Res. C, Emerg. Technol., vol. 114, pp. 463–483, May 2020.

[25]

M. I.-C. Wang, C. H.-P. Wen, and H. J. Chao, “Roadrunner+: An autonomous intersection management cooperating with connected autonomous vehicles and pedestrians with spillback considered,” ACM Trans. Cyber-Phys. Syst., vol. 6, no. 1, pp. 1–29, Jan. 2022.

Digital Library

[26]

M. Ahmane et al., “Modeling and controlling an isolated urban intersection based on cooperative vehicles,” Transp. Res. C, Emerg. Technol., vol. 28, pp. 44–62, Mar. 2013.

[27]

J. Wu, F. Yan, and J. Liu, “Effectiveness proving and control of platoon-based vehicular cyber-physical systems,” IEEE Access, vol. 6, pp. 21140–21151, 2018.

[28]

G. F. Newell, “Properties of vehicle-actuated signals: I. One-way streets,” Transp. Sci., vol. 3, no. 1, pp. 30–52, Feb. 1969.

[29]

X. B. Wang, K. Yin, and H. Liu, “Vehicle actuated signal performance under general traffic at an isolated intersection,” Transp. Res. C, Emerg. Technol., vol. 95, pp. 582–598, Oct. 2018.

[30]

E. Van der Pol and F. A. Oliehoek, “Coordinated deep reinforcement learners for traffic light control,” in Proc. Learn., Inference Control Multi-Agent Syst. (NIPS), 2016, pp. 21–38.

[31]

J. Zeng, J. Hu, and Y. Zhang, “Adaptive traffic signal control with deep recurrent Q-learning,” in Proc. IEEE Intell. Vehicles Symp. (IV), Jun. 2018, pp. 1215–1220.

[32]

H. Wei, G. Zheng, H. Yao, and Z. Li, “IntelliLight: A reinforcement learning approach for intelligent traffic light control,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 2496–2505.

[33]

H. Wei et al., “PressLight: Learning max pressure control to coordinate traffic signals in arterial network,” in Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2019, pp. 1290–1298.

[34]

D. Isele, R. Rahimi, A. Cosgun, K. Subramanian, and K. Fujimura, “Navigating occluded intersections with autonomous vehicles using deep reinforcement learning,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2018, pp. 2034–2039.

[35]

Y. Wu, H. Chen, and F. Zhu, “DCL-AIM: Decentralized coordination learning of autonomous intersection management for connected and automated vehicles,” Transp. Res. C, Emerg. Technol., vol. 103, pp. 246–260, Jun. 2019.

[36]

M. W. Levin, H. Fritz, and S. D. Boyles, “On optimizing reservation-based intersection controls,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 3, pp. 505–515, Mar. 2017.

Digital Library

[37]

W. Anwar, N. Franchi, and G. Fettweis, “Physical layer evaluation of V2X communications technologies: 5G NR-V2X, LTE-V2X, IEEE 802.11bd, and IEEE 802.11p,” in Proc. IEEE 90th Veh. Technol. Conf. (VTC-Fall), Sep. 2019, pp. 1–7.

[38]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 2018.

Digital Library

[39]

V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, Feb. 2015.

[40]

C. J. C. H. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8, nos. 3–4, pp. 279–292, 1992.

Digital Library

[41]

H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proc. AAAI Conf. Artif. Intell., vol. 30, no. 1, pp. 1–7, 2016.

[42]

T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in Proc. 4th Int. Conf. Learn. Represent. (ICLR), San Juan, Puerto Rico, 2016, pp. 1–21.

[43]

X. Hao, A. Abbas-Turki, F. Perronnet, and R. Bouyekhf, “V2I-based velocity synchronization at intersection,” Math. Methods Comput. Techn. Sci. Eng., vol. 37, no. 1, pp. 67–72, Nov. 2014.

[44]

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. 3rd Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, 2015.

[45]

P. A. Lopez et al., “Microscopic traffic simulation using SUMO,” in Proc. 21st Int. Conf. Intell. Transp. Syst. (ITSC), Nov. 2018, pp. 2575–2582. [Online]. Available: https://elib.dlr.de/124092/

[46]

M. Khayatian et al., “A survey on intersection management of connected autonomous vehicles,” ACM Trans. Cyber-Phys. Syst., vol. 4, no. 4, pp. 1–27, Oct. 2020.

Digital Library

[47]

F. Perronnet, A. Abbas-Turki, J. Buisson, A. El Moudni, R. Zeo, and M. Ahmane, “Cooperative intersection management: Real implementation and feasibility study of a sequence based protocol for urban applications,” in Proc. 15th Int. IEEE Conf. Intell. Transp. Syst., Sep. 2012, pp. 42–47.

[48]

M. Keller et al., “Handbook emission factors for road transport 3.1 (HBEFA),” in INFRAS. Bern, Switzerland: INFRAS, 2010.

[49]

Z. Zhao, Y. Liang, and X. Jin, “Handling large-scale action space in deep Q network,” in Proc. Int. Conf. Artif. Intell. Big Data (ICAIBD), May 2018, pp. 93–96.

[50]

H. Gao, Y. Qin, C. Hu, Y. Liu, and K. Li, “An interacting multiple model for trajectory prediction of intelligent vehicles in typical road traffic scenario,” IEEE Trans. Neural Netw. Learn. Syst., early access, Dec. 31, 2021. 10.1109/TNNLS.2021.3136866.

Cited By

El-Qoraychy FDridi MCreput J(2024)Deep Reinforcement Learning for Vehicle Intersection Management in High-Dimensional Action SpacesProceedings of the 2024 7th International Conference on Machine Learning and Machine Intelligence (MLMI)10.1145/3696271.3696278(39-45)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3696271.3696278

Recommendations

Dynamic intersections and self-driving vehicles
ICCPS '18: Proceedings of the 9th ACM/IEEE International Conference on Cyber-Physical Systems

Connected and automated vehicles are expected to be at the core of future intelligent transportation systems. One of the main practical challenges for self-driving vehicles on public roads is safe cooperation and collaboration among multiple vehicles ...
Centralized Cooperation for Connected Autonomous Vehicles at Intersections by Safe Deep Reinforcement Learning
Connected and automated vehicles (CAVs) have the potential to transform traffic management, especially at intersections. Traditional traffic signals might become obsolete with the implementation of autonomous intersection management (AIM) systems, which ...
Hybrid Reinforcement Learning-Based Eco-Driving Strategy for Connected and Automated Vehicles at Signalized Intersections
Taking advantage of both vehicle-to-everything (V2X) communication and automated driving technology, connected and automated vehicles are quickly becoming one of the transformative solutions to many transportation problems. However, in a mixed traffic ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Intelligent Transportation Systems

IEEE Transactions on Intelligent Transportation Systems Volume 24, Issue 7

July 2023

1120 pages

ISSN:1524-9050

Issue’s Table of Contents

1558-0016 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 July 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

El-Qoraychy FDridi MCreput J(2024)Deep Reinforcement Learning for Vehicle Intersection Management in High-Dimensional Action SpacesProceedings of the 2024 7th International Conference on Machine Learning and Machine Intelligence (MLMI)10.1145/3696271.3696278(39-45)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3696271.3696278

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents