Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

RELight: a random ensemble reinforcement learning based method for traffic light control

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Traffic lights are crucial for urban traffic management, as they significantly impact congestion reduction and travel safety. Traditional methods relying on hand-crafted rules and operator experience are limited in their ability to adapt to changing traffic environments. To address this challenge, we have been exploring intelligent traffic light control using deep reinforcement learning techniques. However, current approaches often suffer from inadequate training data and unstable training processes, leading to suboptimal performance and real-world consequences. In this study, we propose RELight, a novel random ensemble reinforcement learning-based traffic light control framework. RELight effectively utilizes collected empirical data, ensuring a stable and efficient training process. To evaluate the performance of our proposed framework, we conducted a comprehensive set of experiments on a variety of datasets, including four synthetic datasets and a real traffic dataset collected from surveillance cameras at an intersection in Hangzhou, China. The results show that RELight outperforms existing approaches, demonstrating its superiority and potential for practical traffic light control applications.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Find the latest articles, discoveries, and news in related topics.

Data Availability

The data that support the findings of this study are openly available in (https://traffic-signal-control.github.io/).

Notes

  1. https://traffic-signal-control.github.io/

References

  1. Abdoos M, Mozayani N, Bazzan AL (2014) Hierarchical control of traffic signals using q-learning with tile coding. Appl Intell 40:201–213

    Article  Google Scholar 

  2. Chacha Chen HW, Xu N, Zheng G et al (2020) Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control. In: Proceedings of the thirty-fourth AAAI conference on artificial intelligence (AAAI’20), New York, NY, USA, pp 7–12

  3. Chen C, Wei H, Xu N et al (2020) Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control. In: Proceedings of the AAAI conference on artificial intelligence, pp 3414–3421

  4. Chen X, Wang C, Zhou Z et al (2021) Randomized ensembled double q-learning: learning fast without a model. In: 9th International conference on learning representations, ICLR 2021, Virtual event, Austria. OpenReview.net, https://openreview.net/forum?id=AY8zfZm0tDd. Accessed 3-7 May 2021

  5. Du W, Ye J, Gu J et al (2023) Safelight: a reinforcement learning method toward collision-free traffic signal control. In: Proceedings of the AAAI conference on artificial intelligence, pp 14,801–14,810

  6. El-Tantawy S, Abdulhai B (2010) An agent-based learning towards decentralized and coordinated traffic signal control. In: 13th International IEEE conference on intelligent transportation systems. IEEE, pp 665–670. https://doi.org/10.1109/ITSC.2010.5625066

  7. El-Tantawy S, Abdulhai B, Abdelgawad H (2013) Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto. IEEE Trans Intell Transp Syst 14(3):1140–1150

    Article  Google Scholar 

  8. Fujimoto S, Hoof H, Meger D (2018) Addressing function approximation error in actor-critic methods. In: International conference on machine learning. PMLR, pp 1587–1596

  9. Gershenson C (2005) Self-organizing traffic lights. Complex Syst 16(1). http://www.complex-systems.com/abstracts/v16_i01_a02.html

  10. Haddad J, De Schutter B, Mahalel D et al (2010) Optimal steady-state control for isolated traffic intersections. IEEE Trans Autom Control 55(11):2612–2617. https://doi.org/10.1109/TAC.2010.2060245

    Article  MathSciNet  Google Scholar 

  11. van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, Phoenix, Arizona, USA, vol 30. AAAI Press, pp 2094–2100. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12389. Accessed 12-17 Feb 2016

  12. Huang J, Tan Q, Li H et al (2022) Monte carlo tree search for dynamic bike repositioning in bike-sharing systems. Appl Intell 52(4):4610–4625. https://doi.org/10.1007/s10489-021-02586-x

    Article  Google Scholar 

  13. Hunt P, Robertson D, Bretherton R et al (1981) Scoot-a traffic responsive method of coordinating signals. Tech rep

  14. Ji S, Zheng Y, Wang Z et al (2019) A deep reinforcement learning-enabled dynamic redeployment system for mobile ambulances. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3(1):1–20. https://doi.org/10.1145/3314402

    Article  Google Scholar 

  15. Jiang Q, Qin M, Shi S et al (2022) Multi-agent reinforcement learning for traffic signal control through universal communication method. arXiv preprint arXiv:2204.12190

  16. Koonce P, Rodegerdts L (2008) Traffic signal timing manual. Tech rep, United States. Federal Highway Administration

  17. Li H, Huang J, Yuan H et al (2021) A two-phase method to balance the result of distributed graph repartitioning. IEEE Transactions on Big Data 8(6):1580–1591

    Google Scholar 

  18. Li H, Li X, Su L et al (2022) Deep spatio-temporal adaptive 3d convolutional neural networks for traffic flow prediction. ACM Transactions on Intelligent Systems and Technology (TIST) 13(2):1–21

    Google Scholar 

  19. Li H, Jin D, Li X et al (2023) Dmgf-net: an efficient dynamic multi-graph fusion network for traffic prediction. ACM Transactions on Knowledge Discovery from Data

  20. Li L, Lv Y, Wang FY (2016) Traffic signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica 3(3):247–254

    Article  MathSciNet  Google Scholar 

  21. Liang X, Du X, Wang G et al (2019) A deep reinforcement learning network for traffic light cycle control. IEEE Trans Veh Technol 68(2):1243–1253

    Article  Google Scholar 

  22. Lillicrap TP, Hunt JJ, Pritzel A et al (2016) Continuous control with deep reinforcement learning. In: 4th International conference on learning representations, ICLR 2016, San Juan, Puerto Rico, Conference track proceedings. arXiv:1509.02971. Accessed 2-4 May 2016

  23. Lowrie P (1990) Scats, sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic

  24. Mao F, Li Z, Li L (2022) A comparison of deep reinforcement learning models for isolated traffic signal control. IEEE Intell Transp Syst Mag 15(1):160–180

    Article  Google Scholar 

  25. Miller AJ (1963) Settings for fixed-cycle traffic signals. Journal of the Operational Research Society 14(4):373–386. https://doi.org/10.1057/jors.1963.61

    Article  Google Scholar 

  26. Mirchandani P, Head L (2001) A real-time traffic signal control system: architecture, algorithms, and analysis. Transportation Research Part C: Emerging Technologies 9(6):415–432. https://doi.org/10.1016/S0968-090X(00)00047-4, https://www.sciencedirect.com/science/article/pii/S0968090X00000474

  27. Mousavi SS, Schukat M, Howley E (2017) Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intel Transport Syst 11(7):417–423

    Article  Google Scholar 

  28. Nikishin E, Schwarzer M, D’Oro P et al (2022) The primacy bias in deep reinforcement learning. In: International conference on machine learning. PMLR, pp 16,828–16,847

  29. Nishi T, Otaki K, Hayakawa K et al (2018) Traffic signal control based on reinforcement learning with graph convolutional neural nets. In: 2018 21st International conference on intelligent transportation systems (ITSC). IEEE, pp 877–883

  30. Noaeen M, Naik A, Goodman L et al (2022) Reinforcement learning in urban network traffic signal control: a systematic literature review. Expert Syst Appl 199:116,830

    Article  Google Scholar 

  31. Van der Pol E, Oliehoek FA (2016) Coordinated deep reinforcement learners for traffic light control. Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016) 8:21–38

    Google Scholar 

  32. Roess RP, Prassas ES, McShane WR (2004) Traffic engineering. Pearson/Prentice Hall

  33. Silver D, Huang A, Maddison CJ et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  34. Silver D, Schrittwieser J, Simonyan K et al (2017) Mastering the game of go without human knowledge. Nature 550(7676):354–359

    Article  Google Scholar 

  35. Urbanik T, Tanaka A, Lozner B et al (2015) Signal timing manual, vol 1. Transportation Research Board Washington, DC

    Book  Google Scholar 

  36. Varaiya P (2013) The max-pressure controller for arbitrary networks of signalized intersections. Springer, New York, NY, pp 27–66. https://doi.org/10.1007/978-1-4614-6243-9_2

  37. Wang M, Wu L, Li J et al (2022) Urban traffic signal control with reinforcement learning from demonstration data. In: 2022 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  38. Wang T, Cao J, Hussain A (2021) Adaptive traffic signal control for large-scale scenario with cooperative group-based multi-agent reinforcement learning. Transportation research part C: emerging technologies 125:103,046

    Article  Google Scholar 

  39. Wei H, Zheng G, Yao H et al (2018) Intellilight: a reinforcement learning approach for intelligent traffic light control. In: Proceedings of the 24th ACM SIGKDD International conference on knowledge discovery & data mining, KDD 2018, London, UK. ACM, pp 2496–2505. https://doi.org/10.1145/3219819.3220096. Accessed 19-23 Aug 2018

  40. Wei H, Chen C, Zheng G et al (2019a) Presslight: learning max pressure control to coordinate traffic signals in arterial network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 1290–1298

  41. Wei H, Xu N, Zhang H et al (2019b) Colight: learning network-level cooperation for traffic signal control. In: Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019, Beijing, China. ACM, pp 1913–1922. https://doi.org/10.1145/3357384.3357902. Accessed 3-7 Nov 2019

  42. Wei H, Zheng G, Gayah V et al (2021) Recent advances in reinforcement learning for traffic signal control: a survey of models and evaluation. ACM SIGKDD Explorations Newsl 22(2):12–18

    Article  Google Scholar 

  43. Wei Y, Mao M, Zhao X et al (2020) City metro network expansion with reinforcement learning. In: KDD ’20: The 26th ACM SIGKDD conference on knowledge discovery and data mining, Virtual event, CA, USA. ACM, pp 2646–2656. https://doi.org/10.1145/3394486.3403315. Accessed 23-27 Aug 2020

  44. Wiering MA et al (2000) Multi-agent reinforcement learning for traffic light control. In: Machine learning: proceedings of the seventeenth international conference (ICML’2000), pp 1151–1158

  45. Wong C, Wong S (2003) Lane-based optimization of signal timings for isolated junctions. Transportation Research Part B: Methodological 37(1):63–84

    Article  Google Scholar 

  46. Wu Q, Wu J, Shen J et al (2022) Distributed agent-based deep reinforcement learning for large scale traffic signal control. Knowl-Based Syst 241:108,304

    Article  Google Scholar 

  47. Xiong Y, Zheng G, Xu K et al (2019) Learning traffic signal control from demonstrations. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 2289–2292

  48. Xu M, Wu J, Huang L et al (2020) Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning. Journal of Intelligent Transportation Systems 24(1):1–10

    Article  Google Scholar 

  49. Ying Z, Cao S, Liu X et al (2022) Privacysignal: privacy-preserving traffic signal control for intelligent transportation system. IEEE Trans Intell Transp Syst 23(9):16,290-16,303

    Article  Google Scholar 

  50. Zang X, Yao H, Zheng G et al (2020) Metalight: value-based meta-reinforcement learning for traffic signal control. In: Proceedings of the AAAI conference on artificial intelligence, pp 1153–1160. https://aaai.org/ojs/index.php/AAAI/article/view/5467

  51. Zhang H, Liu C, Zhang W et al (2020) Generalight: improving environment generalization of traffic signal control via meta reinforcement learning. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp 1783–1792

  52. Zheng G, Xiong Y, Zang X et al (2019) Learning phase competition for traffic signal control. In: Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019, Beijing, China. ACM, pp 1963–1972. https://doi.org/10.1145/3357384.3357900. Accessed 3-7 Nov 2019

Download references

Acknowledgements

This research was supported in part by grant 61876138 from the National Science Foundation of China. Any opinions, findings, and conclusions expressed herein are those of the authors and do not necessarily reflect the views of the funding agencies.

Author information

Author notes

  1. Ruijie Qi and He Li contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Technology, Xidian University, Xi’an, 710071, Shaanxi Province, China

    Jianbin Huang, Qinglin Tan, Ruijie Qi & He Li

Authors
  1. Jianbin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qinglin Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruijie Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. He Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jianbin Huang provided the research ideas and was responsible for the formulation and implementation of the research plan; Qinglin Tan proposed the method framework and conducted experiments; Ruijie Qi collated the data and visualized the experimental results; He Li prepared and revised the initial draft; Qinglin Tan, Ruijie Qi and He Li collaboratively analyzed and discussed the experimental results and drew conclusions.

Corresponding author

Correspondence to Qinglin Tan.

Ethics declarations

Ethical approval

Not applicable.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, J., Tan, Q., Qi, R. et al. RELight: a random ensemble reinforcement learning based method for traffic light control. Appl Intell 54, 95–112 (2024). https://doi.org/10.1007/s10489-023-05197-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05197-w

Keywords

Search

Navigation