Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

MicroRacer: A Didactic Environment for Deep Reinforcement Learning

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13810))

Abstract

MicroRacer is a simple, open source environment inspired by car racing especially meant for the didactics of Deep Reinforcement Learning. The complexity of the environment has been explicitly calibrated to allow users to experiment with many different methods, networks and hyperparameters settings without requiring sophisticated software or exceedingly long training times. Baseline agents for major learning algorithms such as DDPG, PPO, SAC, TD3 and DSAC are provided too, along with a preliminary comparison in terms of training time and performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://aws.amazon.com/it/deepracer/.

  2. 2.

    https://sourceforge.net/projects/torcs/.

  3. 3.

    https://learn-to-race.org/.

  4. 4.

    Our actors exploit a simplified observation of the state discussed in Sect. 5.

References

  1. Asperti, A., Cortesi, D., De Pieri, C., Pedrini, G., Sovrano, F.: Crawling in rogue’s dungeons with deep reinforcement techniques. IEEE Trans. Games 12(2), 177–186 (2020)

    Article  Google Scholar 

  2. Asperti, A., Cortesi, D., Sovrano, F.: Crawling in rogue’s dungeons with (partitioned) A3C. In: Machine Learning, Optimization, and Data Science - 4th International Conference, LOD 2018, Volterra, Italy, 13–16 Sep 2018, Revised Selected Papers, vol. 11331 of Lecture Notes in Computer Science, pp. 264–275. Springer (2018). https://doi.org/10.1007/978-3-030-13709-0_22

  3. Balaji, B., et al. DeepRacer: educational autonomous racing platform for experimentation with sim2real reinforcement learning. arXiv preprint arXiv:abs/1911.01562 (2019)

  4. Behnel, S., Bradshaw, R., Citro, C., Dalcin, L., Seljebotn, D.S., Smith, K.: Cython: the best of both worlds. Comput. Sci. Eng. 13(2), 31–39 (2011)

    Google Scholar 

  5. Bellemare, M.G., Dabney, W., Rowland, M.: Distributional Reinforcement Learning. MIT Press, Cambridge (2022). https://www.distributional-rl.org

  6. Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. (JAIR) 47, 253–279 (2013)

    Article  Google Scholar 

  7. Bisong, E.: Google Colaboratory, pp. 59–64. Apress, Berkeley (2019)

    Google Scholar 

  8. Brockman, G., et al.: Openai gym. arXiv preprint arXiv:abs/1606.01540 (2016)

  9. Cardamone, L., Loiacono, D., Lanzi, P.L., Bardelli, A.P.: Searching for the optimal racing line using genetic algorithms. In: Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games, pp. 388–394 (2010)

    Google Scholar 

  10. Chen, B., Francis, J., Oh, J., Nyberg, E., Herbert, S.L.: Safe autonomous racing via approximate reachability on ego-vision (2021)

    Google Scholar 

  11. Brutto, M.D.: MicroRacer: development of a didactic environment for deep reinforcement learning. Master’s thesis, University of Bologna, School of Science, Session III 2021–22

    Google Scholar 

  12. Dosovitskiy, A., Ros, G., Codevilla, F., López, A.M., Koltun, V.: CARLA: an open urban driving simulator. In: 1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, 13–15 Nov 2017, Proceedings, pp. 1–16. PMLR (2017)

    Google Scholar 

  13. Duan, J., Guan, Y., Li, S.E., Ren, Y., Sun, Q., Cheng, B.: Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15 (2021)

    Google Scholar 

  14. Evans, B., Engelbrecht, H.A., Jordaan, H.W.: Learning the subsystem of local planning for autonomous racing. In: 20th International Conference on Advanced Robotics, ICAR 2021, Ljubljana, Slovenia, Dec 6–10 2021, pp. 601–606. IEEE (2021)

    Google Scholar 

  15. Evans, B., Engelbrecht, H.A., Jordaan, H.W.: Reward signal design for autonomous racing. In: 20th International Conference on Advanced Robotics, ICAR 2021, Ljubljana, Slovenia, 6–10 Dec 2021, pp. 455–460. IEEE (2021)

    Google Scholar 

  16. Fujimoto, S., Hoof, H.V., Meger, D.: Addressing function approximation error in actor-critic methods. In: Jennifer, G., Dy., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 Jul 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 1582–1591. PMLR (2018)

    Google Scholar 

  17. Galletti, G.: Deep reinforcement learning nell’ambiente pytorcs. Master’s thesis, University of Bologna, school of Science, Session III 2021

    Google Scholar 

  18. Goldfain, B., et al.: AutoRally: an open platform for aggressive autonomous driving. IEEE Control Syst. Mag. 39(1), 26–55 (2019)

    Article  MathSciNet  Google Scholar 

  19. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 Jul 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 1856–1865. PMLR (2018)

    Google Scholar 

  20. Herman, J., et al.: Learn-to-Race: a multimodal control environment for autonomous racing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9793–9802 (2021)

    Google Scholar 

  21. Li, C.: Challenging on car racing problem from openai gym. arXiv preprint arXiv:abs/1911.04868 (2019)

  22. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016)

    Google Scholar 

  23. Loiacono, D., et al.: The 2009 simulated car racing championship. IEEE Trans. Comput. Intell. AI Games 2(2), 131–147 (2010)

    Google Scholar 

  24. Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:abs/1312.5602 (2013)

  25. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  26. Paull, L., et al.: Duckietown: an open, inexpensive and flexible platform for autonomy education and research. In: 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 - June 3, 2017, pp. 1497–1504 (2017)

    Google Scholar 

  27. Plappert, M., et al.: Parameter space noise for exploration. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net (2018)

    Google Scholar 

  28. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:abs/1707.06347 (2017)

  29. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.A.: Deterministic policy gradient algorithms. In: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21–26 June 2014, vol. 32 of JMLR Workshop and Conference Proceedings, pp. 387–395. JMLR.org (2014)

    Google Scholar 

  30. Singh, S.P., Barto, A.G., Chentanez, N.: Intrinsically motivated reinforcement learning. In: Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13–18, 2004, Vancouver, British Columbia, Canada], pp. 1281–1288 (2004)

    Google Scholar 

  31. Richard, S.: Sutton and Andrew G, 1st edn. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA (1998)

    Google Scholar 

  32. Vamvoudakis, K.G., Wan, Y., Lewis, F.L., Cansever, D. (eds.): Handbook of Reinforcement Learning and Control. SSDC, vol. 325. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-60990-0

    Book  MATH  Google Scholar 

  33. Vorabbi, S.: Analisi dell’ambiente aws deepracer per la sperimentazione di tecniche di reinforcement learning. Master’s thesis, University of Bologna, school of Science, Session II 2021

    Google Scholar 

  34. Wang, H., et al.: Deep reinforcement learning: a survey. Frontiers Inf. Technol. Electron. Eng. 21(12), 1726–1744 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics: Science and Engineering (DISI), University of Bologna, Bologna, Italy

    Andrea Asperti & Marco Del Brutto

Authors
  1. Andrea Asperti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Del Brutto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrea Asperti .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. University of Reading, Reading, UK

    Varun Ojha

  3. University of Oxford, Oxford, UK

    Emanuele La Malfa

  4. University of Cambridge, Cambridge, UK

    Gabriele La Malfa

  5. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  6. Free University of Bozen-Bolzano, Bolzano, Italy

    Giuseppe Di Fatta

  7. University of Catania, Catania, Italy

    Giovanni Giuffrida

  8. Dana-Farber Cancer Institute, Boston, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Asperti, A., Del Brutto, M. (2023). MicroRacer: A Didactic Environment for Deep Reinforcement Learning. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2022. Lecture Notes in Computer Science, vol 13810. Springer, Cham. https://doi.org/10.1007/978-3-031-25599-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25599-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25598-4

  • Online ISBN: 978-3-031-25599-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics