Abstract
Robotic soccer simulation is a challenging area, where the development of new techniques is paramount to remain competitive. Robotic skill evolution has accelerated with recent developments in deep learning algorithms, leading to improvements in behavior number and complexity. Shooting a ball towards a defined target is one of the most basic yet indispensable skills in soccer. However, fast and accurate kicks pose several challenges. In order to reach that target, the skill is highly dependent on the ability of the agent to self-locate and self-orient, in order to better position itself before the kick. To tackle these issues, a 6D localization technique was devised. To optimize the kick behavior, two scenarios were proposed. In the first, the robot walks to the ball, stops, and then kicks. In the second, it kicks the ball while moving. We used state-of-the-art algorithms — Proximal Policy Optimization and Soft Actor Critic — to solve these complex problems and show their applicability in the context of RoboCup. Obtained results have shown very significant improvements over previously used behaviors by FC Portugal 3D team. The new kick in motion executes 5 times faster than the previous kick, and the new 6D pose estimator has an average error of just 6.3mm, a reduction of more than 97%.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/(2015)
Abdolmaleki, A., Simões, D., Lau, N., Reis, L. P., Neumann, G., Sarıel, S., Lee, D.D.: Learning a Humanoid Kick with Controlled Distance. In: Behnke, S., Sheh, R. (eds.) Robocup 2016: Robot world cup XX, 45–57. Springer International Publishing, Cham (2017)
Abreu, M., Reis, L.P., Lau, N.: Learning to Run Faster in a Humanoid Robot Soccer Environment through Reinforcement Learning. In: Chalup, S., Niemueller, T., Suthakorn, J., Williams, M.A. (eds.) Robocup 2019: Robot World Cup XXIII, 3-15. Springer International Publishing, Cham (2019)
Boedecker, J., Asada, M.: Simspark–concepts and application in the robocup 3d soccer simulation league SIMPAR-2008 Workshop on The Universe of RoboCup Simulators (2008)
Brafman, R. I., Tennenholtz, M.: R-MAX – a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. 3(Oct), 213–231 (2002)
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym (2016)
Bustamante, C.: Probabilistic agent localization and Fuzzy-Bayesian pass evaluation for the RoboCup simulation 3D league. Master’s Thesis, Tecnológico De Monterrey, Monterrey, Mexico (2007)
Coulom, R.: Clop: Confident Local Optimization for Noisy Black-Box Parameter Tuning. In: Advances in Computer Games, pp. 146–157. Springer (2011)
Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., Zhokhov, P.: Openai baselines https://github.com/openai/baselines (2017)
Documentation, A.: NAO - actuator & sensor list. http://doc.aldebaran.com/2-1/family/nao_dcm/actuator_sensor_names.html
Dorer, K.: Learning to Use Toes in a Humanoid Robot. In: Akiyama, H., Obst, O., Sammut, C., Tonidandel, F. (eds.) Robocup 2017: Robot World Cup XXI, 168–179. Springer International Publishing, Cham (2018)
Ferreira, R., Reis, L.P., Moreira, A.P., Lau, N.: Development of an Omnidirectional Kick for a NAO Humanoid Robot. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds.) Advances in Artificial Intelligence – IBERAMIA 2012, 571–580. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)
Golub, G.H., Van Loan, C.F.: An analysis of the total least squares problem. SIAM J. Num. Anal. 17(6), 883–893 (1980)
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 80, pp. 1861–1870. PMLR, Stockholmsmässan, Stockholm Sweden (2018)
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S.: Soft actor-critic algorithms and applications. arXiv:1812.05905 (2018)
Hao, Y., Liang, Z., Liu, J., Li, J., Zhao, H.: The Framework Design of Humanoid Robots in the RoboCup 3D Soccer Simulation Competition. In: 2013 10Th IEEE International conference on control and automation (ICCA), pp. 1423–1428. IEEE (2013)
Hester, T., Quinlan, M., Stone, P.: Generalized Model Learning for Reinforcement Learning on a Humanoid Robot. In: 2010 IEEE International Conference on Robotics and Automation, pp. 2369–2374. IEEE (2010)
Hester, T., Stone, P.: Negative Information and Line Observations for Monte Carlo Localization. In: 2008 IEEE International Conference on Robotics and Automation, pp. 2764–2769. IEEE (2008)
Hester, T., Stone, P.: Generalized model learning for reinforcement learning in factored domains. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 717–724 (2009)
Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y.: Stable baselines https://github.com/hill-a/stable-baselines (2018)
Hoffman, J., Spranger, M., Gohring, D., Jungel, M.: Making Use of What You Don’t See: Negative Information in Markov Localization. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2947–2952. IEEE (2005)
Jakob, W., Rhinelander, J., Moldovan, D.: pybind11 – seamless operability between c++ 11 and python. https://github.com/pybind/pybind11 (2017)
Jouandeau, N., Hugel, V.: Optimization of Parametrised Kicking Motion for Humanoid Soccer Player. In: 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 241–246. IEEE (2014)
Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Matsubara, H.: RoboCup: A challenge problem for AI. AI Magazine 18(1), 73–85 (1997). https://doi.org/10.1609/aimag.v18i1.1276
Lu, W., Zhang, J., Zhao, X., Wang, J., Dang, J.: Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network. J. Ambient. Intell. Humaniz. Comput. 8(6), 885–893 (2017)
MacAlpine, P., Depinet, M., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup-2014: Robot Soccer World Cup XVIII, Lecture Notes in Artificial Intelligence. Springer (2015)
MacAlpine, P., Stone, P.: UT Austin Villa: RoboCup 2017 3D simulation league competition and technical challenges champions. In: Sammut, C., Obst, O., Tonidandel, F., Akyama, H. (eds.) RoboCup 2017: Robot soccer world cup XXI. Springer (2017)
Müller, J., Laue, T., Röfer, T.: Kicking a ball - modeling complex dynamic motions for humanoid robots. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6556 LNAI, pp 109–120 (2011)
Muzio, A., Aguiar, L., Máximo, M.R., Pinto, S.C.: Monte Carlo Localization with Field Lines Observations for Simulated Humanoid Robotic Soccer. In: 2016 XIII Latin American Robotics Symposium and IV Brazilian Robotics Symposium (LARS/SBR), pp. 334–339. IEEE (2016)
Omidvar, M.N., Li, X.: A Comparative Study of Cma-Es on Large Scale Global Optimisation. In: Australasian Joint Conference on Artificial Intelligence, pp. 303–312. Springer (2010)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: an Imperative Style, High-Performance Deep Learning Library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc (2019)
Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., Dormann, N.: Stable baselines3 https://github.com/DLR-RM/stable-baselines3 (2019)
RoboCup: 3D Soccer Simulation League history. https://ssim.robocup.org/3d-simulation/3d-history/ (2021)
Robotics, S.: NAO the humanoid robot. https://www.softbankrobotics.com/emea/en/nao
Shafii, N., Abdolmaleki, A., Ferreira, R., Lau, N., Reis, L.P.: Omnidirectional Walking and Active Balance for Soccer Humanoid Robot. In: Correia, L., Reis, L.P., Cascalho, J. (eds.) Progress in Artificial Intelligence, pp. 283–294. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)
Shafii, N., Lau, N., Reis, L.P.: Learning to walk fast: Optimized hip height movement for simulated and real humanoid robots. J. Intel. & Robotic Syst. 80(3), 555–571 (2015). https://doi.org/10.1007/s10846-015-0191-5
SimSpark: Agentsyncmode — SimSpark. http://simspark.sourceforge.net/wiki/index.php/AgentSyncMode (2012)
Snafii, N., Abdolmaleki, A., Lau, N., Reis, L.P.: Development of an omnidirectional walk engine for soccer humanoid robots. Inter. J. Advanced Robotic Syst. 12(12), 193 (2015). https://doi.org/10.5772/61314
Söderkvist, I.: Using SVD for some fitting problems. https://ltu.se/cms_fs/1.51590!/svd-fitting.pdf (2009). Luleå University of Technology
Stoecker, J., Visser, U.: Roboviz: Programmable visualization for simulated soccer, P. 282–293. Springer-Verlag, Berlin, Heidelberg (2012)
Sutton, R.S., Barto, A.G.: Reinforcement learning: An Introduction, second edn The MIT Press (2018)
Teixeira, H., Silva, T., Abreu, M., Reis, L.P.: Humanoid robot kick in motion ability for playing robotic soccer. In: 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 34–39. https://doi.org/10.1109/ICARSC49921.2020.9096073 (2020)
Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, Cambridge, Massachusetts (2005)
Visser, U., Burkhard, H.D.: Robocup: 10 Years of Achievements and Future Challenges. AI Mag. 28(2), 115–132 (2007)
Watkins, C.: Learning from delayed rewards. Ph.D. thesis, University of Cambridge (1989)
Xu, Y., Vatankhah, H.: Simspark: an open source robot simulator developed by the Robocup community. In: Behnke, S., Veloso, M., Visser, A., Xiong, R. (eds.) Robocup 2013: Robot World Cup XVII, pp. 632–639. Springer, Berlin, Heidelberg (2014)
Funding
The first author is supported by FCT — Foundation for Science and Technology under grant SFRH/BD/139926/2018. The work was also partially funded by COMPETE 2020 and FCT, under projects UIDB/00027/2020 (LIACC) and UIDB/00127/2020 (IEETA).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by all authors. The first draft of the manuscript was written by Miguel Abreu, Tiago Silva, and Henrique Teixeira. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Miguel Abreu and Tiago Silva contributed equally to this work as first authors.
Rights and permissions
About this article
Cite this article
Abreu, M., Silva, T., Teixeira, H. et al. 6D Localization and Kicking for Humanoid Robotic Soccer. J Intell Robot Syst 102, 30 (2021). https://doi.org/10.1007/s10846-021-01385-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-021-01385-3