article

Neuroevolution of Inverted Pendulum Control: A Comparative Study of Simulation Techniques

Authors:

Christiaan J. Pretorius,

Mathys C. Du Plessis,

John W. GonsalvesAuthors Info & Claims

Journal of Intelligent and Robotic Systems, Volume 86, Issue 3-4

Pages 419 - 445

https://doi.org/10.1007/s10846-017-0465-1

Published: 01 June 2017 Publication History

Abstract

The inverted pendulum control problem is a classical benchmark in control theory. Amongst the approaches to developing control programs for an inverted pendulum, the evolution of Artificial Neural Network (ANN) based controllers has received some attention. The authors have previously shown that Evolutionary Robotics (ER) can successfully be used to evolve inverted pendulum stabilization controllers in simulation and that these controllers can transfer successfully from simulation to real-world robotic hardware. During this process, use was made of robotic simulators constructed from empirically-collected data and based on ANNs. The current work aims to compare this method of simulator construction with the more traditional method of building robotic simulators based on physics equations governing the robotic system under consideration. In order to compare ANN-based and physics-based simulators in the evolution of inverted pendulum controllers, a real-world wheeled inverted pendulum robot was considered. Simulators based on ANNs as well as on a system of ordinary differential equations describing the dynamics of the robot were developed. These two simulation techniques were then compared by using each in the simulation-based evolution of controllers. During the evolution process, the effects of injecting different levels of noise into the simulation was furthermore studied. Encouraging results were obtained, with controllers evolved using ANN-based simulators and realistic levels of noise outperforming those evolved using the physics-based simulators.

References

[1]

Encog Machine Learning Framework -- Heaton Research (Accessed January 2015). http://www.heatonresearch.com/encog

[2]

NXT Acceleration / Tilt Sensor (Accessed July 2016). https://www.hitechnic.com/cgi-bin/commerce.cgi?preadd=action&key=NAC1040

[3]

NXT Gyro Sensor (Accessed July 2016). https://www.hitechnic.com/cgi-bin/commerce.cgi?preadd=action&key=NGY1044

[4]

nxtOSEK/JSP: ANSI C/C++ with OSEK/ μITRON RTOS for Lego MindStorms NXT (Accessed November 2014). http://lejos-osek.sourceforge.net/

[5]

Commons Math: The Apache Commons Mathematics Library (Accessed September 2014). http://commons.apache.org/proper/commons-math/

[6]

Atkeson, C.G., Santamaria, J.C.: A Comparison of Direct and Model-Based Reinforcement Learning. In: International Conference on Robotics and Automation, pp. 3557---3564. IEEE Press (1997)

[7]

Bonarini, A., Caccia, C., Lazaric, A., Restelli, M.: Batch Reinforcement Learning for Controlling a Mobile Wheeled Pendulum Robot. In: Artificial Intelligence in Theory and Practice II, pp. 151---160. Springer (2008)

[8]

Bongard, J., Zykov, V., Lipson, H.: Resilient machines through continuous self-modeling. Science 314, 1118---1121 (2006)

[9]

Burden, R., Faires, J.: Numerical Analysis, 8th Edn. Thomson Brooks/Cole, London (2005)

[10]

Canale, M., Brunet, S.C.: A Lego Mindstorms NXT Experiment for Model Predictive Control Education. In: European Control Conference (ECC) (2013)

[11]

Colton, S.: The balance filter: A simple solution for integrating accelerometer and gyroscope measurements for a balancing platform. White paper Massachusetts Institute of Technology (2007)

[12]

De Nardi, R., Holland, O.E.: Coevolutionary modelling of a miniature rotorcraft. Intelligent Autonomous Systems 10 (2008). IAS-10

[13]

Drumwright, E., Hsu, J., Koenig, N., Shell, D.: Extending Open Dynamics Engine for robotics simulation. Simulation, Modeling, and Programming for Autonomous Robots 6472, 38---50 (2010)

[14]

El-Hawwary, M.I., Elshafei, A.L., Emara, H.M., Fattah, H.A.A.: Adaptive fuzzy control of the inverted pendulum problem. IEEE Trans. Control Syst. Technol. 14(6), 1135---1144 (2006)

[15]

Engelbrecht, A.P.: Computational Intelligence: An Introduction, 2nd edn. Wiley, West Sussex (2007)

[16]

Floreano, D., Mondada, F.: Evolution of homing navigation in a real mobile robot. IEEE Trans. Syst. Man Cybern. 26(3), 396---407 (1996)

Digital Library

[17]

Glette, K., Klaus, G., Zagal, J.C., Torresen, J.: Evolution of Locomotion in a Simulated Quadruped Robot and Transferral to Reality. In: Proceedings of the Seventeenth International Symposium on Artificial Life and Robotics (2012)

[18]

Gomez, F.J., Miikkulainen, R.: Transfer of Neuroevolved Controllers in Unstable Domains. In: Proceedings of the Genetic Evolutionary Computation Conference (GECCO-04) (2004)

[19]

Gürocak, H.: A genetic-algorithm-based method for tuning fuzzy logic controllers. Fuzzy Set. Syst. 108(1), 39---47 (1999)

Digital Library

[20]

Hapke, M., Komosinski, M.: Evolutionary design of interpretable fuzzy controllers. Foundation of Computing and Decision Sciences 33(4), 351 (2008)

[21]

Hartland, C., Bredeche, N.: Evolutionary Robotics, Anticipation and the Reality Gap. In: IEEE International Conference on Robotics and Biomimetics (2006)

[22]

Heidrich-Meisner, V., Igel, C.: Neuroevolution strategies for episodic Reinforcement Learning. J. Algoritm. 64(4), 152---168 (2009)

Digital Library

[23]

Herrera, F., Lozano, M., Verdegay, J.: Tuning fuzzy logic controllers by genetic algorithms. Int. J. Approx. Reason. 12(3), 299---315 (1995)

[24]

Holland, J.H.: Adaptation in Natural and Artificial Systems. MIT Press, Cambridge (1992)

[25]

Hotz, P., Gómez, G.: The Transfer Problem from Simulation to the Real World in Artificial Evolution. In: Workshop and Tutorial Proceedings of the Ninth International Conference on the Simulation and Synthesis of Living Systems (Alife IX) (2004)

[26]

Huang, C.H., Wang, W.J., Chiu, C.H.: Design and implementation of fuzzy control on a two-wheel inverted pendulum. IEEE Trans. Ind. Electron. 58(7), 2988---3001 (2011)

[27]

Igel, C.: Neuroevolution for Reinforcement Learning Using Evolution Strategies. In: Congress on Evolutionary Computation (CEC), vol. 4, pp. 2588---2595 (2003)

[28]

Jakobi, N.: Minimal simulations for evolutionary robotics. Ph.D. thesis University of Sussex (1998)

[29]

Jakobi, N.: Running across the Reality Gap: Octopod Locomotion Evolved in a Minimal Simulation. In: Evolutionary Robotics, pp. 39---58. Springer (1998)

[30]

Jakobi, N., Husbands, P., Harvey, I.: Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics, vol. 929. Springer, Berlin (1995)

[31]

Kawada, K., Yamamoto, T., Mada, Y.: A Design of Evolutionary Recurrent Neural-Net Based Controllers for an Inverted Pendulum. In: Fifth Asian Control Conference (2004)

[32]

Klaus, G., Glette, K., TØrresen, J.: A Comparison of Sampling Strategies for Parameter Estimation of a Robot Simulator. In: Proceedings of the Third International Conference on Simulation, Modeling, and Programming for Autonomous Robots (2012)

[33]

Koos, S., Cully, A., Mouret, J.: Fast damage recovery in robotics with the T-Resilience algorithm. Preprint version (available: arXiv:1302.0386,2013)

[34]

Koos, S., Mouret, J., Doncieux, S.: The Transferability Approach: Crossing the reality gap in Evolutionary Robotics. IEEE Trans. Evol. Comput., 1---25 (2012)

[35]

Li, Z., Xu, C.: Adaptive fuzzy logic control of dynamic balance and motion for wheeled inverted pendulums. Fuzzy Set. Syst. 160(12), 1787---1803 (2009)

Digital Library

[36]

Lipson, H., Bongard, J., Zykov, V., Malone, E.: Evolutionary Robotics for Legged Machines: From Simulation to Physical Reality. In: Proceedings of the 9th International Conference on Intelligent Autonomous Systems (2006)

[37]

Moeckel, R., Perov, Y.N., Nguyen, A.T., Vespignani, M., Bonardi, S., Pouya, S., Sproewitz, A., van den Kieboom, J., Wilhelm, F., Ijspeert, A.J.: Gait Optimization for Roombots Modular Robots - Matching Simulation and Reality. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2013)

[38]

Nakamura, S., Hashimoto, S.: Hybrid Learning Strategy to Solve Pendulum Swing-Up Problem for Real Hardware. In: IEEE International Conference on Robotics and Biomimetics (2007)

[39]

Nelles, O.: Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models. Springer, Berlin (2010)

[40]

Nelson, A.L., Barlow, G.J., Doitsidis, L.: Fitness functions in evolutionary robotics: A survey and analysis. Robot. Auton. Syst. 57(4), 345---370 (2009)

Digital Library

[41]

Nguyen-Tuong, D., Peters, J.: Model learning for robot control: A survey. Cogn. Process. 12(4), 319---340 (2011)

[42]

Oh, S.K., Pedrycz, W., Rho, S.B., Ahn, T.C.: Parameter estimation of fuzzy controller and its application to inverted pendulum. Eng. Appl. Artif. Intell. 17(1), 37---60 (2004)

[43]

Pratihar, D.K.: Evolutionary Robotics - A review. Sadhana 28(6), 999---1009 (2003)

[44]

Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: Towards an Artificial Neural Network-Based Simulator for Behavioural Evolution in Evolutionary Robotics. In: Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, pp. 170---178. ACM (2009)

[45]

Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: A Neural Network-Based Kinematic and Light-Perception Simulator for Simple Robotic Evolution. In: IEEE Congress on Evolutionary Computation (CEC), pp. 1---8 (2010)

[46]

Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: A Neural Network Ultrasonic Sensor Simulator for Evolutionary Robotics. In: INFOCOMP 2012, the Second International Conference on Advanced Communications and Computation, pp. 54---61 (2012)

[47]

Pretorius, C.J., du Plessis, M.C., Cilliers, C.B.: Simulating robots without conventional physics: a Neural Network approach. J. Intell. Robot. Syst. 71(3---4), 319---348 (2013)

[48]

Pretorius, C.J., du Plessis, M.C., Gonsalves, J.W.: A Comparison of Neural Networks and Physics Models as Motion Simulators for Simple Robotic Evolution. In: IEEE Congress on Evolutionary Computation (CEC) (2014)

[49]

Riedmiller, M., Braun, H.: A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm. In: IEEE International Conference on Neural Networks (1993)

[50]

Schneider, S., Igel, C., Klaes, C., Dinse, H.R., Wiemer, J.C.: Evolutionary adaptation of nonlinear dynamical systems in computational neuroscience. Genet. Program Evolvable Mach. 5(2), 215---227 (2004)

Digital Library

[51]

Stanley, K.O., Miikkulainen, R.: Evolving Neural Networks Through Augmenting Topologies. Evol. Comput. 10(2), 99---127 (2002)

Digital Library

[52]

Sutton, R., Barto, A.: Reinforcement Learning: an Introduction. MIT Press, Cambridge (1998)

[53]

Whitley, D., Dominic, S., Das, R., Anderson, C.W.: Genetic Reinforcement Learning for neurocontrol problems. Mach. Learn. 13(2), 259---284 (1993)

Digital Library

[54]

Wieland, A.: Evolving Neural Network Controllers for Unstable Systems. In: International Joint Conference on Neural Networks (1991)

[55]

Yamamoto, Y.: NXTway-GS Model-Based Design - Control of self-balancing two-wheeled robot built with LEGO Mindstorms NXT. Online (2008). http://www.pages.drexel.edu/dml46/Tutorials/BalancingBot/files/NXTway-GS

[56]

Yi, J., Yubazaki, N.: Stabilization fuzzy control of inverted pendulum systems. Artif. Intell. Eng. 14(2), 153---163 (2000)

[57]

Zagal, J.C., Ruiz-del Solar, J.: Combining simulation and reality in Evolutionary Robotics. J. Intell. Robot. Syst. 50(1), 19---39 (2007)

Digital Library

Cited By

El Saliby MNadizar GSalvato EMedvet ELi XHandl J(2024)Eventually, all you need is a simple evolutionary algorithm (for neuroevolution of continuous control policies)Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664112(1904-1913)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664112
de Villiers JCalitz AGreyling J(2020)An Evaluation of a Bootstrapped Neuro-Evolution ApproachConference of the South African Institute of Computer Scientists and Information Technologists 202010.1145/3410886.3410896(156-167)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1145/3410886.3410896
Woodford Gdu Plessis MTakadama K(2018)Robotic snake simulation using ensembles of artificial neural networks in evolutionary roboticsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3205455.3205507(173-180)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3205455.3205507
Show More Cited By

Neuroevolution of Inverted Pendulum Control: A Comparative Study of Simulation Techniques
1. Computing methodologies
  1. Modeling and simulation
    1. Model development and analysis

Recommendations

PID Control of Inverted Pendulum Using Adams and Matlab Co-Simulation
ICCMA '16: Proceedings of the 4th International Conference on Control, Mechatronics and Automation

This research is aimed at developing a multi-body simulation model and control of an inverted pendulum. A virtual prototype of the inverted pendulum is built by using MSC Adams software and the plant model is exported to MATLAB. It is co-simulated in ...
Optimal Control of Nonlinear Inverted Pendulum System Using PID Controller and LQR: Performance Analysis Without and With Disturbance Input

Linear quadratic regulator (LQR) and proportional-integral-derivative (PID) control methods, which are generally used for control of linear dynamical systems, are used in this paper to control the nonlinear dynamical system. LQR is one of the optimal ...
Fuzzy control of a real time inverted pendulum system

In this study, a real-time control of the cart-pole inverted pendulum system was developed using fuzzy logic controller (FLC). Swing-up and stabilization of the inverted pendulum were implemented directly in the FLC. The FLC designed in the Matlab-...

Comments

Information & Contributors

Information

Published In

cover image Journal of Intelligent and Robotic Systems

Journal of Intelligent and Robotic Systems Volume 86, Issue 3-4

June 2017

358 pages

ISSN:0921-0296

Issue’s Table of Contents

Copyright © Copyright © 2017 Springer Science+Business Media Dordrecht.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

El Saliby MNadizar GSalvato EMedvet ELi XHandl J(2024)Eventually, all you need is a simple evolutionary algorithm (for neuroevolution of continuous control policies)Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664112(1904-1913)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664112
de Villiers JCalitz AGreyling J(2020)An Evaluation of a Bootstrapped Neuro-Evolution ApproachConference of the South African Institute of Computer Scientists and Information Technologists 202010.1145/3410886.3410896(156-167)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1145/3410886.3410896
Woodford Gdu Plessis MTakadama K(2018)Robotic snake simulation using ensembles of artificial neural networks in evolutionary roboticsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3205455.3205507(173-180)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3205455.3205507
Clemente EMeza-Sánchez MBugarin EAguilar-Bustos A(2018)Adaptive Behaviors in Autonomous Navigation with Collision Avoidance and Bounded Velocity of an Omnidirectional Mobile RobotJournal of Intelligent and Robotic Systems10.1007/s10846-017-0751-y92:2(359-380)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s10846-017-0751-y

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents