Abstract
This article presents an evolutionary algorithm to autonomously construct full-connected multilayered feedforward neural architectures. This algorithm employs grammar-guided genetic programming with a context-free grammar that has been specifically designed to satisfy three important restrictions. First, the sentences that belong to the language produced by the grammar only encode all valid neural architectures. Second, full-connected feedforward neural architectures of any size can be generated. Third, smaller-sized neural architectures are favored to avoid overfitting. The proposed evolutionary neural architectures construction system is applied to compute the terms of the two sequences that define the three-term recurrence relation associated with a sequence of orthogonal polynomials. This application imposes an important constraint: training datasets are always very small. Therefore, an adequate sized neural architecture has to be evolved to achieve satisfactory results, which are presented in terms of accuracy and size of the evolved neural architectures, and convergence speed of the evolutionary process.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abdul-Rahman, O., Munetomo, M., Akama, K.: An improved binary-real coded genetic algorithm for real parameter optimization. In: Proceedings of 3rd World Congress on Nature and Biologically Inspired Computing (2011)
Alonso, F., Martínez, L., Pérez, A., Santamaría, A., Valente, J.: Modeling medical time series using grammar-guided genetic programming. In: Proceedings of the 8th Industrial Conference on Advances in Data Mining: Medical Applications, E-Commerce, Marketing, and Theoretical Aspects (2008)
Augasta, M., Kathirvalavakumar, T.: Pruning algorithms of neural networks - a comparative study. Cent. Eur. J. Comput. Sci. 3(3), 105–115 (2013)
Barrios Rolanía, D., López Lagomasino, G., Torrano, E.: Location of zeros and asymptotics of polynomials satisfying three-terms recurrence relations with complex coefficients. Russ. Acad. Sci. Sb. Math. 80(2), 309–333 (1995)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)
Bernardos, P., Vosniakos, G.: Optimizing feedforward artificial neural network architecture. Eng. Appl. Artif. Intel. 20, 365–382 (2007)
Cecchi, M.M., Zaglia, M.R.: Computing the coefficients of a recurrence formula for numerical integration by moments and modified moments. J. Comput. Appl. Math. 49, 207–216 (1993)
Chebyshev, P.L.: Sur les fractions continues. J. Math. Pures Appl. 2(3), 289–270 (1858)
Cheney, E.: Introduction to Approximation Theory. AMS Chelsea Pub, Providence (2000)
Chihara, T.: An Introduction to Orthogonal Polynomials. Gordon and Breach, New York (1978)
Couchet, J., Manrique, D., Porras, J.: Grammar-guided neural architecture evolution. Lect. Notes Comput. Sci 4527, 437–446 (2007)
De Jong, K.: Evolutionary Computation: a Unified Approach. MIT Press, Cambridge (2006)
Dorado, J.: Cooperative strategies to select automatically training patterns and neural architectures with genetic algorithms. University of La Corua. Spain, Ph.D. Thesis (1999)
Fernando, C., Banarse, D., Besse, F., Jaderberg, M., Pfau, D., Reynolds, M., Lactot, M., Wierstra, D.: Convolution by evolution: Differentiable pattern producing networks. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO 2016. Denver (2016)
Font, J., Manrique, D., Ramos Criado, P., del Río, D.: Partition based real-valued encoding scheme for evolutionary algorithms. Nat. Comput. 15(3), 477–492 (2016)
Font, J., Manrique, D., Ríos, J.: Redes de neuronas artificiales y computación evolutiva. Madrid, Spain. Fundación General de la Universidad Politécnica de Madrid (2009)
Garro, B., Sossa, H., Vázquez, R.: Artificial neural network synthesis by means of artificial bee colony (abc) algorithm. In: Proceedings of IEEE Congress on Evolutionary Computation (2011)
Gautschi, W.: Computational aspects of three-term recurrence relations. SIAM Rev. 9(1), 24–82 (1967)
Gautschi, W.: Orthogonal Polynomials: Computation and Approximation. Numerical Mathematics and Scientific Computation. Oxford University Press, New York (2004)
Gautschi, W.: Orthogonal Polynomials in Matlab. Exercises and Solutions. Software, Environments and Tools. SIAM, Philadelphia (2016)
Golub, G.H., Welsch, J.H.: Calculation of gauss quadrature rules. Math. Comp. 23(106), 221–230 (1969)
Han, H., Qiao, J.: A structure optimisation algorithm for feedforward neural network construction. Neurocomputing 99, 347–357 (2013)
Haykin, S.: Neural Networks and Learning Machines. Pearson, Upper Saddle River (2010)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580v1 (2012)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991)
Huang, G., Chen, L., Siew, C.: Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 17(4), 879–892 (2006)
Kari, L., Rozenberg, G.: The many facets of natural computing. Commun. ACM 51(10), 72–83 (2008)
Koza, J.: Genetic programming: on the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)
Loiacono, D., Cardamone, L., Lanzi, P.: Automatic track generation for high-end racing games using evolutionary computation. IEEE Trans. Comput. Intell. AI Games 3(3), 245–259 (2011)
Loshchilov, I., Hutter, F.: CMA-ES For hyperparameter optimization of deep neural networks.arXiv:1604.07269v1 (2016)
Maas, A., Hannun, A., Ng, A.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML Workshop on Deep Learning for Audio, Speech and Language Processing (2013)
Manrique, D., Ríos, J., Rodr(́i)guez-Patón, A.: Artificial Neural Networks in Real Life Applications, Ch. Self-Adapting Neutral Intelligent Systems Using Evolutionary Techniques. Idea Group, Barcelona (2006)
Mathai, A., Haubold, H.: Erdélyi-kober fractional integral operators from a statistical perspective II. Cogent Math. 4(1). https://doi.org/10.1080/23311835.2017.1309769 (2017)
McKay, R., Hoai, N., Whigham, P., Shan, Y., O’Neill, M.: Grammar-based genetic programming: a survey. GGenet. Program Evolvable Mach. 11(3), 365–396 (1988)
Miikkulainen, R.: Encyclopedia of Machine Learning, Chap. Neuroevolution. Springer, New York (2010)
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. arXiv:1703.00548v2 (2017)
Nielsen, M.: Neural Networks and Deep Learning. Determination Press (2015)
Poli, R., Langdon, W., McPhee, N., Koza, J.: A Field Guide to Genetic Programming. www.Lulu.com, UK (2008)
Principe, J.C., Euliano, N.R., Lefebvre, W.C.: Neural and Adaptive Systems, Fundamentals Through Simulations. Wiley, New York (2000)
Pulina, L., Tacchella, A.: NeVer: a tool for artificial neural networks verification. Ann. Math. Artif. Intell. 62(3), 403–425 (2011)
Ramos Criado, P.: New techniques for grammar guided genetic programming: dealing with large derivation trees and high cardinality terminal symbol sets. Ph.D. Thesis, Universidad Politécnica de Madrid. https://doi.org/10.20868/UPM.thesis.48795 (2017)
Samarasinghe, S.: Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition. CRC Press, Boca Raton (2017)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)
Siddiqi, A., Lucas, S.: A comparison of matrix rewriting versus direct encoding method for evolving neural networks. In: Proceedings of 1998 IEEE International Conference on Evolutionary Computation (1998)
Simon, D.: Evolutionary Optimization Algorithms. Wiley, New York (2013)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10, 99–127 (2002)
Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)
Sullivan, T.J.: Orthogonal Polynomials and Applications, pp 133–164. Springer, New York (2015)
Tapas, S., Simanta, H., Jana, N.: Artificial neural network training using differential evolutionary algorithm for classification. Adv. Intel. Soft Comp. 132, 769–778 (2012)
Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, pp 1058–1066. PMLR, Atlanta (2013)
Wang, J., Wang, H., Chen, Y., Liu, C.: A constructive algorithm for unsupervised learning with incremental neural network. J. Appl. Res. Technol. 13(2), 188–196 (2015)
Whigham, P.: Grammatically-based genetic programming. In: Proceedings of the Workshop on Genetic Programming: From Theory to Real-World Applications (1995)
Yu, X., Gen, M.: Introduction to Evolutionary Algorithms. Springer, London (2010)
Zayernouri, M., Zhang, Z.: New trends in applications of orthogonal polynomials and special functions. Math 3 (special issue) (2016)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv:1611.01578v2 (2016)
Acknowledgements
Thanks to the referees and editors for their valuable comments and suggestions that helped us to improve the quality and contents of the article.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was partially supported by research grant MTM2014-54053-P of Ministerio de Economía y Competitividad, Spain.
Rights and permissions
About this article
Cite this article
Barrios Rolanía, D., Delgado Martínez, G. & Manrique, D. Multilayered neural architectures evolution for computing sequences of orthogonal polynomials. Ann Math Artif Intell 84, 161–184 (2018). https://doi.org/10.1007/s10472-018-9601-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-018-9601-2
Keywords
- Evolutionary computation
- Grammar-guided genetic programming
- Artificial neural networks
- Orthogonal polynomials