Abstract
This chapter describes an evolutionary approach to deep learning networks. We first explain neuroevolution approach, which can adaptively learn a network structure and size appropriate to the task. A typical example of neuroevolution is NEAT. NEAT has demonstrated performance superior to that of conventional methods in a large number of problems. Then, several studies on deep neural networks with evolutionary optimization are explained, such as Genetic CNNs, hierarchical feature construction using GP, and Differentiable pattern-producing network (DPPSN).
A sonnet written by a machine would be better appreciated by another machine.
(Alan Turing)
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A columnar structure in the cerebrum where neurons with similar properties concentrate. In mouse perceptual fields, there are believed to be columns corresponding to each individual whisker.
- 2.
This image was created by LGPC for Art, a simulator created with reference to Sbart. This tool can be used to “nurture” to influence their creation. See our Web site for information on installing and using LGPC for Art.
- 3.
Here hypot gives the Euclidean distance on two dimensions, defined as \(hypot(x,y)=\sqrt{x^2+y^2}\).
- 4.
- 5.
During training, each layer’s inputs are normalized across the current minibatch to the Gaussian distributions (usually zero mean and unit variance). It has been shown to have several benefits, e.g., faster convergence, easier to escape from local optima, more robust network.
- 6.
A function outputting 0 when the input is 0 or less and outputting the input as it is when the input is greater than 1.
- 7.
BCE is a loss function \(\mathcal {L}\) commonly used for a binary classification, which is a special case of multiclass cross-entropy. The definition is given as follows: \(\mathcal {L}(\theta )= -\frac{1}{n}\sum _{i=1}^n \left[ y_i \log (p_i) + (1-y_i) \log (1-p_i)\right] \), where n is the number of samples, \(y_i\) is the sample label of ith sample, and \(p_i\) is the prediction for the ith sample. Smaller values indicate a better prediction.
- 8.
- 9.
1623 different handwritten characters from 50 different alphabets. See https://github.com/brendenlake/omniglot for details.
References
Fernando, C., Banarse, D., Reynolds, M., Besse, F., Pfau, D., Jaderberg, M., Lanctot, M., Wierstra, D.: Convolution by evolution–differentiable pattern producing networks. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016. (GECCO16), pp. 109–116 (2016)
Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone, P.: HyperNEAT-GGP: A HyperNEAT-based Atari general game player. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2012), pp. 217–224 (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Huang, G., Liu, Z., Weinbergerz, K.: Densely connected convolutional networks. In: Proceedings of Computer Vision and Pattern Recognition (CVPR2017) (2017)
Iba, H.: Agent-Based Modeling and Simulation with Swarm. Chapman and Hall/CRC, London (2013)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp. 448–456 (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR2015)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report 1. Computer Science Department, University of Toronto (2009)
Krizhevsky, A., Sutskerver, I. Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NIPS), pp. 1097–1105 (2012)
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks (2017). arXiv:1703.00548
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.: Reading digits in natural images with unsupervised feature learning. In: Proceedings of NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)
Rozenberg, G. (ed.): The Book of L. Springer, Berlin (1986)
Schrum, J., Miikkulainen, R.: Evolving multimodal behavior with modular neural networks in Ms. Pac-Man. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 325–332 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations (2014)
Stanley, K.O.: Compositional pattern producing networks: a novel abstraction of development. Genet. Program. Evolvable Mach. (Special Issue on Dev. Syst.) 8(2), 131–162 (2007)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)
Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation. Conference 2017 (GECCO2017), pp. 497–504 (2017)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of Computer Vision and Pattern Recognition (CVPR2016) (2016)
Unemi, T.: SBART2.4: Breeding 2D CG images and movies, and creating a type of collage. In: Proceedings of The Third International Conference on Knowledge-based Intelligent Information Engineering Systems, pp. 288–291 (1999)
Xie, L., Yuille, A.: Genetic CNN. In: Proceedings of IEEE International Conference on Computer Vision (ICCV)
Yao, X.: A review of evolutionary artificial neural networks. Int. J. Intell. Syst. 8, 539–567 (1993)
Zagoruyko, S., Komodakis, N.: Wide residual networks (2016). arXiv: 1605.07146
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Iba, H. (2018). Evolutionary Approach to Deep Learning. In: Evolutionary Approach to Machine Learning and Deep Neural Networks. Springer, Singapore. https://doi.org/10.1007/978-981-13-0200-8_3
Download citation
DOI: https://doi.org/10.1007/978-981-13-0200-8_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0199-5
Online ISBN: 978-981-13-0200-8
eBook Packages: Computer ScienceComputer Science (R0)