Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Can Differential Evolution Be an Efficient Engine to Optimize Neural Networks?

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Big Data (MOD 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10710))

Included in the following conference series:

  • 3161 Accesses

Abstract

In this paper we present an algorithm that optimizes artificial neural networks using Differential Evolution. The evolutionary algorithm is applied according the conventional neuroevolution approach, i.e. to evolve the network weights instead of backpropagation or other optimization methods based on backpropagation. A batch system, similar to that one used in stochastic gradient descent, is adopted to reduce the computation time. Preliminary experimental results are very encouraging because we obtained good performance also in real classification dataset like MNIST, that are usually considered prohibitive for this kind of approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Source code available at https://github.com/Gabriele91/DENN.

  2. 2.

    https://archive.ics.uci.edu/ml/datasets.

  3. 3.

    http://yann.lecun.com/exdb/mnist/.

References

  1. Bari, G.D.: Denn: Differential evolution for neural networks. Master thesis (2017)

    Google Scholar 

  2. Bengio, Y., Goodfellow, I.J., Courville, A.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

  3. Brest, J., Boskovic, B., Mernik, M., Zumer, V.: Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Trans. Evol. Comput. 10(6), 646–657 (2006)

    Article  Google Scholar 

  4. Cardamone, L., Loiacono, D., Lanzi, P.L.: Evolving competitive car controllers for racing games with neuroevolution. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1179–1186. ACM (2009)

    Google Scholar 

  5. Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 160–167. ACM, New York (2008)

    Google Scholar 

  6. Das, S., Abraham, A., Chakraborty, U.K., Konar, A.: Differential evolution using a neighborhood-based mutation operator. IEEE Trans. Evol. Comput. 13(3), 526–553 (2009)

    Article  Google Scholar 

  7. Das, S., Mullick, S.S., Suganthan, P.: Recent advances in differential evolution an updated survey. Swarm Evol. Comput. 27, 1–30 (2016)

    Article  Google Scholar 

  8. Das, S., Suganthan, P.N.: Differential evolution: a survey of the state-of-the-art. IEEE Trans. Evol. Comput. 15(1), 4–31 (2011)

    Article  Google Scholar 

  9. Donate, J.P., Li, X., Sánchez, G.G., de Miguel, A.S.: Time series forecasting by evolving artificial neural networks with genetic algorithms, differential evolution and estimation of distribution algorithm. Neural Comput. Appl. 22(1), 11–20 (2013)

    Article  Google Scholar 

  10. Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: from architectures to learning. Evol. Intell. 1(1), 47–62 (2008)

    Article  Google Scholar 

  11. Graves, A., Wayne, G., Danihelka, I.: Neural turing machines. arXiv preprint arXiv:1410.5401 (2014)

  12. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J., et al.: Hybrid computing using a neural network with dynamic external memory. Nature 538(7626), 471–476 (2016)

    Article  Google Scholar 

  13. Hausknecht, M., Lehman, J., Miikkulainen, R., Stone, P.: A neuroevolution approach to general atari game playing. IEEE Trans. Comput. Intell. AI Games 6(4), 355–366 (2014)

    Article  Google Scholar 

  14. Heidrich-Meisner, V., Igel, C.: Neuroevolution strategies for episodic reinforcement learning. J. Algorithms 64(4), 152–168 (2009)

    Article  Google Scholar 

  15. Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)

    Article  Google Scholar 

  16. Igel, C.: Neuroevolution for reinforcement learning using evolution strategies. In: The 2003 Congress on Evolutionary Computation, 2003. CEC 2003, vol. 4, pp. 2588–2595 (2003)

    Google Scholar 

  17. Ilonen, J., Kamarainen, J.K., Lampinen, J.: Differential evolution training algorithm for feed-forward neural networks. Neural Process. Lett. 17(1), 93–105 (2003)

    Article  Google Scholar 

  18. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc. (2012)

    Google Scholar 

  19. Leema, N., Nehemiah, H.K., Kannan, A.: Neural network classifier optimization using differential evolution with global information and back propagation algorithm for clinical datasets. Appl. Soft Comput. 49, 834–844 (2016). http://www.sciencedirect.com/science/article/pii/S1568494616303866

    Article  Google Scholar 

  20. Masters, T., Land, W.: A new training algorithm for the general regression neural network. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics, vol. 3, pp. 1990–1994 (1997)

    Google Scholar 

  21. Mattiussi, C., Dürr, P., Floreano, D.: Center of mass encoding: a self-adaptive representation with adjustable redundancy for real-valued parameters. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO 2007, pp. 1304–1311. ACM, New York (2007)

    Google Scholar 

  22. Miikkulainen, R.: Neuroevolution, pp. 716–720. Springer, Boston (2010). https://doi.org/10.1007/978-0-387-30164-8_589

    Book  Google Scholar 

  23. Morse, G., Stanley, K.O.: Simple evolutionary optimization can rival stochastic gradient descent in neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2016, pp. 477–484. ACM, New York (2016)

    Google Scholar 

  24. Piotrowski, A.P.: Differential evolution algorithms applied to neural network training suffer from stagnation. Appl. Soft Comput. 21, 382–406 (2014)

    Article  Google Scholar 

  25. Price, K., Storn, R.M., Lampinen, J.A.: Differential Evolution: A Practical Approach to Global Optimization. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-31306-0

    Book  MATH  Google Scholar 

  26. Reed, S., de Freitas, N.: Neural programmer-interpreters. Technical report, arXiv:1511.06279 (2015). http://arxiv.org/abs/1511.06279

  27. Santucci, V., Baioletti, M., Milani, A.: Algebraic differential evolution algorithm for the permutation flowshop scheduling problem with total flowtime criterion. IEEE Trans. Evol. Comput. 20(5), 682–694 (2016)

    Article  Google Scholar 

  28. Schaffer, J.D., Whitley, D., Eshelman, L.J.: Combinations of genetic algorithms and neural networks: a survey of the state of the art. In: Proceedings of COGANN 1992: International Workshop on Combinations of Genetic Algorithms and Neural Networks, pp. 1–37 (1992)

    Google Scholar 

  29. Schraudolph, N.N., Belew, R.K.: Dynamic parameter encoding for genetic algorithms. Mach. Learn. 9(1), 9–21 (1992)

    Google Scholar 

  30. Tracolli, M.: Enhancing denn with adaboost and self adaptation. Master thesis (2017)

    Google Scholar 

  31. Vesterstrom, J., Thomsen, R.: A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems. In: Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753), vol. 2, pp. 1980–1987 (2004)

    Google Scholar 

  32. Wang, L., Zeng, Y., Chen, T.: Back propagation neural network with adaptive differential evolution algorithm for time series forecasting. Expert Syst. Appl. 42(2), 855–863 (2015)

    Article  Google Scholar 

  33. Yao, X.: Evolving artificial neural networks. Proc. IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Perugia, Perugia, Italy

    Marco Baioletti, Gabriele Di Bari, Valentina Poggioni & Mirco Tracolli

Authors
  1. Marco Baioletti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriele Di Bari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Valentina Poggioni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mirco Tracolli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valentina Poggioni .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  3. University of Catania, Catania, Italy

    Giovanni Giuffrida

  4. Harvard University, Cambridge, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Baioletti, M., Di Bari, G., Poggioni, V., Tracolli, M. (2018). Can Differential Evolution Be an Efficient Engine to Optimize Neural Networks?. In: Nicosia, G., Pardalos, P., Giuffrida, G., Umeton, R. (eds) Machine Learning, Optimization, and Big Data. MOD 2017. Lecture Notes in Computer Science(), vol 10710. Springer, Cham. https://doi.org/10.1007/978-3-319-72926-8_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-72926-8_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-72925-1

  • Online ISBN: 978-3-319-72926-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation