Abstract
Deep Learning opened artificial intelligence to an unprecedented number of new applications. A critical success factor is the ability to train deeper neural networks, striving for stable and accurate models. This translates into Artificial Neural Networks (ANN) that become unmanageable as the number of features increases. The novelty of our approach is to employ Network Science strategies to tackle the complexity of the actual ANNs at each epoch of the training process. The work presented herein originates in our earlier publications, where we explored the acceleration effects obtained by enforcing, in turn, scale freeness, small worldness, and sparsity during the ANN training process. The efficiency of our approach has also been recently confirmed by independent researchers, who managed to train a million-node ANN on non-specialized laptops. Encouraged by these results, we have now moved into having a closer look at some tunable parameters of our previous approach to pursue a further acceleration effect. We now investigate on the revise fraction parameter, to verify the necessity of the role of its double-check. Our method is independent of specific machine learning algorithms or datasets, since we operate merely on the topology of the ANNs. We demonstrate that the revise phase can be avoided in order to half the overall execution time with an almost negligible loss of quality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Barabási, A.L., Pósfai, M.: Network Science. Cambridge University Press, Cambridge (2016). http://barabasi.com/networksciencebook/
Berman, D.S., Buczak, A., Chavis, J., Corbett, C.: A survey of deep learning methods for cyber security. Information 10, 122 (2019). https://doi.org/10.3390/info10040122
Cao, C., et al.: Deep learning and its applications in biomedicine. Genomics Proteomics Bioinform. 16(1), 17–32 (2018). https://doi.org/10.1016/j.gpb.2017.07.003
Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., Blaschke, T.: The rise of deep learning in drug discovery. Drug Discov. Today 23(6), 1241–1250 (2018). https://doi.org/10.1016/j.drudis.2018.01.039
Erdös, P., Rényi, A.: On random graphs I. Publ. Math. Debr. 6, 290–297 (1959)
Gardner, M., Dorling, S.: Artificial neural networks (the multilayer perceptron)–a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)
Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org
Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012). https://doi.org/10.1109/MSP.2012.2205597
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386
Latora, V., Nicosia, V., Russo, G.: Complex Networks: Principles, Methods and Applications. Cambridge University Press, Cambridge (2017)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nat. Cell Biol. 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539
Liu, S., Mocanu, D.C., Matavalam, A., Pei, Y., Pechenizkiy, M.: Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. arXiv:1901.09181 (2019)
Mocanu, D.C., Mocanu, E., Stone, P., Nguyen, P., Gibescu, M., Liotta, A.: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nat. Commun. 9, 2383 (2018). https://doi.org/10.1038/s41467-018-04316-3
Ruano-Ordás, D., Yevseyeva, I., Fernandes, V.B., Méndez, J.R., Emmerich, M.T.M.: Improving the drug discovery process by using multiple classifier systems. Expert Syst. Appl. 121, 292–303 (2019). https://doi.org/10.1016/j.eswa.2018.12.032
Yu, D., Deng, L.: Deep learning and its applications to signal and information processing [exploratory DSP]. IEEE Signal Process. Mag. 28(1), 145–154 (2011). https://doi.org/10.1109/MSP.2010.939038
Acknowledgments
We thank Dr Decebal Costantin Mocanu for providing constructive feedback.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Cavallaro, L., Bagdasar, O., De Meo, P., Fiumara, G., Liotta, A. (2020). Artificial Neural Networks Training Acceleration Through Network Science Strategies. In: Sergeyev, Y., Kvasov, D. (eds) Numerical Computations: Theory and Algorithms. NUMTA 2019. Lecture Notes in Computer Science(), vol 11974. Springer, Cham. https://doi.org/10.1007/978-3-030-40616-5_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-40616-5_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-40615-8
Online ISBN: 978-3-030-40616-5
eBook Packages: Computer ScienceComputer Science (R0)