Abstract
Deep Neural Networks are increasingly deployed on tiny systems such as microcontrollers or embedded systems. Notwithstanding the recent success of Deep Learning, also enabled by the availability of Automated Machine Learning and Neural Architecture Search solutions, the computational requirements of the optimization of the structure and the hyperparameters of Deep Neural Networks usually far exceed what is available on tiny systems. Therefore, the deployability becomes critical when the learned model must be deployed on a tiny system. To overcome this critical issue, we propose a framework, based on Bayesian Optimization, to optimize the hyperparameters of a Deep Neural Network by dealing with black-box deployability constraints. Encouraging results obtained on a classification benchmark problem on a real microcontroller by STMicroelectronics are presented.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Netw. 5(1), 54–65 (1994)
Miikkulainen, R., et al.: Evolving deep neural networks. arXiv:1703.00548, March 2017
Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Genetic and Evolutionary Computation Conference (2017)
Bianco, S., Buzzelli, M., Ciocca, G., Schettini, R.: Neural architecture search for image saliency fusion. Inf. Fusion 57, 89–101 (2020)
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. In: International Conference on Learning Representations (2018b)
Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. In: International Conference on Learning Representations (2017a)
Zhong, Z., Yan, J., Wu, W., Shao, J., Liu, C.L.: Practical block-wise neural network architecture generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2423–2432 (2018a)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: International Conference on Learning Representations (2017)
Jin, H., Song, Q., Hu, X.: Auto-keras: an efficient neural architecture search system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1946–1956, July 2019
Zela, A., Klein, A., Falkner, S., Hutter, F.: Towards automated deep learning: efficient joint neural architecture and hyperparameter search. In: ICML 2018 Workshop on AutoML (AutoML 2018) (2018)
Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., Xing, E.: Neural architecture search with Bayesian optimisation and optimal transport. arXiv:1802.07191, February 2018
Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. SO. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24494-1
Oh, C., Tomczak, J., Gavves, E., Welling, M.: Combinatorial Bayesian optimization using the graph cartesian product. In: Advances in Neural Information Processing Systems, pp. 2910–2920 (2019)
Baptista, R., Poloczek, M.: Bayesian optimization of combinatorial structures. arXiv preprint arXiv:1806.08838 (2018)
Elsken, T., Metzen, J. H., Hutter, F.: Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804.09081 (2018)
Dong, J.-D., Cheng, A.-C., Juan, D.-C., Wei, W., Sun, M.: DPP-Net: device-aware progressive search for pareto-optimal neural architectures. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 540–555. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_32
Antonio, C.: Sequential model based optimization of partially defined functions under unknown constraints. J. Global Optim. 1–23 (2019). https://doi.org/10.1007/s10898-019-00860-4
Saunders, C., Stitson, M.O., Weston, J., Bottou, L., Smola, A.: Support vector machine-reference manual (1998)
Candelieri, A., Galuzzi, B., Giordani, I., Perego, R., Archetti, F.: Optimizing partially defined black-box functions under unknown constraints via sequential model based optimization: an application to pump scheduling optimization in water distribution networks. In: Matsatsinis, N.F., Marinakis, Y., Pardalos, P. (eds.) LION 2019. LNCS, vol. 11968, pp. 77–93. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-38629-0_7
Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.: Gaussian process optimization in the bandit setting: no regret and experimental design. arXiv preprint arXiv:0912.3995 (2009)
Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.W.: Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theor. 58(5), 3250–3265 (2012)
Casale, P., Pujol, O., Radeva, P.: Personalization and user verification in wearable systems using biometric walking patterns. Pers. Ubiquit. Comput. 16(5), 563–580 (2012)
Acknowledgements
We greatly acknowledge the DEMS Data Science Lab of the Department of Economics Management and Statistics (DEMS) for supporting this work by providing computational resources.
We want to thank STMicroelectronics company that provided us MCUs for the experiments and the valid support from its community.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Perego, R., Candelieri, A., Archetti, F., Pau, D. (2020). Tuning Deep Neural Network’s Hyperparameters Constrained to Deployability on Tiny Systems. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-61616-8_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61615-1
Online ISBN: 978-3-030-61616-8
eBook Packages: Computer ScienceComputer Science (R0)