Abstract
We consider the task of using a neural network for controlling a quadrotor drone to perform flight maneuvers. For that, the network must be evaluated with high frequency on the microcontroller of the drone. In order to maintain the evaluation frequency for larger networks, we search for structures in the weight matrices of the trained network. By exploiting structures in the weight matrices, the propagation of information through the network can be made more efficient. In this paper, we focus on four structure classes, namely low rank matrices, matrices of low displacement rank, sequentially semiseparable matrices and products of sparse matrices. We approximate the trained weight matrices with matrices from each structure class and analyze the flying capabilities of the approximated neural network controller. Our results show that there is structure in the weight matrices, which can be exploited to speed up the inference, while still being able to perform the flight maneuvers in the real world. The best results were obtained with products of sparse matrices, which could even outperform non-approximated networks with the same number of parameters in some cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Blalock, D., Ortiz, J.J.G., Frankle, J., Guttag, J.: What is the state of neural network pruning? arXiv preprint arXiv:2003.03033 (2020)
Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2013). https://doi.org/10.1007/s10107-013-0701-9
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2(4), 303–314 (1989)
Dao, T., Gu, A., Eichhorn, M., Rudra, A., Ré, C.: Learning fast algorithms for linear transforms using butterfly factorizations. In: International Conference on Machine Learning, pp. 1517–1527. PMLR (2019)
Dao, T., et al.: Kaleidoscope: an efficient, learnable representation for all structured linear maps. arXiv preprint arXiv:2012.14966 (2020)
De Sa, C., Cu, A., Puttagunta, R., Ré, C., Rudra, A.: A two-pronged progress in structured dense matrix vector multiplication. In: Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1060–1079. SIAM (2018)
Dewilde, P., Van der Veen, A.J.: Time-Varying Systems and Computations. Springer, New York (1998). https://doi.org/10.1007/978-1-4757-2817-0
Furber, S.B., Galluppi, F., Temple, S., Plana, L.A.: The spinnaker project. Proc. IEEE 102(5), 652–665 (2014)
Gronauer, S., Kissel, M., Sacchetto, L., Korte, M., Diepold, K.: Using simulation optimization to improve zero-shot policy transfer of quadrotors. arXiv preprint arXiv:2201.01369 (2022)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kung, S., Lin, D.: Optimal Hankel-norm model reductions: multivariable systems. IEEE Trans. Autom. Control 26(4), 832–852 (1981)
Le Magoarou, L., Gribonval, R.: Flexible multilayer sparse approximations of matrices and applications. IEEE J. Sel. Top. Signal Process. 10(4), 688–700 (2016)
LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Adv. Neural Inf. Processing Syst. 2 (1989)
Lee, E.H., Miyashita, D., Chai, E., Murmann, B., Wong, S.S.: LogNet: energy-efficient neural networks using logarithmic computation. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5900–5904. IEEE (2017)
Pan, V.: Structured Matrices and Polynomials: Unified Superfast Algorithms. Springer, Boston (2001). https://doi.org/10.1007/978-1-4612-0129-8
Sindhwani, V., Sainath, T.N., Kumar, S.: Structured transforms for small-footprint deep learning. arXiv preprint arXiv:1510.01722 (2015)
Sze, V., Chen, Y.H., Yang, T.J., Emer, J.S.: Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)
Thomas, A.T., Gu, A., Dao, T., Rudra, A., Ré, C.: Learning compressed transforms with low displacement rank. Adv. Neural. Inf. Process. Syst. 2018, 9052 (2018)
Zhao, L., Liao, S., Wang, Y., Li, Z., Tang, J., Yuan, B.: Theoretical properties for neural networks with weight matrices of low displacement rank. In: International Conference on Machine Learning, pp. 4082–4090. PMLR (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kissel, M., Gronauer, S., Korte, M., Sacchetto, L., Diepold, K. (2022). Exploiting Structures in Weight Matrices for Efficient Real-Time Drone Control with Neural Networks. In: Marreiros, G., Martins, B., Paiva, A., Ribeiro, B., Sardinha, A. (eds) Progress in Artificial Intelligence. EPIA 2022. Lecture Notes in Computer Science(), vol 13566. Springer, Cham. https://doi.org/10.1007/978-3-031-16474-3_43
Download citation
DOI: https://doi.org/10.1007/978-3-031-16474-3_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16473-6
Online ISBN: 978-3-031-16474-3
eBook Packages: Computer ScienceComputer Science (R0)