Synaptic motor adaptation: A three-factor learning rule for adaptive robotic control in spiking neural networks
Article No.: 1, Pages 1 - 9
Abstract
Legged robots operating in real-world environments must possess the ability to rapidly adapt to unexpected conditions, such as changing terrains and varying payloads. This paper introduces the Synaptic Motor Adaptation (SMA) algorithm, a novel approach to achieving real-time online adaptation in quadruped robots through the utilization of neuroscience-derived rules of synaptic plasticity with three-factor learning. To facilitate rapid adaptation, we metaoptimize a three-factor learning rule via gradient descent to adapt to uncertainty by approximating an embedding produced by privileged information using only locally accessible onboard sensing data. Our algorithm performs similarly to state-of-the-art motor adaptation algorithms and presents a clear path toward achieving adaptive robotics with neuromorphic hardware.
References
[1]
Raibert, M. H. Legged robots that balance (MIT press, 1986).
[2]
Raibert, M., Blankespoor, K., Nelson, G. & Playter, R. Bigdog, the rough-terrain quadruped robot. IFAC Proceedings Volumes 41, 10822--10825 (2008).
[3]
Feng, S., Whitman, E., Xinjilefu, X. & Atkeson, C. G. Optimization based full body control for the atlas robot. In 2014 IEEE-RAS International Conference on Humanoid Robots, 120--127 (IEEE, 2014).
[4]
Kuindersma, S. et al. Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots 40, 429--455 (2016).
[5]
Yang, Y. et al. Data efficient reinforcement learning for legged robots. In Conference on Robot Learning, 1--10 (PMLR, 2020).
[6]
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. & Hutter, M. Learning quadrupedal locomotion over challenging terrain. Science robotics 5, eabc5986 (2020).
[7]
Rudin, N., Hoeller, D., Reist, P. & Hutter, M. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, 91--100 (PMLR, 2022).
[8]
Höfer, S. et al. Perspectives on sim2real transfer for robotics: A summary of the r: Ss 2020 workshop. arXiv preprint arXiv:2012.03806 (2020).
[9]
Painkras, E. et al. Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation. IEEE Journal of Solid-State Circuits 48, 1943--1953 (2013).
[10]
Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. CoRR abs/1603.08270 (2016). URL http://arxiv.org/abs/1603.08270.1603.08270.
[11]
Davies, M. et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82--99 (2018).
[12]
Pehle, C. et al. The brainscales-2 accelerated neuromorphic system with hybrid plasticity. Frontiers in Neuroscience 16 (2022).
[13]
Jin, X., Rast, A., Galluppi, F., Davies, S. & Furber, S. Implementing spike-timing-dependent plasticity on spinnaker neuromorphic hardware. In The 2010 international joint conference on neural networks (IJCNN), 1--8 (IEEE, 2010).
[14]
Vertechi, P., Brendel, W. & Machens, C. K. Unsupervised learning of an efficient short-term memory network. Advances in neural information processing systems 27 (2014).
[15]
Kaiser, J., Mostafa, H. & Neftci, E. Synaptic plasticity dynamics for deep continuous local learning (decolle). Frontiers in Neuroscience 14, 424 (2020).
[16]
Wu, Y. et al. Brain-inspired global-local learning incorporated with neuromorphic computing. Nature Communications 13, 65 (2022).
[17]
Frémaux, N. & Gerstner, W. Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules. Frontiers in neural circuits 9, 85 (2016).
[18]
Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D. & Brea, J. Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. Frontiers in neural circuits 12, 53 (2018).
[19]
Bellec, G. et al. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications 11, 3625 (2020).
[20]
Schmidgall, S., Ashkanazy, J., Lawson, W. & Hays, J. Spikepropamine: Differentiable plasticity in spiking neural networks. Frontiers in neurorobotics 120 (2021).
[21]
Kumar, A., Fu, Z., Pathak, D. & Malik, J. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034 (2021).
[22]
Kumar, A. et al. Adapting rapid motor adaptation for bipedal robots. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1161--1168 (IEEE, 2022).
[23]
Agarwal, A., Kumar, A., Malik, J. & Pathak, D. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning, 403--415 (PMLR, 2023).
[24]
Qi, H., Kumar, A., Calandra, R., Ma, Y. & Malik, J. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, 1722--1732 (PMLR, 2023).
[25]
Fu, Z., Cheng, X. & Pathak, D. Deep whole-body control: learning a unified policy for manipulation and locomotion. In Conference on Robot Learning, 138--149 (PMLR, 2023).
[26]
Schmidgall, S. & Hays, J. Learning to learn online with neuromodulated synaptic plasticity in spiking neural networks. bioRxiv 2022--06 (2022).
[27]
Schmidgall, S. & Hays, J. Meta-spikepropamine: Learning to learn with synaptic plasticity in spiking neural networks. Frontiers in neuroscience (2023).
[28]
Citri, A. & Malenka, R. C. Synaptic plasticity: multiple forms, functions, and mechanisms. Neuropsychopharmacology 33, 18--41 (2008).
[29]
Abraham, W. C., Jones, O. D. & Glanzman, D. L. Is plasticity of synapses the mechanism of long-term memory storage?.
[30]
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. nature 323, 533--536 (1986).
[31]
Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J. & Hinton, G. Backpropagation and the brain. Nature Reviews Neuroscience 21, 335--346 (2020).
[32]
Caporale, N. & Dan, Y. Spike timing-dependent plasticity: a hebbian learning rule. Annu. Rev. Neurosci. 31, 25--46 (2008).
[33]
Gerstner, W., Kistler, W. M., Naud, R. & Paninski, L. Neuronal dynamics: From single neurons to networks and models of cognition (Cambridge University Press, 2014).
[34]
Bellec, G. et al. Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets. arXiv preprint arXiv:1901.09049 (2019).
[35]
Aitchison, L. et al. Synaptic plasticity as bayesian inference. Nature neuroscience 24, 565--571 (2021).
[36]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
[37]
Schulman, J., Moritz, P., Levine, S., Jordan, M. & Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015).
[38]
Manngård, M., Kronqvist, J. & Böling, J. M. Structural learning in artificial neural networks using sparse optimization. Neurocomputing 272, 660--667 (2018).
[39]
Mordvintsev, A., Randazzo, E., Niklasson, E. & Levin, M. Growing neural cellular automata. Distill 5, e23 (2020).
[40]
Najarro, E., Sudhakaran, S., Glanois, C. & Risi, S. Hypernca: Growing developmental networks with neural cellular automata. arXiv preprint arXiv:2204.11674 (2022).
[41]
Gilpin, W. Cellular automata as convolutional neural networks. Physical Review E 100, 032402 (2019).
[42]
Schmidgall, S. Self-constructing neural networks through random mutation. arXiv preprint arXiv:2103.15692 (2021).
[43]
Kepecs, A., Van Rossum, M. C., Song, S. & Tegner, J. Spike-timing-dependent plasticity: common themes and divergent vistas. Biological cybernetics 87, 446--458 (2002).
Index Terms
- Synaptic motor adaptation: A three-factor learning rule for adaptive robotic control in spiking neural networks
Recommendations
A Simple Aplysia-Like Spiking Neural Network to Generate Adaptive Behavior in Autonomous Robots
In this article, we describe an adaptive controller for an autonomous mobile robot with a simple structure. Sensorimotor connections were made using a three-layered spiking neural network (SNN) with only one hidden-layer ...
A new synaptic plasticity rule for networks of spiking neurons
In this paper, we describe a new Synaptic Plasticity Activity Rule (SAPR) developed for use in networks of spiking neurons. Such networks can be used for simulations of physiological experiments as well as for other computations like image analysis. ...
Supervised associative learning in spiking neural network
ICANN'10: Proceedings of the 20th international conference on Artificial neural networks: Part IIn this paper, we propose a simple supervised associative learning approach for spiking neural networks. In an excitatory-inhibitory network paradigm with Izhikevich spiking neurons, synaptic plasticity is implemented on excitatory to excitatory ...
Comments
Information & Contributors
Information
Published In
August 2023
270 pages
ISBN:9798400701757
DOI:10.1145/3589737
- Conference Chair:
- Catherine Schuman,
- Program Co-chairs:
- Melika Payvand,
- Maryam Parsa
Copyright © 2023 Owner/Author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 August 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
ICONS '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 13 of 22 submissions, 59%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 140Total Downloads
- Downloads (Last 12 months)103
- Downloads (Last 6 weeks)19
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in