Abstract
Spiking Neural Networks (SNNs) are emerging as energy-efficient alternatives to artificial neural networks (ANNs) due to their event-driven computation and effective processing of temporal information. While Neural Architecture Search (NAS) has been extensively used to optimize neural network structures, its application to SNNs remains limited. Existing studies often overlook the temporal differences in information propagation between ANNs and SNNs. Instead, they focus on shared structures such as convolutional, recurrent, or pooling modules. This work introduces a novel neural architecture search framework, MixedSNN, explicitly designed for SNNs. Inspired by the human brain, MixedSNN incorporates a novel search space called SSP, which explores the impact of utilizing Mixed spiking neurons and Variable thresholds on SNN performance. Additionally, we propose a training-free evaluation strategy called Period-Based Spike Evaluation (PBSE), which leverages spike activation patterns to incorporate temporal features in SNNs. The performance of SNN architectures obtained through MixedSNN is evaluated on three datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. Results demonstrate that MixedSNN can achieve state-of-the-art performance with significantly lower timesteps.
Supported by The Belt and Road Special Foundation of the State Key Laboratory of Hydrology-Water Resources and Hydraulic Engineering under Grant 2021490811.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbott, L.F.: Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain Res. Bull. 50(5–6), 303–304 (1999)
Chen, W., Gong, X., Wang, Z.: Neural architecture search on ImageNet in four GPU hours: a theoretically inspired perspective. In: International Conference on Learning Representations (2021)
Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. In: International Conference on Learning Representations (2022)
Dong, X., Yang, Y.: NAS-BENCH-201: extending the scope of reproducible neural architecture search. In: 8th International Conference on Learning Representations, (ICLR), Addis Ababa, Ethiopia, 26–30 April 2020 (2020)
Fang, W., Chen, Y., Ding, J., Chen, D., Yu, Z., et al.: SpikingJelly (2020). https://github.com/fangwei123456/spikingjelly. Accessed 20 June 2023
Fang, W., Yu, Z., et al.: Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2661–2671 (2021)
Gerstner, W., Kistler, et al.: Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press (2014)
Guo, Y., Peng, W., Chen, Y., et al.: Joint A-SNN: joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recogn. 142, 109639 (2023)
He, K., Zhang, X., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Kim, Y., Li, Y., Park, H., et al.: Neural architecture search for spiking neural networks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XXIV, pp. 36–56. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20053-3_3
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report 0, University of Toronto, Toronto, Ontario (2009)
Kundu, S., Datta, G., Pedram, M., Beerel, P.A.: Spike-thrift: towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3953–3962 (2021)
Li, H., Liu, H., Ji, X., et al.: CIFAR10-DVS: an event-stream dataset for object classification. Front. Neurosci. 11, 309 (2017)
Li, W., Wen, S., Shi, K., Yang, Y., Huang, T.: Neural architecture search with a lightweight transformer for text-to-image synthesis. IEEE Trans. Netw. Sci. Eng. 9(3), 1567–1576 (2022)
Li, X., Epitropakis, M.G., Deb, K., et al.: Seeking multiple solutions: an updated survey on niching methods and their applications. IEEE Trans. Evol. Comput. 21(4), 518–538 (2016)
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: International Conference on Learning Representations (2019)
Lopes, V., Alirezazadeh, S., Alexandre, L.A.: EPE-NAS: efficient performance estimation without training for neural architecture search. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds.) ICANN 2021. LNCS, vol. 12895, pp. 552–563. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86383-8_44
Lu, S., Sengupta, A.: Exploring the connection between binary and spiking neural networks. Front. Neurosci. 14, 535 (2020)
Maass, W.: Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10(9), 1659–1671 (1997)
Mellor, J., Turner, J., et al.: Neural architecture search without training. In: International Conference on Machine Learning, pp. 7588–7598. PMLR (2021)
Meng, Q., Xiao, M., Yan, S., et al.: Training high-performance low-latency spiking neural networks by differentiation on spike representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12444–12453 (2022)
Na, B., Mok, J., Park, S., et al.: AutoSNN: towards energy-efficient spiking neural networks. In: International Conference on Machine Learning, pp. 16253–16269. PMLR (2022)
Rathi, N., Roy, K.: DIET-SNN: a low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Trans. Neural Netw. Learn. Syst. (2021)
Rathi, N., Srinivasan, G., Panda, P., Roy, K.: Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In: International Conference on Learning Representations (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, (ICLR), San Diego, CA, USA, 7–9 May 2015 (2015)
Sun, Z., Lin, M., Sun, et al.: MAE-DET: revisiting maximum entropy principle in zero-shot NAS for efficient object detection. In: International Conference on Machine Learning, pp. 20810–20826. PMLR (2022)
Taherkhani, A., Belatreche, A., Li, Y., et al.: A review of learning in biologically plausible spiking neural networks. Neural Netw. 122, 253–272 (2020)
Wang, Y., Zhang, M., Chen, Y., Qu, H.: Signed neuron with memory: towards simple, accurate and high-efficient ANN-SNN conversion. In: International Joint Conference on Artificial Intelligence (2022)
Wu, J., Chua, Y., Zhang, M., et al.: A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Trans. Neural Netw. Learn. Syst. (2021)
Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1311–1318 (2019)
Wu, Z., Zhang, H., et al.: LIAF-Net: leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Trans. Neural Netw. Learn. Syst. 33(11), 6249–6262 (2021)
Xu, J., Zhao, L., Lin, J., et al.: KNAS: green neural architecture search. In: International Conference on Machine Learning, pp. 11613–11625. PMLR (2021)
Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-Bench-101: towards reproducible neural architecture search. In: International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)
Zheng, H., Wu, Y., Deng, L., et al.: Going deeper with directly-trained larger spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11062–11070 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xie, Z., Liu, Z., Chen, P., Zhang, J. (2024). Efficient Spiking Neural Architecture Search with Mixed Neuron Models and Variable Thresholds. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14448. Springer, Singapore. https://doi.org/10.1007/978-981-99-8082-6_36
Download citation
DOI: https://doi.org/10.1007/978-981-99-8082-6_36
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8081-9
Online ISBN: 978-981-99-8082-6
eBook Packages: Computer ScienceComputer Science (R0)