Abstract
Neural network pruning is a widely used strategy for reducing model storage and computing requirements. It allows to lower the complexity of the network by introducing sparsity in the weights. Because taking advantage of sparse matrices is still challenging, pruning is often performed in a structured way, i.e. removing entire convolution filters in the case of ConvNets, according to a chosen pruning criteria. Common pruning criteria, such as \(l_1\)-norm or movement, usually do not consider the individual utility of filters, which may lead to: (1) the removal of filters exhibiting rare, thus important and discriminative behaviour, and (2) the retaining of filters with redundant information. In this paper, we present a technique solving those two issues, and which can be appended to any pruning criteria. This technique ensures that the criteria of selection focuses on redundant filters, while retaining the rare ones, thus maximizing the variety of remaining filters. The experimental results, carried out on different datasets (CIFAR-10, CIFAR-100 and CALTECH-101) and using different architectures (VGG-16 and ResNet-18) demonstrate that it is possible to achieve similar sparsity levels while maintaining a higher performance when appending our filter selection technique to pruning criteria. Moreover, we assess the quality of the found sparse subnetworks by applying the Lottery Ticket Hypothesis and find that the addition of our method allows to discover better performing tickets in most cases.
N. Hubens—This research has been conducted in the context of a joint-PhD between the two institutions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (ICLR) (2019)
Sanh, V., Wolf, T., Rush, A.M.: Movement pruning: adaptive sparsity by fine-tuning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
Qin, Z., Fuxun, Y., Liu, C., Chen, X.: Functionality-oriented convolutional filter pruning. In: British Machine Vision Conference (BMVC) (2019)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems (NeurIPS) (1990)
Hassibi, B., Stork, G.D., Wolff, G.: Optimal brain surgeon and general network pruning. In: International Conference on Neural Networks (ICANN) (1993)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.: Pruning filters for efficient ConvNets. In: International Conference on Learning Representations (ICLR) (2017)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: International Conference on Computer Vision (ICCV) (2017)
Hubens, N., et al.: An experimental study of the impact of pre-training on the pruning of a convolutional neural network. In: International Conference on Applications of Intelligent Systems (APPIS) (2020)
Tartaglione, E., Lepsøy, S., Fiandrotti, A., Francini, G.: Learning sparse neural networks via sensitivity-driven regularization. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)
Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neural networks. In: International Conference on Machine Learning (ICML) (2017)
Gale, T., Elsen, E., Hooker, S.: The state of sparsity in deep neural networks. In: The International Conference on Machine Learning (ICML) (2019)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural networks. In: International Conference on Neural Information Processing Systems (ICPS) (2015)
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. In: International Conference on Learning Representations (ICLR) (2017)
Zhu, M., Suyog, G.: To prune, or not to prune: exploring the efficacy of pruning for model compression. In: International Conference on Learning Representations (ICLR) (2018)
Hubens, N.: One-cycle pruning: pruning ConvNets with tight training budget. arXiv: abs/2107.02086 (2021)
Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. University of Montreal, vol. 1341, p. 3 (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2012)
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: International Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. University of Toronto (2009)
Li, F., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In: International Conference on Computer Vision and Pattern Recognition Workshop (CVPR) (2004)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: International Conference on Learning Representations (ICLR) (2019)
Smith, L., Topin, N.: Super-convergence: very fast training of neural networks using large learning rates. In: SPIE Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications (2019)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)
Howard, J., Gugger, S.: fastai: a layered API for deep learning. In: MDPI Information (2020)
Hubens, N.: FasterAI: a library to make smaller and faster neural networks (2020). https://github.com/nathanhubens/fasterai
Swee Kiat, L.: Lucent (2019). https://github.com/greentfrapp/lucent
Frankle, J., Dziugaite, K., Roy, D., Carbin, M.: Linear mode connectivity and the lottery ticket hypothesis. In: International Conference on Machine Learning (ICML) (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hubens, N., Mancas, M., Gosselin, B., Preda, M., Zaharia, T. (2022). Improve Convolutional Neural Network Pruning by Maximizing Filter Variety. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13231. Springer, Cham. https://doi.org/10.1007/978-3-031-06427-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-06427-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06426-5
Online ISBN: 978-3-031-06427-2
eBook Packages: Computer ScienceComputer Science (R0)