Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Searching for N:M Fine-grained Sparsity of Weights and Activations in Neural Networks

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13807))

Included in the following conference series:

  • 2317 Accesses

Abstract

Sparsity in deep neural networks has been extensively studied to compress and accelerate models for environments with limited resources. The general approach of pruning aims at enforcing sparsity on the obtained model, with minimal accuracy loss, but with a sparsity structure that enables acceleration on hardware. The sparsity can be enforced on either the weights or activations of the network, and existing works tend to focus on either one for the entire network. In this paper, we suggest a strategy based on Neural Architecture Search (NAS) to sparsify both activations and weights throughout the network, while utilizing the recent approach of N:M fine-grained structured sparsity that enables practical acceleration on dedicated GPUs. We show that a combination of weight and activation pruning is superior to each option separately. Furthermore, during the training, the choice between pruning the weights of activations can be motivated by practical inference costs (e.g., memory bandwidth). We demonstrate the efficiency of the approach on several image classification datasets.

R. Akiva-Hochman and S. E. Finder—Contributed equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    To the best of our knowledge, only sparse-dense matrix-matrix products can be efficiently applied in hardware, and the speedup of sparse-sparse products is much more complicated to achieve. Hence, we consider the pruning of either the weights or the activations. However, the pruning of both can also be considered in our framework.

References

  1. Banner, R., Hubara, I., Hoffer, E., Soudry, D.: Scalable methods for 8-bit training of neural networks. In: Advances in Neural Information Processing Systems, pp. 5145–5153 (2018)

    Google Scholar 

  2. Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradients through stochastic neurons for conditional computation. preprint arXiv:1308.3432 (2013)

  3. Cai, Z., Vasconcelos, N.: Rethinking differentiable search for mixed-precision neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2349–2358 (2020)

    Google Scholar 

  4. Deng, D., Socher, L., K-Li, F.F.L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  5. Dong, Y.: Network pruning via transformable architecture search. Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  6. Dong, X., Huang, J., Yang, Y., Yan, S.: More is less: a more complicated network with less inference complexity. In: CVPR, pp. 5840–5848 (2017)

    Google Scholar 

  7. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)

    MathSciNet  MATH  Google Scholar 

  8. Finder, S.E., Zohav, Y., Ashkenazi, M., Treister, E.: Wavelet feature maps compression for image-to-image CNNs. arXiv preprint arXiv:2205.12268 (2022)

  9. Georgiadis, G.: Accelerating convolutional neural networks via activation map compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7085–7095 (2019)

    Google Scholar 

  10. Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT Press, Cambridge (2016)

    Google Scholar 

  11. Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vision 129, 1789–1819 (2021)

    Article  Google Scholar 

  12. Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient DNNs. In: Advances in Neural Information Processing Systems (NIPS), pp. 1379–1387 (2016)

    Google Scholar 

  13. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)

    Google Scholar 

  14. He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. In: IJCAI International Joint Conference on Artificial Intelligence (2018)

    Google Scholar 

  15. He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)

    Google Scholar 

  16. Howard, A., et al.: Searching for MobileNetV3. In: IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  17. Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J., Soudry, D.: Accelerated sparse neural training: a provable and efficient method to find n: m transposable masks. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 21099–21111 (2021)

    Google Scholar 

  18. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Quantized neural networks: training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18(1), 6869–6898 (2017)

    MathSciNet  MATH  Google Scholar 

  19. Jang, G., Poole: categorical reparameterization with gumbel-softmax. In: International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  20. Krizhevsky, H.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)

    Google Scholar 

  21. Kurtz, M., et al.: Inducing and exploiting activation sparsity for fast inference on deep neural networks. In: International Conference on Machine Learning (ICML), pp. 5533–5543. PMLR (2020)

    Google Scholar 

  22. Lee, N., Ajanthan, T., Torr, P.H.: Snip: Single-shot network pruning based on connection sensitivity. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  23. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. In: International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  24. Liu, H., Simonyan, K., Yang, Y.: Darts: differentiable architecture search. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  25. Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2

    Chapter  Google Scholar 

  26. Mishra, A., et al.: Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 (2021)

  27. Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11264–11272 (2019)

    Google Scholar 

  28. Raihan, M.A., Aamodt, T.: Sparse weight activation training. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 15625–15638 (2020)

    Google Scholar 

  29. Real, E., et al.: Large-scale evolution of image classifiers. In: International Conference on Machine Learning, pp. 2902–2911. PMLR (2017)

    Google Scholar 

  30. Ren, P., et al.: A comprehensive survey of neural architecture search: challenges and solutions. ACM Computing Surveys (CSUR), pp. 1–34 (2021)

    Google Scholar 

  31. Ronen, M., Finder, S.E., Freifeld, O.: Deepdpm: Deep clustering with an unknown number of clusters. In: Conference on Computer Vision and Pattern Recognition (2022)

    Google Scholar 

  32. Sun, Q., Li, X., Jiao, L., Ren, Y., Shang, F., Liu, F.: fast and effective: a novel sequential single-path search for mixed-precision-quantized networks. IEEE Trans. Cybern. 1–13 (2022)

    Google Scholar 

  33. Sun, W., et al.: Dominosearch: Find layer-wise fine-grained n: M sparse schemes from dense neural networks. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 20721–20732 (2021)

    Google Scholar 

  34. Tan, C., Pang, V., Sandler, H., Le, V.: MnasNet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820–2828 (2019)

    Google Scholar 

  35. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114 (2019)

    Google Scholar 

  36. Tanaka, H., Kunin, D., Yamins, D.L., Ganguli, S.: Pruning neural networks without any data by iteratively conserving synaptic flow. preprint arXiv:2006.05467 (2020)

  37. Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., Van Gool, L.: SCAN: learning to classify images without labels. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12355, pp. 268–285. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58607-2_16

    Chapter  Google Scholar 

  38. Wang, C., Zhang, G., Grosse, R.: Picking winning tickets before training by preserving gradient flow. In: International Conference on Learning Representations (2020)

    Google Scholar 

  39. Wu, D., Zhang, W., Sun, W., Tian, V., Jia, K.: FBNet: hardware-aware efficient convnet design via differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734–10742 (2019)

    Google Scholar 

  40. Yin, P., Lyu, J., Zhang, S., Osher, S., Qi, Y., Xin, J.: Understanding straight-through estimator in training activation quantized neural nets. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  41. Yu, H., Han, Q., Li, J., Shi, J., Cheng, G., Fan, B.: Search what you want: barrier panelty NAS for mixed precision quantization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 1–16. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_1

    Chapter  Google Scholar 

  42. Zhou, M., Zhu, L., Zhang, Y., Sun, L.: Learning N: M fine-grained structured sparse neural networks from scratch. In: International Conference on Learning Representations (2021)

    Google Scholar 

  43. Zhu, M., Zhang, T., Gu, Z., Xie, Y.: Sparse tensor core: algorithm and hardware co-design for vector-wise sparse neural networks on modern GPUs. In: Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 359–371 (2019)

    Google Scholar 

  44. Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017)

  45. Zoph, V., Shlens, L.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Israel Innovation Authority through the Avatar consortium. The authors also thank the Israeli Council for Higher Education (CHE) via the Data Science Research Center and the Lynn and William Frankel Center for Computer Science at BGU. SF is also supported by Kreitman High-tech scholarship.

Author information

Authors and Affiliations

  1. Ben-Gurion University of the Negev, Be’er Sheva, Israel

    Ruth Akiva-Hochman, Shahaf E. Finder & Eran Treister

  2. Intel Labs, Hillsboro, OR, USA

    Javier S. Turek

Authors
  1. Ruth Akiva-Hochman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahaf E. Finder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Javier S. Turek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eran Treister
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruth Akiva-Hochman .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Akiva-Hochman, R., Finder, S.E., Turek, J.S., Treister, E. (2023). Searching for N:M Fine-grained Sparsity of Weights and Activations in Neural Networks. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13807. Springer, Cham. https://doi.org/10.1007/978-3-031-25082-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25082-8_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25081-1

  • Online ISBN: 978-3-031-25082-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation