Abstract
Although deep neural networks (DNNs) have achieved excellent performance in computer vision applications in recent years, it’s still challenging to deploy them on resource-limited devices due to their high computation costs and memory footprint. Meanwhile, training DNNs consumes huge energy, leading to excessive carbon emissions and accelerating global warming. To solve these problems, we first propose a novel filter pruning algorithm for neural network named Discrete Rank Pruning (DRP). It is convenient to deploy large scale models on resource-limited devices. Second, we propose a novel calculation method named Neural Network Carbon Emission Calculator (NNCEC) to quantify DNNs energy consumption and carbon emission. It makes the environmental cost of neural network become transparent. Moreover, many pruning methods apply sparse regularization on the filter weights of the convolution layers to reduce the degradation of performance after pruning. We analyze these methods and find that it is necessary to consider the influence of the bias term. Based on these, we propose a novel sparse method named Consideration Bias Sparsity (CBS). Extensive experiments on MNIST, CIFAR-10 and CIFAR-100 datasets with LeNet-5, VGGNet-16, ResNet-56, GoogLeNet and DenseNet-40 demonstrate the effectiveness of CBS and DRP. For LeNet-5, CBS achieves 1.87% increase in accuracy than sparse regularization method on MNIST. For VGGNet-16, DRP achieves 66.6% reduction in FLOPs by removing 83.3% parameters with only 0.36% decrease in accuracy on CIFAR-10. For ResNet-56, DRP achieves 47.89% reduction in FLOPs by removing 42.8% parameters with only 0.82% decrease in accuracy on CIFAR-100. For GoogLeNet, DRP achieves over 50% carbon emissions reduction on CIFAR-10 and CIFAR-100.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Dhar, P.: The carbon impact of artificial intelligence. Nat. Mach. Intell. 2(8), 423–425 (2020)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. Adv. Neural Inf. Process. Syst. 28 (2015)
Hassibi, B., Stork, D.: Second order derivatives for network pruning: optimal brain surgeon. Adv. Neural Inf. Process. Syst. 5 (1992)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., Pineau, J.: Towards the systematic reporting of the energy and carbon footprints of machine learning. J. Mach. Learn. Res. 21(1), 10039–10081 (2020)
Hölzle, U.: Data centers are more energy efficient than ever. Available from Google: https://www.blog.google/outreach-initiatives/sustainability/data-centers-energy-efficient (2020)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Jouppi, N.P., Yoon, D.H., Ashcraft, M., Gottscho, M., Jablin, T.B., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., et al.: Ten lessons from three generations shaped google’s tpuv4i: Industrial product. In: 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 1–14 (2021). IEEE
Kang, Z., Zhang, P., Zhang, X., Sun, J., Zheng, N.: Instance-conditional knowledge distillation for object detection. Adv. Neural. Inf. Process. Syst. 34, 16468–16480 (2021)
Kanwal, S., Mehran, M.T., Hassan, M., Anwar, M., Naqvi, S.R., Khoja, A.H.: An integrated future approach for the energy security of Pakistan: replacement of fossil fuels with syngas for better environment and socio-economic development. Renew. Sustain. Energy Rev. 156, 111978 (2022)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Leopold, G.: Aws to offer nvidia’s t4 gpus for ai inferencing. https://www.hpcwire.com/2019/03/19/aws-upgrades-its-gpu-backed-ai-inference-platform/ (2019)
Li, Y., Gu, S., Mayer, C., Gool, L.V., Timofte, R.: Group sparsity: The hinge between filter pruning and decomposition for network compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018–8027 (2020)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., Shao, L.: Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538 (2020)
Liu, B., Wang, M., Foroosh, H., Tappen, M., Pensky, M.: Sparse convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 806–814 (2015)
Liu, Z., Wang, Y., Han, K., Ma, S., Gao, W.: Instance-aware dynamic neural network quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12434–12443 (2022)
McIlvain, G., Cerjanic, A.M., Christodoulou, A.G., McGarry, M.D., Johnson, C.L.: Oscillate: a low-rank approach for accelerated magnetic resonance elastography. Magn. Reson. Med. 88(4), 1659–1672 (2022)
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., Dean, J.: Carbon Emissions and Large Neural Network Training. arXiv preprint arXiv:2104.10350 (2021)
Pei, S., Luo, J., Liang, S.: Drp: Discrete rank pruning for neural network. In: Network and Parallel Computing: 19th IFIP WG 10.3 International Conference, NPC 2022, Jinan, China, September 24–25, 2022, Proceedings, pp. 168–179 (2022). Springer
Pei, S., Wu, Y., Qiu, M.: Neural network compression and acceleration by federated pruning. In: Algorithms and Architectures for Parallel Processing: 20th International Conference, ICA3PP 2020, New York City, NY, USA, October 2–4, 2020, Proceedings, Part II, pp. 173–183 (2020). Springer
Phan, A.-H., Sobolev, K., Sozykin, K., Ermilov, D., Gusak, J., Tichavskỳ, P., Glukhov, V., Oseledets, I., Cichocki, A.: Stable low-rank tensor decomposition for compression of convolutional neural network. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pp. 522–539 (2020). Springer
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnor-net: Imagenet classification using binary convolutional neural networks. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV, pp. 525–542 (2016). Springer
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243 (2019)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
USEPA: Emissions & generation resource integrated database (egrid) (2000)
Wang, Y., Wang, Y., Cai, J., Lee, T.K., Miao, C., Wang, Z.J.: Ssd-kd: a self-supervised diverse knowledge distillation method for lightweight skin lesion classification using dermoscopic images. Med. Image Anal. 84, 102693 (2023)
Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. Adv. Neural Inf. Process. Syst. 29 (2016)
Acknowledgements
The authors would like to thank the anonymous reviewers for their invaluable comments. This work was partially funded by the National Natural Science Foundation of China under Grant no. 61975124, Shanghai Natural Science Foundation (20ZR1438500), State Key Laboratory of Computer Architecture (ICT, CAS) under Grant No.CARCHA202111, and Engineering Research Center of Software/Hardware Co-design Technology and Application, Ministry of Education, East China Normal University under Grant no. OP202202. Any opinions, findings and conclusions expressed in this paper are those of the authors and do not necessarily reflect the views of the sponsors.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Pei, S., Luo, J., Liang, S. et al. Carbon Emissions Reduction of Neural Network by Discrete Rank Pruning. CCF Trans. HPC 5, 334–346 (2023). https://doi.org/10.1007/s42514-023-00144-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42514-023-00144-0