ACO-Pruning for Deep Neural Networks: A Case Study in CNNs
Authors: 
Renato Sellaro Dorighello, Myriam Regattieri Delgado, Ricardo Lüders, Daniel Fernando PigattoAuthors Info & Claims
Renato Sellaro Dorighello, Myriam Regattieri Delgado, Ricardo Lüders, Daniel Fernando PigattoAuthors Info & ClaimsPages 1895 - 1903
Abstract
Deep Neural Networks (DNNs) are successful in several tasks, mainly due to their ability to process a large volume of data, given their huge number of parameters and computational operations. Larger and deeper models have been developed to improve their performance with an increasing computational cost. Pruning algorithms are strategies necessary to mitigate the computational burden and achieve better performance by eliminating parts of the network structure while maintaining good training and testing results. Dynamic network pruning increases performance through online choices of inference paths depending on various inputs. This work proposes a new Ant Colony Optimization Pruning (ACO-P) algorithm for dynamic pruning based on swarm intelligence to compress the model without jeopardizing accuracy. We validate ACO-P with a CNN model on the MNIST dataset by comparing it with a baseline pruner that uses random choices, and a well-established dynamic pruning method based on a secondary neural network. The results show that our proposal is a computationally more efficient alternative, capable of achieving higher pruning rates.
References
[1]
Mohamed Abd Elaziz, Abdelghani Dahou, Laith Abualigah, Liyang Yu, Mohammad Alshinwan, Ahmad M Khasawneh, and Songfeng Lu. 2021. Advanced metaheuristic optimization techniques in applications of deep neural networks: a review. Neural Computing and Applications (2021), 1--21.
[2]
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. 2014. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on audio, speech, and language processing 22, 10 (2014), 1533--1545.
[3]
Rand Jawad Kadhim Almahmood and Adem Tekerek. 2022. Issues and Solutions in Deep Learning-Enabled Recommendation Systems within the E-Commerce Field. Applied Sciences 12, 21 (2022), 11256.
[4]
EO Arkhangelskaya and Sergei Igorevich Nikolenko. 2023. Deep learning for natural language processing: a survey. Journal of Mathematical Sciences (2023), 1--50.
[5]
Edvinas Byla and Wei Pang. 2019. Deepswarm: Optimising convolutional neural networks using swarm intelligence. In UK Workshop on Computational Intelligence. Springer, 119--130.
[6]
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2023. A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. arXiv preprint arXiv:2308.06767 (2023).
[7]
Shaveta Dargan, Munish Kumar, Maruthi Rohit Ayyagari, and Gulshan Kumar. 2020. A survey of deep learning and its applications: a new paradigm to machine learning. Archives of Computational Methods in Engineering 27 (2020), 1071--1092.
[8]
Tiancan Deng. 2022. A survey of convolutional neural networks for image classification: Models and datasets. In 2022 International Conference on Big Data, Information and Computer Network (BDICN). IEEE, 746--749.
[9]
Travis Desell, Sophine Clachar, James Higgins, and Brandon Wild. 2015. Evolving deep recurrent neural networks using ant colony optimization. In Evolutionary Computation in Combinatorial Optimization: 15th European Conference, EvoCOP 2015, Copenhagen, Denmark, April 8--10, 2015, Proceedings 15. Springer, 86--98.
[10]
Shi Dong, Ping Wang, and Khushnood Abbas. 2021. A survey on deep learning and its applications. Computer Science Review 40 (2021), 100379.
[11]
Marco Dorigo. 1992. Optimization, learning and natural algorithms. Ph. D. Thesis, Politecnico di Milano (1992).
[12]
Jan Egger, Christina Gsaxner, Antonio Pepe, Kelsey L Pomykala, Frederic Jonske, Manuel Kurz, Jianning Li, and Jens Kleesiek. 2022. Medical deep learning---A systematic meta-review. Computer methods and programs in biomedicine 221 (2022), 106874.
[13]
AbdElRahman A ElSaid, Alexander G Ororbia, and Travis J Desell. 2019. The ant swarm neuro-evolution procedure for optimizing recurrent networks. CoRR abs/1909.11849 (2019).
[14]
Mehdi Gheisari, Fereshteh Ebrahimzadeh, Mohamadtaghi Rahimi, Mahdieh Moazzamigodarzi, Yang Liu, Pijush Kanti Dutta Pramanik, Mohammad Ali Heravi, Abolfazl Mehbodniya, Mustafa Ghaderzadeh, Mohammad Reza Feylizadeh, et al. 2023. Deep learning: Applications, architectures, models, tools, and frameworks: A comprehensive survey. CAAI Transactions on Intelligence Technology 8, 3 (2023), 581--606.
[15]
Google. [n. d.]. Google Colab Pro. https://colab.research.google.com. Accessed on: March, 2024.
[16]
Tasneem Gorach. 2018. Deep convolutional neural networks-a review. Int Res J Eng Tech (IRJET) 5, 07 (2018).
[17]
Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2--4, 2016, Conference Track Proceedings. (2016).
[18]
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28 (2015).
[19]
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Ng. 2014. Deep speech: Scaling up end-to-end speech recognition. CoRR abs/1412.5567 (2014).
[20]
Stephen Hanson and Lorien Pratt. 1988. Comparing biases for minimal network construction with back-propagation. Advances in neural information processing systems 1 (1988), 177--185.
[21]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision. 1026--1034.
[22]
David A. Huffman. 1952. A Method for the Construction of Minimum-Redundancy Codes. Proceedings of the IRE 40, 9 (1952), 1098--1101.
[23]
Saidul Islam, Hanae Elmekki, Ahmed Elsebai, Jamal Bentahar, Nagat Drawel, Gaith Rjoub, and Witold Pedrycz. 2023. A comprehensive survey on applications of transformers for deep learning tasks. Expert Systems with Applications (2023), 122666.
[24]
Athanasios Karapantelakis, Pegah Alizadeh, Abdulrahman Alabassi, Kaushik Dey, and Alexandros Nikou. 2024. Generative ai in mobile networks: a survey. Annals of Telecommunications 79, 1 (2024), 15--33.
[25]
Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi. 2020. A survey of the recent architectures of deep convolutional neural networks. Artificial intelligence review 53, 8 (2020), 5455--5516.
[26]
Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. [n. d.]. The MNIST Database of handwritten digits. http://yann.lecun.com/exdb/mnist/. Access: 2023-07-09.
[27]
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461 (2021), 370--403.
[28]
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. 2017. Runtime neural pruning. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 2178--2188.
[29]
Min Lin, Qiang Chen, and Shuicheng Yan. 2014. Network in network. 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, 14--16 (2014).
[30]
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. AI Open (2022).
[31]
Meng Liu, Hongyang Gao, and Shuiwang Ji. 2020. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 338--348.
[32]
Qing Liu, Ningyu Zhang, Wenzhu Yang, Wang Sile, Zhenchao Cui, Xiangyang Chen, and Liping Chen. 2017. A Review of Image Recognition with Deep Convolutional Neural Network. In International Conference on Intelligent Computing. 69--80.
[33]
Wenping Ma, Xiaobo Zhou, Hao Zhu, Longwei Li, and Licheng Jiao. 2021. A two-stage hybrid ant colony optimization for high-dimensional feature selection. Pattern Recognition 116 (2021), 107933.
[34]
Amina N Muhammad, Ali M Aseere, Haruna Chiroma, Habib Shah, Abdulsalam Y Gital, and Ibrahim Abaker Targio Hashem. 2021. Deep learning application in smart cities: recent development, taxonomy, challenges and research prospects. Neural computing and applications 33 (2021), 2973--3009.
[35]
Simon Parsons. 2005. Ant Colony Optimization by Marco Dorigo and Thomas Stützle. The Knowledge Engineering Review 20, 1 (2005), 92--93.
[36]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems. 8024--8035.
[37]
Yongming Rao, Jiwen Lu, Ji Lin, and Jie Zhou. 2018. Runtime network routing for efficient image classification. IEEE transactions on pattern analysis and machine intelligence 41, 10 (2018), 2291--2304.
[38]
Khalid Salama and Ashraf M Abdelbar. 2014. A novel ant colony algorithm for building neural network topologies. In International Conference on Swarm Intelligence. Springer, 1--12.
[39]
Koosha Sharifani and Mahyar Amini. 2023. Machine Learning and Deep Learning: A Review of Methods and Applications. World Information Technology and Engineering Journal 10, 07 (2023), 3897--3904.
[40]
Xiaoxuan Shen, Baolin Yi, Zhaoli Zhang, Jiangbo Shu, and Hai Liu. 2016. Automatic recommendation technology for learning resources with convolutional neural network. In 2016 international symposium on educational technology (ISET). IEEE, 30--34.
[41]
Jiankun Sun, Xiong Luo, Honghao Gao, Weiping Wang, Yang Gao, and Xi Yang. 2020. Categorizing malware via a word2vec-based temporal convolutional network scheme. Journal of Cloud Computing 9, 1 (2020), 1--14.
[42]
Pete Warden and Daniel Situnayake. 2019. Tinyml: Machine learning with tensorflow lite on arduino and ultra-low-power microcontrollers. O'Reilly Media.
[43]
Sheng Xu, Anran Huang, Lei Chen, and Baochang Zhang. 2020. Convolutional Neural Network Pruning: A Survey. In 2020 39th Chinese Control Conference (CCC). IEEE, 7458--7463.
[44]
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2023. Diffusion models: A comprehensive survey of methods and applications. Comput. Surveys 56, 4 (2023), 1--39.
[45]
Zhaolin Yuan, Jinlong Hu, Di Wu, and Xiaojuan Ban. 2020. A dual-attention recurrent neural network method for deep cone thickener underflow concentration prediction. Sensors 20, 5 (2020), 1260.
[46]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.
Index Terms
- ACO-Pruning for Deep Neural Networks: A Case Study in CNNs
Recommendations
Structured Pruning with Automatic Pruning Rate Derivation for Image Processing Neural Networks
ISMSI '22: Proceedings of the 2022 6th International Conference on Intelligent Systems, Metaheuristics & Swarm IntelligenceStructured pruning has been proposed for network model compression. Because most of existing structured pruning methods assign pruning rate manually, finding appropriate pruning rate to suppress the degradation of pruned model accuracy is difficult. ...
Dynamic structure pruning for compressing CNNs
AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial IntelligenceStructure pruning is an effective method to compress and accelerate neural networks. While filter and channel pruning are preferable to other structure pruning methods in terms of realistic acceleration and hardware compatibility, pruning methods with a ...
G-HABC Algorithm for Training Artificial Neural Networks
Learning problems for Neural Network (NN) has widely been explored in the past two decades. Researchers have focused more on population-based algorithms because of its natural behavior processing. The population-based algorithms are Ant Colony ...
Comments
Information & Contributors
Information
Published In
July 2024
2187 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
GECCO '24 Companion
Sponsor:
GECCO '24 Companion: Genetic and Evolutionary Computation Conference Companion
July 14 - 18, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 18Total Downloads
- Downloads (Last 12 months)18
- Downloads (Last 6 weeks)5
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in