Efficient and Flexible Method for Reducing Moderate-Size Deep Neural Networks with Condensation
Abstract
:1. Introduction
2. Materials and Methods
2.1. Condensation Reduction in FNN
2.1.1. FNN
2.1.2. Cosine Similarity
2.1.3. Condensation of FNN
2.1.4. Condensation Reduction
2.2. Condensation Reduction in CNN
2.2.1. CNN
2.2.2. Condensation of CNN
2.2.3. Condensation Reduction
2.3. Manual Condensation Reduction and Automatic Condensation Reduction
2.3.1. Manual Condensation Reduction
2.3.2. Automatic Condensation Reduction
3. Results
3.1. Acceleration for Combustion Simulation
3.1.1. Background on Task
3.1.2. Training Setup
3.1.3. Reduction Setup and Result
3.2. Classification of CIFAR10
3.2.1. Background on Task
3.2.2. Training and Reduction Setup
3.2.3. Reduction Result
4. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
FNN | Fully connected neural network. |
CNN | Convolutional neural network. |
CFD | Computational fluid dynamics. |
Appendix A
Appendix A.1. Detail of Condensation in CNN
Appendix A.2. Details of Condensation Reduction in Traditional CNN
Appendix A.3. Details of Condensation Reduction in MobileNetV2
References
- Reiser, P.; Neubert, M.; Eberhard, A.; Torresi, L.; Zhou, C.; Shao, C.; Metni, H.; van Hoesel, C.; Schopmans, H.; Sommer, T.; et al. Graph neural networks for materials science and chemistry. Commun. Mater. 2022, 3, 93. [Google Scholar] [CrossRef]
- Sarvamangala, D.; Kulkarni, R.V. Convolutional neural networks in medical image understanding: A survey. Evol. Intell. 2022, 15, 1–22. [Google Scholar] [CrossRef] [PubMed]
- Shlomi, J.; Battaglia, P.; Vlimant, J.R. Graph neural networks in particle physics. Mach. Learn. Sci. Technol. 2020, 2, 021001. [Google Scholar] [CrossRef]
- Smith, M.J.; Geach, J.E. Astronomia ex machina: A history, primer and outlook on neural networks in astronomy. R. Soc. Open Sci. 2023, 10, 221454. [Google Scholar] [CrossRef] [PubMed]
- Zhang, X.M.; Liang, L.; Liu, L.; Tang, M.J. Graph neural networks and their current applications in bioinformatics. Front. Genet. 2021, 12, 690049. [Google Scholar] [CrossRef]
- Zhong, S.; Zhang, K.; Bagheri, M.; Burken, J.G.; Gu, A.; Li, B.; Ma, X.; Marrone, B.L.; Ren, Z.J.; Schrier, J.; et al. Machine learning: New ideas and tools in environmental science and engineering. Environ. Sci. Technol. 2021, 55, 12741–12754. [Google Scholar] [CrossRef] [PubMed]
- Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 2019, 378, 686–707. [Google Scholar] [CrossRef]
- Blechschmidt, J.; Ernst, O.G. Three ways to solve partial differential equations with neural networks—A review. GAMM-Mitteilungen 2021, 44, e202100006. [Google Scholar] [CrossRef]
- Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhattacharya, K.; Stuart, A.; Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv 2020, arXiv:2003.03485. [Google Scholar]
- Michoski, C.; Milosavljević, M.; Oliver, T.; Hatch, D.R. Solving differential equations using deep neural networks. Neurocomputing 2020, 399, 193–212. [Google Scholar] [CrossRef]
- Xu, Z.Q.J.; Yao, J.; Yi, Y.; Hang, L.; Zhang, Y.; Zhang, T. Solving multiscale dynamical systems by deep learning. arXiv 2024, arXiv:2401.01220. [Google Scholar]
- Liu, Z.; Sun, M.; Zhou, T.; Huang, G.; Darrell, T. Rethinking the value of network pruning. arXiv 2018, arXiv:1810.05270. [Google Scholar]
- Wang, Z.; Li, C.; Wang, X. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 14913–14922. [Google Scholar]
- Wesseling, P. Introduction to Multigrid Methods; Technical Report; NASA: Washington, DC, USA, 1995. [Google Scholar]
- Erichson, N.B.; Mathelin, L.; Yao, Z.; Brunton, S.L.; Mahoney, M.W.; Kutz, J.N. Shallow neural networks for fluid flow reconstruction with limited sensors. Proc. R. Soc. A 2020, 476, 20200097. [Google Scholar] [CrossRef] [PubMed]
- Roth, W.; Schindler, G.; Klein, B.; Peharz, R.; Tschiatschek, S.; Fröning, H.; Pernkopf, F.; Ghahramani, Z. Resource-efficient neural networks for embedded systems. J. Mach. Learn. Res. 2024, 25, 1–51. [Google Scholar]
- Luo, T.; Xu, Z.Q.J.; Ma, Z.; Zhang, Y. Phase diagram for two-layer relu neural networks at infinite-width limit. J. Mach. Learn. Res. 2021, 22, 1–47. [Google Scholar]
- Zhang, Y.; Zhang, Z.; Luo, T.; Xu, Z.J. Embedding principle of loss landscape of deep neural networks. Adv. Neural Inf. Process. Syst. 2021, 34, 14848–14859. [Google Scholar]
- Frankle, J.; Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv 2018, arXiv:1803.03635. [Google Scholar]
- Liu, D.; Zheng, M.; Sepulveda, N.A. Using Artificial Neural Network Condensation to Facilitate Adaptation of Machine Learning in Medical Settings by Reducing Computational Burden: Model Design and Evaluation Study. JMIR Form. Res. 2021, 5, e20767. [Google Scholar] [CrossRef] [PubMed]
- Cheng, H.; Zhang, M.; Shi, J.Q. A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. arXiv 2023, arXiv:2308.06767. [Google Scholar]
- Hanson, S.; Pratt, L. Comparing biases for minimal network construction with back-propagation. Adv. Neural Inf. Process. Syst. 1988, 1. [Google Scholar]
- LeCun, Y.; Denker, J.; Solla, S. Optimal brain damage. Adv. Neural Inf. Process. Syst. 1989, 2. Available online: https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf (accessed on 27 April 2024).
- You, Z.; Yan, K.; Ye, J.; Ma, M.; Wang, P. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2019, 32. Available online: https://proceedings.neurips.cc/paper_files/paper/2019/file/b51a15f382ac914391a58850ab343b00-Paper.pdf (accessed on 27 April 2024).
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar]
- Zirwes, T.; Zhang, F.; Habisreuther, P.; Hansinger, M.; Bockhorn, H.; Pfitzner, M.; Trimis, D. Quasi-DNS dataset of a piloted flame with inhomogeneous inlet conditions. Flow Turbul. Combust. 2020, 104, 997–1027. [Google Scholar] [CrossRef]
Model | Structure | Validation Loss | Parameter | Reduction Ratio |
---|---|---|---|---|
Original | 23-3200-1600-800-400-23 | 0.02276 | 6,802,800 | 100% |
1st reduced | 23-2205-1600-800-400-23 | 0.01694 | 5,187,915 | 76.26% |
2nd reduced | 23-1105-1600-800-400-23 | 0.01454 | 3,402,615 | 50.01% |
3rd reduced | 23-1105-1309-800-400-23 | 0.01434 | 2,848,260 | 41.87% |
Layer Index | 2 | 3 | 4 | 5–6 | 7 | 8–10 | 11 | 12–13 | 14 | 15–16 | 17 |
---|---|---|---|---|---|---|---|---|---|---|---|
, ReLU6 | 96 | 144 | 144 | 192 | 192 | 384 | 384 | 576 | 576 | 960 | 960 |
, ReLU6 | 96 | 144 | 144 | 192 | 192 | 384 | 384 | 576 | 576 | 960 | 960 |
, linear | 24 | 24 | 32 | 32 | 64 | 64 | 96 | 96 | 160 | 160 | 320 |
Model | Parameter | Reduction Ratio | Accuracy |
---|---|---|---|
Original | 2,236,682 | 100% | 88.16% |
1st reduced | 1,143,421 | 51.12% | 88.01% |
2nd reduced | 910,370 | 40.70% | 88.00% |
3rd reduced | 827,032 | 36.98% | 87.24% |
4th reduced | 776,011 | 34.69% | 86.36% |
5th reduced | 735,527 | 32.88% | 85.88% |
6th reduced | 704,004 | 31.48% | 85.80% |
7th reduced | 669,228 | 29.92% | 83.88% |
Final reduced | 258,212 | 11.54% | 83.21% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, T.; Xu, Z.-Q.J. Efficient and Flexible Method for Reducing Moderate-Size Deep Neural Networks with Condensation. Entropy 2024, 26, 567. https://doi.org/10.3390/e26070567
Chen T, Xu Z-QJ. Efficient and Flexible Method for Reducing Moderate-Size Deep Neural Networks with Condensation. Entropy. 2024; 26(7):567. https://doi.org/10.3390/e26070567
Chicago/Turabian StyleChen, Tianyi, and Zhi-Qin John Xu. 2024. "Efficient and Flexible Method for Reducing Moderate-Size Deep Neural Networks with Condensation" Entropy 26, no. 7: 567. https://doi.org/10.3390/e26070567