research-article

Efficiently Combining SVD, Pruning, Clustering and Retraining for Enhanced Neural Network Compression

Authors:

Koen Goetschalckx,

Bert Moons,

Patrick Wambacq,

Marian VerhelstAuthors Info & Claims

EMDL'18: Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning

Pages 1 - 6

https://doi.org/10.1145/3212725.3212733

Published: 15 June 2018 Publication History

Get Access

References

[1]

Bojarski, M., Yeres, P., Choromanska, A., Choromanski, K., Firner, B., Jackel, L., and Muller, U. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911 (2017).

Google Scholar

[2]

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 (2009).

Google Scholar

[3]

Dieleman, S., Schlüter, J., Raffel, C., Olson, E., Sønderby, S. K., Nouri, D., et al. Lasagne: First release., Aug. 2015.

Google Scholar

[4]

Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., and Dahlgren, N. L. Darpa timit acoustic phonetic continuous speech corpus cdrom, 1993.

Google Scholar

[5]

Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J. Eie: efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture (2016), IEEE Press, pp. 243--254.

Digital Library

Google Scholar

[6]

Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149 2 (2015).

Google Scholar

[7]

Horowitz, M. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) (Feb 2014), pp. 10--14.

Crossref

Google Scholar

[8]

Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-datacenter performance analysis of a tensor processing unit. International Symposium on Computer Architecture (ISCA) (2017).

Digital Library

Google Scholar

[9]

Krizhevsky, A., and Hinton, G. Learning multiple layers of features from tiny images. 32--35.

Google Scholar

[10]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (2012), pp. 1097--1105.

Digital Library

Google Scholar

[11]

Shin, D., Lee, J., Lee, J., and Yoo, H. J. 14.2 dnpu: An 8.1tops/w reconfigurable cnn-rnn processor for general-purpose deep neural networks. In 2017 IEEE International Solid-State Circuits Conference (ISSCC) (Feb 2017), pp. 240--241.

Crossref

Google Scholar

[12]

Sundermeyer, M., Schlüter, R., and Ney, H. Lstm neural networks for language modeling. In Interspeech (2012).

Google Scholar

[13]

Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition (2014), pp. 1701--1708.

Digital Library

Google Scholar

[14]

Taylor, G., and Ding, W. Theano-based large-scale visual recognition with multiple gpus, 2015.

Google Scholar

[15]

Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688 (May 2016).

Google Scholar

[16]

Verhelst, M., and Moons, B. Embedded deep neural network processing: Algorithmic and processor techniques bring deep learning to iot and edge devices. IEEE Solid-State Circuits Magazine 9, 4 (Fall 2017), 55--65.

Crossref

Google Scholar

[17]

Xue, J., Li, J., and Gong, Y. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech (2013), pp. 2365--2369.

Google Scholar

Cited By

View all

Capozzoli ACurcio CLiseno A(2024)A Neural Network Approach for the Solution of an Electromagnetic Inverse Source Problem with SVD-based Pruning2024 IEEE-APS Topical Conference on Antennas and Propagation in Wireless Communications (APWC)10.1109/APWC61918.2024.10701760(147-151)Online publication date: 2-Sep-2024
https://doi.org/10.1109/APWC61918.2024.10701760
Leblanc BGermain P(2024)Seeking Interpretability and Explainability in Binary Activated Neural NetworksExplainable Artificial Intelligence10.1007/978-3-031-63787-2_1(3-20)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-63787-2_1
Yeung TCheung KNg MSee SYip A(2023)Transfer Learning With Singular Value Decomposition of Multichannel Convolution MatricesNeural Computation10.1162/neco_a_0160835:10(1678-1712)Online publication date: 8-Sep-2023
https://doi.org/10.1162/neco_a_01608
Show More Cited By

Recommendations

An adaptive growing and pruning algorithm for designing recurrent neural network

The training of recurrent neural networks (RNNs) concerns the selection of their structures and the connection weights. To efficiently enhance generalization capabilities of RNNs, a recurrent self-organizing neural networks (RSONN), using an adaptive ...
An enhanced ART2 neural network for clustering analysis
e-Forensics '08: Proceedings of the 1st international conference on Forensic applications and techniques in telecommunications, information, and multimedia and workshop

The adaptive resonance theory 2 (ART2) neural network exhibits several properties which can be useful in the data mining and which are lacking in most other neural networks. But ART2 has deficiencies that the categories clustered by ART2 are very ...
Fast retraining of artificial neural networks
RSFDGrC'03: Proceedings of the 9th international conference on Rough sets, fuzzy sets, data mining, and granular computing

In this paper we propose a practical mechanism for extracting information directly from the weights of a reference artificial neural network (ANN). We use this information to train a structurally identical ANN that has some variations of the global ...

Comments

Information & Contributors

Information

Published In

EMDL'18: Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning

June 2018

51 pages

ISBN:9781450358446

DOI:10.1145/3212725

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

MobiSys '18

Sponsor:

SIGMOBILE

MobiSys '18: The 16th Annual International Conference on Mobile Systems, Applications, and Services

June 15, 2018

Munich, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
319
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)4

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Capozzoli ACurcio CLiseno A(2024)A Neural Network Approach for the Solution of an Electromagnetic Inverse Source Problem with SVD-based Pruning2024 IEEE-APS Topical Conference on Antennas and Propagation in Wireless Communications (APWC)10.1109/APWC61918.2024.10701760(147-151)Online publication date: 2-Sep-2024
https://doi.org/10.1109/APWC61918.2024.10701760
Leblanc BGermain P(2024)Seeking Interpretability and Explainability in Binary Activated Neural NetworksExplainable Artificial Intelligence10.1007/978-3-031-63787-2_1(3-20)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-63787-2_1
Yeung TCheung KNg MSee SYip A(2023)Transfer Learning With Singular Value Decomposition of Multichannel Convolution MatricesNeural Computation10.1162/neco_a_0160835:10(1678-1712)Online publication date: 8-Sep-2023
https://doi.org/10.1162/neco_a_01608
Hu YSun XTian YSong LTan K(2023)Communication Efficient Federated Learning With Heterogeneous Structured Client ModelsIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2022.32093457:3(753-767)Online publication date: Jun-2023
https://doi.org/10.1109/TETCI.2022.3209345
Yu ZBouganis C(2023)Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition2023 33rd International Conference on Field-Programmable Logic and Applications (FPL)10.1109/FPL60245.2023.00036(204-211)Online publication date: 4-Sep-2023
https://doi.org/10.1109/FPL60245.2023.00036
Vadera SAmeen S(2022)Methods for Pruning Deep Neural NetworksIEEE Access10.1109/ACCESS.2022.318265910(63280-63300)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3182659
Wan XLi HZhang LWu Y(2022)Dimensionality reduction for multivariate time-series data miningThe Journal of Supercomputing10.1007/s11227-021-04303-478:7(9862-9878)Online publication date: 19-Jan-2022
https://doi.org/10.1007/s11227-021-04303-4
Heidari MGhatee MNickabadi APourhasan Nezhad A(2022)Diverse and styled image captioning using singular value decomposition‐based mixture of recurrent expertsConcurrency and Computation: Practice and Experience10.1002/cpe.686634:22Online publication date: Feb-2022
https://doi.org/10.1002/cpe.6866
Chen YZheng BZhang ZWang QShen CZhang Q(2020)Deep Learning on Mobile and Embedded DevicesACM Computing Surveys10.1145/339820953:4(1-37)Online publication date: 20-Aug-2020
https://dl.acm.org/doi/10.1145/3398209
Mai ATran LTran LTrinh N(2020)VGG deep neural network compression via SVD and CUR decomposition techniques2020 7th NAFOSTED Conference on Information and Computer Science (NICS)10.1109/NICS51282.2020.9335842(118-123)Online publication date: 26-Nov-2020
https://doi.org/10.1109/NICS51282.2020.9335842

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

An adaptive growing and pruning algorithm for designing recurrent neural network

An enhanced ART2 neural network for clustering analysis

Fast retraining of artificial neural networks

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

References

Cited By

Recommendations

An adaptive growing and pruning algorithm for designing recurrent neural network

An enhanced ART2 neural network for clustering analysis

Fast retraining of artificial neural networks

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations