research-article

Open access

Winograd convolution: a perspective from fault tolerance

Authors:

Ying WangAuthors Info & Claims

DAC '22: Proceedings of the 59th ACM/IEEE Design Automation Conference

Pages 853 - 858

https://doi.org/10.1145/3489517.3530531

Published: 23 August 2022 Publication History

Abstract

Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation. Other than the computing efficiency, we observe its great potential in improving NN fault tolerance and evaluate its fault tolerance comprehensively for the first time. Then, we explore the use of fault tolerance of winograd convolution for either fault-tolerant or energy-efficient NN processing. According to our experiments, winograd convolution can be utilized to reduce fault-tolerant design overhead by 27.49% or energy consumption by 7.19% without any accuracy loss compared to that without being aware of the fault tolerance.

References

[1]

Ozen Elbruz Alex. 2019. Sanity-Check: Boosting the Reliability of Safety-Critical Deep Neural Network Applications. In 2019 IEEE 28th Asian Test Symposium (ATS). 7--75.

[2]

Mark T Bohr and Ian A Young. 2017. CMOS scaling trends and beyond. IEEE Micro 37, 6 (2017), 20--29.

[3]

Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. 2020. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proc. IEEE 108, 4 (2020), 485--532.

[4]

Prasenjit Dey, Kaustuv Nag, Tandra Pal, and Nikhil R Pal. 2017. Regularizing multilayer perceptron for robustness. IEEE Transactions on Systems, Man, and Cybernetics: Systems 48, 8 (2017), 1255--1266.

[5]

Sanghamitra Dutta, Ziqian Bai, Tze Meng Low, and Pulkit Grover. 2019. CodeNet: Training large scale neural networks in presence of soft-errors. arXiv preprint arXiv:1903.01042 (2019).

[6]

Chen Zitao et al. 2020. TensorFI: A flexible fault injection framework for TensorFlow applications. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 426--435.

[7]

Deng Jiacnao et al. 2015. Retraining-based timing error mitigation for hardware neural networks. In 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 593--596.

[8]

Du Yuxuan et al. 2020. An Energy-Efficient Time-Domain Binary Neural Network Accelerator with Error-Detection in 28nm CMOS. In 2020 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS). IEEE, 70--73.

[9]

Fernandez Marques et al. 2020. Searching for winograd-aware quantized networks. arXiv preprint arXiv:2002.10711 (2020).

[10]

He Yi et al. 2020. Fidelity: Efficient resilience analysis framework for deep learning accelerators. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 270--281.

[11]

Jia Zhen et al. 2018. Optimizing N-dimensional, winograd-based convolution for manycore CPUs. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 109--123.

[12]

Lavin Andrew et al. 2016. Fast algorithms for convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4013--4021.

[13]

Li Guangli et al. 2020. Lance: efficient low-precision quantized winograd convolution for neural networks based on graphics processing units. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3842--3846.

[14]

Lu Liqiang et al. 2017. Evaluating fast algorithms for convolutional neural networks on FPGAs. In 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE, 101--108.

[15]

Lu Liqiang et al. 2018. SpWA: An efficient sparse winograd convolutional neural networks accelerator on FPGAs. In Proceedings of the 55th Annual Design Automation Conference. 1--6.

[16]

Li Wenshuo et al. 2020. FTT-NAS: Discovering Fault-Tolerant Neural Architecture. In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC). 211--216.

Digital Library

[17]

Liu Xingyu et al. 2018. Efficient sparse-winograd convolutional neural networks. arXiv preprint arXiv:1802.06367 (2018).

[18]

Mahmoud Abdulrahman et al. 2020. Pytorchfi: A runtime perturbation tool for dnns. In 2020 50th Annual LEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W). IEEE, 25--31.

[19]

Ning Xuefei et al. 2021. FTT-NAS: Discovering fault-tolerant convolutional neural architecture. ACM Transactions on Design Automation of Electronic Systems (TODAES) 26, 6 (2021), 1--24.

Digital Library

[20]

Reagen Brandon et al. 2016. Minerva: Enabling low-power, highly-accurate deep neural network accelerators. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 267--278.

[21]

Rathore Mallika et al. 2020. Error Probability Models for Voltage-Scaled Multiply-Accumulate Units. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 28, 7 (2020), 1665--1675.

[22]

Samajdar Ananda et al. 2018. Scale-sim: Systolic cnn accelerator simulator. arXiv preprint arXiv:1811.02883 (2018).

[23]

Shen Junzhong et al. 2018. Towards a uniform template-based architecture for accelerating 2D and 3D CNNs on FPGA. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 97--106.

[24]

Shen Junzhong et al. 2019. Toward an efficient deep pipelined template-based architecture for accelerating the entire 2-D and 3-D CNNs on FPGA. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 39, 7 (2019), 1442--1455.

Digital Library

[25]

Shafique Muhammad et al. 2020. Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead. IEEE Design Test 37, 2 (2020), 30--57.

[26]

Whatmough Paul et al. 2018. DNN engine: A 28-nm timing-error tolerant sparse deep neural network processor for IoT applications. IEEE Journal of Solid-State Circuits 53, 9 (2018), 2722--2731.

[27]

Xygkis Athanasios et al. 2018. Efficient winograd-based convolution kernel implementation on edge devices. In 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC). IEEE, 1--6.

[28]

Xu Dawen et al. 2021. R2F: A Remote Retraining Framework for AIoT Processors With Computing Errors. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 29, 11 (2021), 1955--1966.

[29]

Zhang Jeff et al. 2018. Thundervolt: enabling aggressive voltage underscaling and timing error resilience for energy efficient deep learning accelerators. In Proceedings of the 55th Annual Design Automation Conference. 1--6.

[30]

Ghouthi Boukli Hacene, François Leduc-Primeau, Amal Ben Soussia, Vincent Gripon, and François Gagnon. 2019. Training modern deep neural networks for memory-fault robustness. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 1--5.

[31]

Xin He, Liu Ke, Wenyan Lu, Guihai Yan, and Xuan Zhang. 2018. AxTrain: Hardware-oriented neural network training for approximate inference. In Proceedings of the International Symposium on Low Power Electronics and Design. 1--6.

Digital Library

[32]

Di Huang, Xishan Zhang, Rui Zhang, Tian Zhi, Deyuan He, Jiaming Guo, Chang Liu, Qi Guo, Zidong Du, Shaoli Liu, et al. 2020. DWM: A decomposable winograd method for convolution acceleration. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4174--4181.

[33]

Kaige Jia, Zheyu Liu, Qi Wei, Fei Qiao, Xinjun Liu, Yi Yang, Hua Fan, and Huazhong Yang. 2018. Calibrating process variation at system level with in-situ low-precision transfer learning for analog neural network processors. In Proceedings of the 55th Annual Design Automation Conference. 1--6.

Digital Library

[34]

Jack Kosaian and KV Rashmi. 2021. Arithmetic-intensity-guided fault tolerance for neural network inference on gpus. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1--15.

Digital Library

[35]

Li Li, Dawen Xu, Kouzi Xing, Cheng Liu, Ying Wang, Huawei Li, and Xiaowei Li. 2019. Squeezing the last MHz for CNN acceleration on FPGAs. In 2019 IEEE International Test Conference in Asia (ITC-Asia). IEEE, 151--156.

[36]

Shanshan Liu, Pedro Reviriego, Jing Guo, Jie Han, and Fabrizio Lombardi. 2019. Exploiting asymmetry in eDRAM errors for redundancy-free error-tolerant design. IEEE Transactions on Emerging Topics in Computing 9, 4 (2019), 2064--2075.

[37]

Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E Alsaadi. 2017. A survey of deep neural network architectures and their applications. Neurocomputing 234 (2017), 11--26.

[38]

Thibaut Marty, Tomofumi Yuki, and Steven Derrien. 2020. Safe overclocking for cnn accelerators through algorithm-level error detection. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 39, 12 (2020), 4777--4790.

[39]

Sparsh Mittal. 2020. A survey on modeling and improving reliability of DNN algorithms and accelerators. Journal of Systems Architecture 104 (2020), 101689.

Digital Library

[40]

Elbruz Ozen and Alex Orailoglu. 2021. SNR: S queezing N umerical R ange Defuses Bit Error Vulnerability Surface in Deep Neural Networks. ACM Transactions on Embedded Computing Systems (TECS) 20, 5s (2021), 1--25.

Digital Library

[41]

Pramesh Pandey, Prabal Basu, Koushik Chakraborty, and Sanghamitra Roy. 2019. GreenTPU: Improving timing error resilience of a near-threshold tensor processing unit. In 2019 56th ACM/IEEE Design Automation Conference (DAC). IEEE, 1--6.

Digital Library

[42]

Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu-Yeon Wei. 2018. Ares: A framework for quantifying the resilience of deep neural networks. In 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC). IEEE, 1--6.

Digital Library

[43]

Olivier Temam. 2012. A defect-tolerant accelerator for emerging high-performance applications. In 2012 39th Annual International Symposium on Computer Architecture (ISCA). IEEE, 356--367.

[44]

Cesar Torres-Huitzil and Bernard Girau. 2017. Fault and error tolerance in neural networks: A review. IEEE Access 5 (2017), 17322--17341.

[45]

Fengbin Tu, Weiwei Wu, Shouyi Yin, Leibo Liu, and Shaojun Wei. 2018. RANA: Towards efficient neural acceleration with refresh-optimized embedded DRAM. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 340--352.

Digital Library

[46]

Lei Zhao, Youtao Zhang, and Jun Yang. 2017. AEP: An error-bearing neural network accelerator for energy efficiency and model protection. In 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 1047--1053.

Digital Library

Cited By

Huang HLiu CXue XLiu BLi HLi X(2024)MRFI: An Open-Source Multiresolution Fault Injection Framework for Neural Network ProcessingIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2024.338440432:7(1325-1335)Online publication date: Jul-2024
https://doi.org/10.1109/TVLSI.2024.3384404
Xue XLiu CWang YYang BLuo TZhang LLi HLi X(2023)Soft Error Reliability Analysis of Vision TransformersIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.331713831:12(2126-2136)Online publication date: 5-Oct-2023
https://dl.acm.org/doi/10.1109/TVLSI.2023.3317138
Xue XLiu CLiu BHuang HWang YLuo TZhang LLi HLi X(2023)Exploring Winograd Convolution for Cost-Effective Neural Network Fault ToleranceIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330689431:11(1763-1773)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TVLSI.2023.3306894
Show More Cited By

Recommendations

Winograd Convolution for Deep Neural Networks: Efficient Point Selection
Convolutional neural networks (CNNs) have dramatically improved the accuracy of image, video, and audio processing for tasks such as object recognition, image segmentation, and interactive speech systems. CNNs require large amounts of computing resources ...
Exploring Winograd Convolution for Cost-Effective Neural Network Fault Tolerance
Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great ...
Optimizing Winograd Convolution on GPUs via Partial Kernel Fusion
Network and Parallel Computing
Abstract
Convolution operations are the essential components in modern CNNs (Convolutional Neural Networks), which are also the most time-consuming. Several fast convolution algorithms include FFT and Winograd, have been proposed to solve this problem. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DAC '22: Proceedings of the 59th ACM/IEEE Design Automation Conference

July 2022

1462 pages

ISBN:9781450391429

DOI:10.1145/3489517

General Chair:
Rob Oshana
NXP

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGDA: ACM Special Interest Group on Design Automation
IEEE CEDA

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Singapore Government
National Natural Science Foundation of China
Chinese Academy of Sciences

Conference

DAC '22

Sponsor:

SIGDA

DAC '22: 59th ACM/IEEE Design Automation Conference

July 10 - 14, 2022

California, San Francisco

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Upcoming Conference

DAC '25

Sponsor:
sigda

62nd ACM/IEEE Design Automation Conference

June 22 - 26, 2025

San Francisco , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
444
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)8

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang HLiu CXue XLiu BLi HLi X(2024)MRFI: An Open-Source Multiresolution Fault Injection Framework for Neural Network ProcessingIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2024.338440432:7(1325-1335)Online publication date: Jul-2024
https://doi.org/10.1109/TVLSI.2024.3384404
Xue XLiu CWang YYang BLuo TZhang LLi HLi X(2023)Soft Error Reliability Analysis of Vision TransformersIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.331713831:12(2126-2136)Online publication date: 5-Oct-2023
https://dl.acm.org/doi/10.1109/TVLSI.2023.3317138
Xue XLiu CLiu BHuang HWang YLuo TZhang LLi HLi X(2023)Exploring Winograd Convolution for Cost-Effective Neural Network Fault ToleranceIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330689431:11(1763-1773)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TVLSI.2023.3306894
Huang HXue XLiu CWang YLuo TCheng LLi HLi X(2023)Statistical Modeling of Soft Error Influence on Neural NetworksIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.326640542:11(4152-4163)Online publication date: 11-Apr-2023
https://dl.acm.org/doi/10.1109/TCAD.2023.3266405
Lin CHsu HHuang T(2023)Redundant Lagrange Interpolation for Fault-Tolerant Winograd Convolution2023 International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan)10.1109/ICCE-Taiwan58799.2023.10226694(97-98)Online publication date: 17-Jul-2023
https://doi.org/10.1109/ICCE-Taiwan58799.2023.10226694
Bolchini CCassano LMiele ANazzari APassarello D(2023)Analyzing the Reliability of Alternative Convolution Implementations for Deep Learning Applications2023 IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT)10.1109/DFT59622.2023.10313558(1-6)Online publication date: 3-Oct-2023
https://doi.org/10.1109/DFT59622.2023.10313558
Wen ZMao YDai J(2023)PDCNN-MRW: a parallel Winograd convolutional neural network algorithm base on MapReduceInternational Journal of Machine Learning and Cybernetics10.1007/s13042-023-02007-015:5(1949-1966)Online publication date: 7-Nov-2023
https://doi.org/10.1007/s13042-023-02007-0
Liu CGao ZLiu SNing XLi HLi X(2022)Special Session: Fault-Tolerant Deep Learning: A Hierarchical Perspective2022 IEEE 40th VLSI Test Symposium (VTS)10.1109/VTS52500.2021.9794239(1-12)Online publication date: 25-Apr-2022
https://doi.org/10.1109/VTS52500.2021.9794239
Kao YLiu CDing YHuang T(2022)An Efficient Fault-Tolerant Winograd Convolution for Convolutional Neural Networks2022 IET International Conference on Engineering Technologies and Applications (IET-ICETA)10.1109/IET-ICETA56553.2022.9971628(1-2)Online publication date: 14-Oct-2022
https://doi.org/10.1109/IET-ICETA56553.2022.9971628

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents