Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1007/978-3-030-46147-8_29guideproceedingsArticle/Chapter ViewAbstractPublication PagesConference Proceedingsacm-pubtype
Article

Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours

Published: 16 September 2019 Publication History

Abstract

Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 h. Our contributions are as follows: 1. Single-path search space: Compared to previous differentiable NAS methods, Single-Path NAS uses one single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional kernel parameters, hence drastically decreasing the number of trainable parameters and the search cost down to few epochs. 2. Hardware-efficient ImageNet classification: Single-Path NAS achieves top-1 accuracy on ImageNet with 79 ms latency on a Pixel 1 phone, which is state-of-the-art accuracy compared to NAS methods with similar inference latency constraints (80 ms). 3. NAS efficiency: Single-Path NAS search cost is only 8 epochs (30 TPU-hours), which is up to 5,000faster compared to prior work. 4. Reproducibility: Unlike all recent mobile-efficient NAS methods which only release pretrained models, we open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas.

References

[1]
Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine Learning, pp. 549–558 (2018)
[2]
Cai, E., Juan, D.C., Stamoulis, D., Marculescu, D.: Neuralpower: predict and deploy energy-efficient convolutional neural networks. In: Asian Conference on Machine Learning, pp. 622–637 (2017)
[3]
Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. In: International Conference on Learning Representations (2019)
[4]
Chin, T.W., Zhang, C., Marculescu, D.: Layer-compensated pruning for resource-constrained convolutional neural networks. arXiv preprint arXiv:1810.00518 (2018)
[5]
Dai, X., et al.: Chamnet: towards efficient network design through platform-aware model adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11398–11407 (2019)
[6]
Ding, R., Liu, Z., Chin, T.W., Marculescu, D., Blanton, R.: FLightNNs: lightweight quantized deep neural networks for fast and accurate inference. In: 2019 Design Automation Conference (DAC) (2019)
[7]
Ding, R., Liu, Z., Shi, R., Marculescu, D., Blanton, R.: LightNN: filling the gap between conventional deep neural networks and binarized networks. In: Proceedings of the on Great Lakes Symposium on VLSI 2017, pp. 35–40. ACM (2017)
[8]
Dong J-D, Cheng A-C, Juan D-C, Wei W, and Sun M Ferrari V, Hebert M, Sminchisescu C, and Weiss Y DPP-Net: device-aware progressive search for pareto-optimal neural architectures Computer Vision – ECCV 2018 2018 Cham Springer 540-555
[9]
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
[10]
Hsu, C.H., et al.: MONAS: multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018)
[11]
Jouppi, N.P., et al.: In-datacenter performance analysis of a tensor processing unit. In: 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pp. 1–12. IEEE (2017)
[12]
Liu, H., Simonyan, K., Yang, Y.: Darts: differentiable architecture search. In: International Conference on Learning Representations (2018)
[13]
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameter sharing. In: International Conference on Machine Learning, pp. 4092–4101 (2018)
[14]
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548 (2018)
[15]
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
[16]
Stamoulis, D., Cai, E., Juan, D.C., Marculescu, D.: Hyperpower: power-and memory-constrained hyper-parameter optimization for neural networks. In: 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE (2018)
[17]
Stamoulis, D., et al.: Designing adaptive neural networks for energy-constrained image classification. In: Proceedings of the International Conference on Computer-Aided Design. ACM (2018)
[18]
Tan, M., et al.: MnasNet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
[19]
Wu, B., et al.: FBNet: hardware-aware efficient convnet design via differentiable neural architecture search. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
[20]
Xie, S., Zheng, H., Liu, C., Lin, L.: SNAS: stochastic neural architecture search. In: International Conference on Learning Representations (2019)
[21]
Yu, J., Yang, L., Xu, N., Yang, J., Huang, T.: Slimmable neural networks. In: International Conference on Learning Representations (2019)
[22]
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
[23]
Zhou, Y., Ebrahimi, S., Arık, S.Ö., Yu, H., Liu, H., Diamos, G.: Resource-efficient neural architect. arXiv preprint arXiv:1806.07912 (2018)
[24]
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: International Conference on Machine Learning (2017)
[25]
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

Cited By

View all
  • (2024)FLEXTRONProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692276(5298-5311)Online publication date: 21-Jul-2024
  • (2024)Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future EnvisionACM Transactions on Embedded Computing Systems10.1145/370172824:1(1-100)Online publication date: 24-Oct-2024
  • (2022)Neural Architecture Search Survey: A Hardware PerspectiveACM Computing Surveys10.1145/352450055:4(1-36)Online publication date: 21-Nov-2022
  • Show More Cited By

Index Terms

  1. Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image Guide Proceedings
        Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II
        Sep 2019
        747 pages
        ISBN:978-3-030-46146-1
        DOI:10.1007/978-3-030-46147-8

        Publisher

        Springer-Verlag

        Berlin, Heidelberg

        Publication History

        Published: 16 September 2019

        Author Tags

        1. Neural Architecture Search
        2. Hardware-aware ConvNets

        Qualifiers

        • Article

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)0
        • Downloads (Last 6 weeks)0
        Reflects downloads up to 29 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)FLEXTRONProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692276(5298-5311)Online publication date: 21-Jul-2024
        • (2024)Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future EnvisionACM Transactions on Embedded Computing Systems10.1145/370172824:1(1-100)Online publication date: 24-Oct-2024
        • (2022)Neural Architecture Search Survey: A Hardware PerspectiveACM Computing Surveys10.1145/352450055:4(1-36)Online publication date: 21-Nov-2022
        • (2022)NASPipe: high performance and reproducible pipeline parallel supernet training via causal synchronous parallelismProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3503222.3507735(374-387)Online publication date: 28-Feb-2022
        • (2022)RT-RCG: Neural Network and Accelerator Search Towards Effective and Real-time ECG Reconstruction from Intracardiac ElectrogramsACM Journal on Emerging Technologies in Computing Systems10.1145/346537218:2(1-25)Online publication date: 16-Mar-2022
        • (2022)U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture SearchComputer Vision – ECCV 202210.1007/978-3-031-19775-8_11(173-190)Online publication date: 23-Oct-2022
        • (2021)AD-DARTS: Adaptive Dropout for Differentiable Architecture SearchArtificial Intelligence10.1007/978-3-030-93049-3_10(115-126)Online publication date: 5-Jun-2021
        • (2020)Searching Efficient 3D Architectures with Sparse Point-Voxel ConvolutionComputer Vision – ECCV 202010.1007/978-3-030-58604-1_41(685-702)Online publication date: 23-Aug-2020

        View Options

        View options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media