research-article

Searching toward pareto-optimal device-aware neural architectures

Authors:

An-Chieh Cheng,

Shu-Huan Chang,

Shih-Chieh Chang,

Da-Cheng JuanAuthors Info & Claims

ICCAD '18: Proceedings of the International Conference on Computer-Aided Design

November 2018

Article No.: 136, Pages 1 - 7

https://doi.org/10.1145/3240765.3243494

Published: 05 November 2018 Publication History

Abstract

Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performance in many tasks such as image classification and language understanding. However, most existing works only optimize for model accuracy and largely ignore other important factors imposed by the underlying hardware and devices, such as latency and energy, when making inference. In this paper, we first introduce the problem of NAS and provide a survey on recent works. Then we deep dive into two recent advancements on extending NAS into multiple-objective frameworks: MONAS [19] and DPP-Net [10]. Both MONAS and DPP-Net are capable of optimizing accuracy and other objectives imposed by devices, searching for neural architectures that can be best deployed on a wide spectrum of devices: from embedded systems and mobile devices to workstations. Experimental results are poised to show that architectures found by MONAS and DPP-Net achieves Pareto optimality w.r.t the given objectives for various devices.

References

[1]

Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing neural network architectures using reinforcement learning. ICLR'17 (2016).

[2]

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2018. Accelerating neural architecture search using performance prediction. (2018).

[3]

Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and Simplifying One-Shot Architecture Search. In International Conference on Machine Learning. 549--558.

[4]

Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. 2017. SMASH: one-shot model architecture search through hypernetworks. ICLR'18 (2017).

[5]

Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. 2017. Efficient Architecture Search by Network Transformation. AAAI'18 (2017).

[6]

Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. 2018. Path-Level Network Transformation for Efficient Architecture Search. arXiv preprint arXiv:1806.02639 (2018).

[7]

Yukang Chen, Qian Zhang, Chang Huang, Lisen Mu, Gaofeng Meng, and Xinggang Wang. 2018. Reinforced Evolutionary Neural Architecture Search. arXiv preprint arXiv:1808.00193 (2018).

[8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 248--255.

[9]

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In IJCAI, Vol. 15. 3460--8.

Digital Library

[10]

Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. 2018. DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures. arXiv preprint arXiv:1806.08198 (2018).

[11]

Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. 2018. PPP-Net: Platform-aware Progressive Search for Pareto-optimal Neural Architectures. (2018).

[12]

Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. 2017. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528 (2017).

[13]

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2018. Multi-objective architecture search for cnns. arXiv preprint arXiv:1804.09081 (2018).

[14]

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2018. Neural Architecture Search: A Survey. arXiv preprint arXiv:1808.05377 (2018).

[15]

David E Goldberg and Kalyanmoy Deb. 1991. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of genetic algorithms. Vol. 1. Elsevier, 69--93.

[16]

Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 (2014).

[17]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[18]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).

[19]

Chi-Hung Hsu, Shu-Huan Chang, Da-Cheng Juan, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Shih-Chieh Chang. 2018. MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning. arXiv preprint arXiv:1806.10332 (2018).

[20]

Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q Weinberger. 2017. CondenseNet: An Efficient DenseNet using Learned Group Convolutions. arXiv preprint arXiv:1711.09224 (2017).

[21]

Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. 2017. Densely connected convolutional networks. CVPR'17 (2017).

[22]

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. International Conference on Learning and Intelligent Optimization (2011).

Digital Library

[23]

Haifeng Jin, Qingquan Song, and Xia Hu. 2018. Efficient Neural Architecture Search with Network Morphism. arXiv preprint arXiv:1806.10282 (2018).

[24]

Ye-Hoon Kim, Bhargava Reddy, Sojung Yun, and Chanwon Seo. 2017. NEMO: Neuro-Evolution with Multiobjective Optimization of Deep Neural Network for Speed and Accuracy. ICML'17 AutoML Workshop (2017).

[25]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.

Digital Library

[26]

Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2017. Progressive neural architecture search. arXiv preprint arXiv:1712.00559 (2017).

[27]

Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2017. Hierarchical representations for efficient architecture search. ICLR'18 (2017).

[28]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. DARTS: Differentiable Architecture Search. arXiv preprint arXiv:1806.09055 (2018).

[29]

Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. 2016. Towards automatically-tuned neural networks. In Workshop on Automatic Machine Learning. 58--65.

[30]

Renato Negrinho and Geoff Gordon. 2017. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792 (2017).

[31]

Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient Neural Architecture Search via Parameter Sharing. arXiv preprint arXiv:1802.03268 (2018).

[32]

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2018. Regularized Evolution for Image Classifier Architecture Search. arXiv preprint arXiv:1802.01548 (2018).

[33]

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc Le, and Alex Kurakin. 2017. Large-scale evolution of image classifiers. ICML'17 (2017).

[34]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation. arXiv preprint arXiv:1801.04381 (2018).

[35]

Shreyas Saxena and Jakob Verbeek. 2016. Convolutional neural fabrics. In Advances in Neural Information Processing Systems. 4053--4061.

Digital Library

[36]

Sean C Smithson, Guang Yang, Warren J Gross, and Brett H Meyer. 2016. Neural networks designing neural networks: multi-objective hyper-parameter optimization. In Proceedings of the 35th International Conference on Computer-Aided Design. ACM, 104.

Digital Library

[37]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104--3112.

Digital Library

[38]

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. 2018. MnasNet: Platform-Aware Neural Architecture Search for Mobile. arXiv preprint arXiv:1807.11626 (2018).

[39]

Lingxi Xie and Alan Yuille. 2017. Genetic cnn. ICCV'17 (2017).

[40]

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2017. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083 (2017).

[41]

Zhao Zhong, Junjie Yan, and Cheng-Lin Liu. 2017. Practical Network Blocks Design with Q-Learning. AAAI'18 (2017).

[42]

Yanqi Zhou, Siavash Ebrahimi, Sercan Ö Arık, Haonan Yu, Hairong Liu, and Greg Diamos. 2018. Resource-Efficient Neural Architect. arXiv preprint arXiv:1806.07912 (2018).

[43]

Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. ICLR'17 (2016).

[44]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2017. Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012 (2017).

Cited By

Lyu BYang YCao YWang PZhu JChang JWen S(2024)Efficient Multi-objective Neural Architecture Search Framework via Policy Gradient AlgorithmInformation Sciences10.1016/j.ins.2024.120186(120186)Online publication date: Jan-2024
https://doi.org/10.1016/j.ins.2024.120186
Liu YSun YXue BZhang MYen GTan K(2023)A Survey on Evolutionary Neural Architecture SearchIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.310055434:2(550-570)Online publication date: Feb-2023
https://doi.org/10.1109/TNNLS.2021.3100554
Guo YChen YZheng YChen QZhao PHuang JChen JTan M(2023)Pareto-aware Neural Architecture Generation for Diverse Computational Budgets2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00219(2248-2258)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00219
Show More Cited By

Recommendations

DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures
Computer Vision – ECCV 2018
Abstract
Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performances in applications such as image classification and language modeling. However, these techniques typically ignore device-related objectives such as ...
Read More
Methodology to select solutions from the pareto-optimal set: a comparative study
GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation

The resolution of a Multi-Objective Optimization Problem (MOOP) does not end when the Pareto-optimal set is found. In real problems, a single solution must be selected. Ideally, this solution must belong to the non-dominated solutions set and must take ...
Read More
Searching for robust pareto-optimal solutions in multi-objective optimization
EMO'05: Proceedings of the Third international conference on Evolutionary Multi-Criterion Optimization

In optimization studies including multi-objective optimization, the main focus is usually placed in finding the global optimum or global Pareto-optimal frontier, representing the best possible objective values. However, in practice, users may not always ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICCAD '18: Proceedings of the International Conference on Computer-Aided Design

November 2018

1020 pages

ISBN:9781450359504

DOI:10.1145/3240765

General Chair:
Iris Bahar
Brown Univ. School of Engineering, Providence, RI

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

IEEE-EDS: Electronic Devices Society
IEEE CAS
IEEE CEDA

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICCAD '18

Sponsor:

IEEE-EDS

ICCAD '18: IEEE/ACM INTERNATIONAL CONFERENCE ON COMPUTER-AIDED DESIGN

November 5 - 8, 2018

California, San Diego

Acceptance Rates

Overall Acceptance Rate 457 of 1,762 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
313
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Other Metrics

View Author Metrics

Citations

Cited By

Lyu BYang YCao YWang PZhu JChang JWen S(2024)Efficient Multi-objective Neural Architecture Search Framework via Policy Gradient AlgorithmInformation Sciences10.1016/j.ins.2024.120186(120186)Online publication date: Jan-2024
https://doi.org/10.1016/j.ins.2024.120186
Liu YSun YXue BZhang MYen GTan K(2023)A Survey on Evolutionary Neural Architecture SearchIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.310055434:2(550-570)Online publication date: Feb-2023
https://doi.org/10.1109/TNNLS.2021.3100554
Guo YChen YZheng YChen QZhao PHuang JChen JTan M(2023)Pareto-aware Neural Architecture Generation for Diverse Computational Budgets2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00219(2248-2258)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00219
Lyu BLu LHamdi MWen SYang YLi K(2023)MTLP-JRComputers and Electrical Engineering10.1016/j.compeleceng.2022.108474105:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.compeleceng.2022.108474
Minakova SSapra DStefanov TPimentel A(2022)Scenario Based Run-Time Switching for Adaptive CNN-Based Applications at the EdgeACM Transactions on Embedded Computing Systems10.1145/348871821:2(1-33)Online publication date: 8-Feb-2022
https://dl.acm.org/doi/10.1145/3488718
Kim SKwon HKwon EChoi YOh TKang S(2021)MDARTS: Multi-objective Differentiable Neural Architecture Search2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE51398.2021.9474068(1344-1349)Online publication date: 1-Feb-2021
https://doi.org/10.23919/DATE51398.2021.9474068
Xie LChen XBi KWei LXu YWang LChen ZXiao AChang JZhang XTian Q(2021)Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization GapACM Computing Surveys10.1145/347333054:9(1-37)Online publication date: 8-Oct-2021
https://dl.acm.org/doi/10.1145/3473330
Dhar SGuo JLiu JTripathi SKurup UShah M(2021)A Survey of On-Device Machine LearningACM Transactions on Internet of Things10.1145/34504942:3(1-49)Online publication date: 8-Jul-2021
https://dl.acm.org/doi/10.1145/3450494
Rizzo RPeluso VCalimera A(2021)TVFS: Topology Voltage Frequency Scaling for Reliable Embedded ConvNetsIEEE Transactions on Circuits and Systems II: Express Briefs10.1109/TCSII.2020.301753868:2(672-676)Online publication date: Feb-2021
https://doi.org/10.1109/TCSII.2020.3017538
Wang DLi MGong CChandra V(2021)AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.00635(6414-6423)Online publication date: Jun-2021
https://doi.org/10.1109/CVPR46437.2021.00635
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents