short-paper

Open access

Cerebro: Efficient and Reproducible Model Selection on Deep Learning Systems

Authors:

Supun Nakandala,

Yuhao Zhang,

Arun KumarAuthors Info & Claims

DEEM'19: Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning

Article No.: 6, Pages 1 - 4

https://doi.org/10.1145/3329486.3329496

Published: 30 June 2019 Publication History

PDF eReader

Abstract

Artificial Neural Networks (ANNs) are revolutionizing many machine learning (ML) applications. But there is a major bottleneck to wider adoption: the pain of model selection. This empirical process involves exploring the ANN architecture and hyper-parameters, often requiring hundreds of trials. Alas, most ML systems focus on training one model at a time, reducing throughput and raising costs; some also sacrifice reproducibility. We present our vision of Cerebro, a system to raise ANN model selection throughput at scale and ensure reproducibility. Cerebro uses a novel parallel execution strategy we call model hopper parallelism. We discuss the research questions in building Cerebro and present promising initial empirical results.

References

[1]

Max Jaderberg et al. 2017. Population based training of neural networks. arXiv preprint arXiv:1711.09846 (2017).

Google Scholar

[2]

Arun Kumar et al. 2016. Model selection management systems: The next frontier of advanced analytics. SIGMOD Record (2016).

Digital Library

Google Scholar

[3]

Lisha Li et al. 2016. Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560 (2016).

Digital Library

Google Scholar

[4]

Mu Li et al. 2014. Scaling Distributed Machine Learning with the Parameter Server. In OSDI.

Digital Library

Google Scholar

[5]

Philipp Moritz et al. 2018. Ray: A Distributed Framework for Emerging AI Applications. In OSDI.

Digital Library

Google Scholar

[6]

Szilard Pafka. Accessed February 28, 2019. Big RAM is eating big data - Size of datasets used for analytics. https://www.kdnuggets.com/2015/11/big-ram-big-data-size-datasets.html.

Google Scholar

[7]

Timos K Sellis. 1988. Multiple-query optimization. TODS (1988).

Digital Library

Google Scholar

[8]

Alexander Sergeev et al. 2018. Horovod: fast and easy distributed deep learning in TF. arXiv preprint arXiv:1802.05799 (2018).

Google Scholar

[9]

Gerhard J Woeginger. 2018. The Open Shop Scheduling Problem. In STACS.

Google Scholar

Cited By

View all

Zhang YKumar A(2023)Lotan: Bridging the Gap between GNNs and Scalable Graph Analytics EnginesProceedings of the VLDB Endowment10.14778/3611479.361148316:11(2728-2741)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611483
Siddiqi SKern RBoehm M(2023)SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning ApplicationsProceedings of the ACM on Management of Data10.1145/36173381:3(1-26)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3617338
Boehm MInterlandi MJermaine CDas SPandis ISelçuk Candan KAmer-Yahia S(2023)Optimizing Tensor Computations: From Applications to Compilation and Runtime TechniquesCompanion of the 2023 International Conference on Management of Data10.1145/3555041.3589407(53-59)Online publication date: 4-Jun-2023
https://dl.acm.org/doi/10.1145/3555041.3589407
Show More Cited By

Recommendations

Cerebro: a data system for optimized deep learning model selection

Deep neural networks (deep nets) are revolutionizing many machine learning (ML) applications. But there is a major bottleneck to wider adoption: the pain and resource intensiveness of model selection. This empirical process involves exploring deep net ...
Intermittent human-in-the-loop model selection using cerebro: a demonstration

Deep learning (DL) is revolutionizing many fields. However, there is a major bottleneck for the wide adoption of DL: the pain of model selection, which requires exploring a large config space of model architecture and training hyper-parameters before ...
Identification and control using MLP, Elman, NARXSP and radial basis function networks: a comparative analysis

This paper describes four neural networks multilayer perceptron (MLP) network, Elman network, NARXSP network and radial basis function (RBF) network. Neural networks are applied for identification and control of DC servo motor and benchmark nonlinear ...

Comments

Information & Contributors

Information

Published In

DEEM'19: Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning

June 2019

72 pages

ISBN:9781450367974

DOI:10.1145/3329486

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Conference

SIGMOD/PODS '19

Sponsor:

SIGMOD

SIGMOD/PODS '19: International Conference on Management of Data

June 30, 2019

Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 44 of 67 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
467
Total Downloads

Downloads (Last 12 months)93
Downloads (Last 6 weeks)7

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhang YKumar A(2023)Lotan: Bridging the Gap between GNNs and Scalable Graph Analytics EnginesProceedings of the VLDB Endowment10.14778/3611479.361148316:11(2728-2741)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611483
Siddiqi SKern RBoehm M(2023)SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning ApplicationsProceedings of the ACM on Management of Data10.1145/36173381:3(1-26)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3617338
Boehm MInterlandi MJermaine CDas SPandis ISelçuk Candan KAmer-Yahia S(2023)Optimizing Tensor Computations: From Applications to Compilation and Runtime TechniquesCompanion of the 2023 International Conference on Management of Data10.1145/3555041.3589407(53-59)Online publication date: 4-Jun-2023
https://dl.acm.org/doi/10.1145/3555041.3589407
Chai CWang JLuo YNiu ZLi G(2022)Data Management for Machine Learning: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3148237(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3148237
Zhou XChai CLi GSun J(2022)Database Meets Artificial Intelligence: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.299464134:3(1096-1116)Online publication date: 1-Mar-2022
https://doi.org/10.1109/TKDE.2020.2994641
Li SKumar A(2021)Towards an optimized GROUP by abstraction for large-scale machine learningProceedings of the VLDB Endowment10.14778/3476249.347628414:11(2327-2340)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.14778/3476249.3476284
Zhang YMcQuillan FJayaram NKak NKhanna EKislal OValdano DKumar A(2021)Distributed deep learning on data systemsProceedings of the VLDB Endowment10.14778/3467861.346786714:10(1769-1782)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.14778/3467861.3467867
Wiese LHöltje DBoehm MStoyanovich JWhang S(2021)NNCompareProceedings of the Fifth Workshop on Data Management for End-To-End Machine Learning10.1145/3462462.3468884(1-7)Online publication date: 20-Jun-2021
https://dl.acm.org/doi/10.1145/3462462.3468884
Nakandala SZhang YKumar A(2020)CerebroProceedings of the VLDB Endowment10.14778/3407790.340781613:12(2159-2173)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3407790.3407816
Renz-Wieland AGemulla RZeuch SMarkl V(2020)Dynamic parameter allocation in parameter serversProceedings of the VLDB Endowment10.14778/3407790.340779613:12(1877-1890)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3407790.3407796
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Cerebro: a data system for optimized deep learning model selection

Intermittent human-in-the-loop model selection using cerebro: a demonstration

Identification and control using MLP, Elman, NARXSP and radial basis function networks: a comparative analysis

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

PDF

eReader

Login options

Full Access

Abstract

References

Cited By

Recommendations

Cerebro: a data system for optimized deep learning model selection

Intermittent human-in-the-loop model selection using cerebro: a demonstration

Identification and control using MLP, Elman, NARXSP and radial basis function networks: a comparative analysis

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations