Accelerating Hyperparameter Optimisation with PyCOMPSs
Authors: 
Albert Njoroge Kahira, Leonardo Bautista Gomez, Javier Conejero, 
Rosa M. BadiaAuthors Info & Claims
Albert Njoroge Kahira, Leonardo Bautista Gomez, Javier Conejero, Rosa M. BadiaAuthors Info & ClaimsICPP Workshops '19: Workshop Proceedings of the 48th International Conference on Parallel Processing
Article No.: 20, Pages 1 - 8
Abstract
Machine Learning applications now span across multiple domains due to the increase in computational power of modern systems. There has been a recent surge in Machine Learning applications in High Performance Computing (HPC) in an attempt to speed up training. However, besides training, hyperparameters optimisation(HPO) is one of the most time consuming and resource intensive parts in a Machine Learning Workflow. Numerous algorithms and tools exist to accelerate the process of finding the right parameters for a model. Most of these tools do not utilize the parallelism provided by modern systems and are serial or limited to a single node. The few ones that are offer distributed execution require a serious amount of programming effort.
There is, therefore, a need for a tool/scheme that can scale and leverage HPC infrastructures such as supercomputers, with minimum programmers effort and little or no overhead in performance. We present a HPO scheme built on top of PyCOMPSs, a programming model and runtime which aims to ease the development of parallel applications for distributed infrastructures. We show that PyCOMPSs is a powerful framework that can accelerate the process of Hyperparameter Optimisation across multiple devices and computing units. We also show that PyCOMPSs provides easy programmability, seamless distribution and scalability, key features missing in existing tools. Furthermore, we perform a detailed performance analysis showing different configurations to demonstrate the effectiveness our approach.
References
[1]
{n. d.}. Paraver: a flexible performance analysis tool. ({n. d.}). https://tools.bsc.es/paraver
[2]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). https://www.tensorflow.org/ Software available from tensorflow.org.
[3]
J. Bergstra, D. Yamins, and D. D. Cox. 2013. Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 (ICML'13). JMLR.org, I-115--I-123. http://dl.acm.org/citation.cfm?id=3042817.3042832
[4]
James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. In NIPS Proceedings.
[5]
James Bergstra JAMESBERGSTRA and Umontrealca Yoshua Bengio YOSHUABENGIO. 2012. Random Search for HyperParameter Optimization. Journal of Machine Learning Research (2012). arXiv:1504.05070
[6]
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. Technical Report. arXiv:1604.07316v1 https://arxiv.org/pdf/1604.07316.pdf
[7]
Lars Hertel, Julian Collado, Peter Sadowski, Julian Collado, and Pierre Baldi. 2018. Sherpa: Hyperparameter Optimization for Machine Learning Models. Nips (2018). https://www.semanticscholar.org/paper/Sherpa-{%}3A-Hyperparameter-Optimization-for-Machine-Hertel-Collado/342c5b941398c733659dd6fe9e0b3b4e3f210877
[8]
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).
[9]
Jeffery Kinnison, Nathaniel Kremer-Herman, Douglas Thain, and Walter Scheirer. 2018. SHADHO: Massively scalable hardware-aware distributed hyperparameter optimization. In Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018.
[10]
Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Technical Report. Citeseer.
[11]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. {n. d.}. ImageNet Classification with Deep Convolutional Neural Networks. Technical Report. http://code.google.com/p/cuda-convnet/
[12]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (may 2015), 436--444.
[13]
Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. (2010). http://yann.lecun.com/exdb/mnist/
[14]
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Distributed Model Selection and Training. 2012 (jul 2018). arXiv:1807.05118 http://arxiv.org/abs/1807.05118
[15]
Francisco Madrigal, Camille Maurice, and Frédéric Lerasle. 2018. Hyperparameter optimization tools comparison for multiple object tracking applications. Machine Vision and Applications 30, 2 (mar 2018), 269--289.
[16]
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2017. Ray: A Distributed Framework for Emerging AI Applications. (dec 2017). arXiv:1712.05889 http://arxiv.org/abs/1712.05889
[17]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.
[18]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.
[19]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. {n. d.}. Practical Bayesian Optimization of Machine Learning Algorithms. Technical Report. https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf
[20]
William Song and Jim Cai. 2015. End-to-end deep neural network for automatic speech recognition. Standford CS224D Reports (2015).
[21]
Enric Tejedor, Yolanda Becerra, Guillem Alomar, Anna Queralt, Rosa M. Badia, Jordi Torres, Toni Cortes, and Jesús Labarta. 2017. PyCOMPSs: Parallel computational workflows in Python. International Journal of High Performance Computing Applications (2017).
Index Terms
- Accelerating Hyperparameter Optimisation with PyCOMPSs
Recommendations
GPU Cache System for COMPSs: A Task-Based Distributed Computing Framework
Euro-Par 2024: Parallel ProcessingAbstractIn this paper, we propose a novel GPU cache system for COMPSs, a task-based distributed computing framework that enables the execution of parallel applications on heterogeneous clusters. GPU COMPSs tasks can exploit the computational power of ...
Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs
PyHPC'17: Proceedings of the 7th Workshop on Python for High-Performance and Scientific ComputingPython has been adopted as programming language by a large number of scientific communities. Additionally to the easy programming interface, the large number of libraries and modules that have been made available by a large number of contributors, have ...
Implementation of OpenMP Data-Sharing on CAPE
SoICT '18: Proceedings of the 9th International Symposium on Information and Communication TechnologyCAPE (Checkpointing-Aided Parallel Execution) is a framework that automatically translates and executes OpenMP on distributed-memory architectures based on checkpoint technique. In some experiments, this approach shows high-performance on distributed-...
Comments
Information & Contributors
Information
Published In
August 2019
241 pages
ISBN:9781450371964
DOI:10.1145/3339186
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
In-Cooperation
- University of Tsukuba: University of Tsukuba
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 August 2019
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- ?la Caixa? Foundation
Conference
ICPP 2019
Acceptance Rates
Overall Acceptance Rate 91 of 313 submissions, 29%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 99Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)1
Reflects downloads up to 12 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in