-
MLSys: The New Frontier of Machine Learning Systems
Authors:
Alexander Ratner,
Dan Alistarh,
Gustavo Alonso,
David G. Andersen,
Peter Bailis,
Sarah Bird,
Nicholas Carlini,
Bryan Catanzaro,
Jennifer Chayes,
Eric Chung,
Bill Dally,
Jeff Dean,
Inderjit S. Dhillon,
Alexandros Dimakis,
Pradeep Dubey,
Charles Elkan,
Grigori Fursin,
Gregory R. Ganger,
Lise Getoor,
Phillip B. Gibbons,
Garth A. Gibson,
Joseph E. Gonzalez,
Justin Gottschlich,
Song Han,
Kim Hazelwood
, et al. (44 additional authors not shown)
Abstract:
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains a significant obstacle, in large part due to the radically different development and deployment profile of modern ML methods, and the range of practical concerns that come with broader adoption. We propose to foster a ne…
▽ More
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains a significant obstacle, in large part due to the radically different development and deployment profile of modern ML methods, and the range of practical concerns that come with broader adoption. We propose to foster a new systems machine learning research community at the intersection of the traditional systems and ML communities, focused on topics such as hardware systems for ML, software systems for ML, and ML optimized for metrics beyond predictive accuracy. To do this, we describe a new conference, MLSys, that explicitly targets research at the intersection of systems and machine learning with a program committee split evenly between experts in systems and ML, and an explicit focus on topics at the intersection of the two.
△ Less
Submitted 1 December, 2019; v1 submitted 29 March, 2019;
originally announced April 2019.
-
Primitives for Dynamic Big Model Parallelism
Authors:
Seunghak Lee,
Jin Kyu Kim,
Xun Zheng,
Qirong Ho,
Garth A. Gibson,
Eric P. Xing
Abstract:
When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness additional memory and processors. However, naive, unstructured parallelization of M…
▽ More
When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness additional memory and processors. However, naive, unstructured parallelization of ML algorithms can make inefficient use of distributed memory, while failing to obtain proportional convergence speedups - or can even result in divergence. We develop a framework of primitives for dynamic model-parallelism, STRADS, in order to explore partitioning and update scheduling of model variables in distributed ML algorithms - thus improving their memory efficiency while presenting new opportunities to speed up convergence without compromising inference correctness. We demonstrate the efficacy of model-parallel algorithms implemented in STRADS versus popular implementations for Topic Modeling, Matrix Factorization and Lasso.
△ Less
Submitted 17 June, 2014;
originally announced June 2014.
-
Structure-Aware Dynamic Scheduler for Parallel Machine Learning
Authors:
Seunghak Lee,
Jin Kyu Kim,
Qirong Ho,
Garth A. Gibson,
Eric P. Xing
Abstract:
Training large machine learning (ML) models with many variables or parameters can take a long time if one employs sequential procedures even with stochastic updates. A natural solution is to turn to distributed computing on a cluster; however, naive, unstructured parallelization of ML algorithms does not usually lead to a proportional speedup and can even result in divergence, because dependencies…
▽ More
Training large machine learning (ML) models with many variables or parameters can take a long time if one employs sequential procedures even with stochastic updates. A natural solution is to turn to distributed computing on a cluster; however, naive, unstructured parallelization of ML algorithms does not usually lead to a proportional speedup and can even result in divergence, because dependencies between model elements can attenuate the computational gains from parallelization and compromise correctness of inference. Recent efforts toward this issue have benefited from exploiting the static, a priori block structures residing in ML algorithms. In this paper, we take this path further by exploring the dynamic block structures and workloads therein present during ML program execution, which offers new opportunities for improving convergence, correctness, and load balancing in distributed ML. We propose and showcase a general-purpose scheduler, STRADS, for coordinating distributed updates in ML algorithms, which harnesses the aforementioned opportunities in a systematic way. We provide theoretical guarantees for our scheduler, and demonstrate its efficacy versus static block structures on Lasso and Matrix Factorization.
△ Less
Submitted 30 December, 2013; v1 submitted 19 December, 2013;
originally announced December 2013.
-
Structural and Electronic Properties of Amorphous and Polycrystalline In2Se3 Films
Authors:
A. Chaiken,
K. Nauka,
G. A. Gibson,
Heon Lee,
C. C. Yang,
J. Wu,
J. W. Ager,
K. M. Yu,
W. Walukiewicz
Abstract:
Structural and electronic properties of amorphous and single-phase polycrystalline films of gamma- and kappa-In2Se3 have been measured. The stable gamma phase nucleates homogeneously in the film bulk and has a high resistivity, while the metastable kappa phase nucleates at the film surface and has a moderate resistivity. The microstructures of hot-deposited and post-annealed cold-deposited gamma…
▽ More
Structural and electronic properties of amorphous and single-phase polycrystalline films of gamma- and kappa-In2Se3 have been measured. The stable gamma phase nucleates homogeneously in the film bulk and has a high resistivity, while the metastable kappa phase nucleates at the film surface and has a moderate resistivity. The microstructures of hot-deposited and post-annealed cold-deposited gamma films are quite different but the electronic properties are similar. The increase in the resistivity of amorphous In2Se3 films upon annealing is interpreted in terms of the replacement of In-In bonds with In-Se bonds during crystallization. Great care must be taken in the preparation of In2Se3 films for electrical measurements as the presence of excess chalcogen or surface oxidation may greatly affect the film properties.
△ Less
Submitted 19 March, 2003;
originally announced March 2003.
