In-Database Machine Learning with CorgiPile: Stochastic Gradient Descent without Full Data Shuffle

Stochastic gradient descent (SGD) is the cornerstone of modern ML systems. However, SGD requires random data access that is inefficient when implemented in systems that rely on secondary storage such as HDD and SSD, e.g., in-DB ML systems and TensorFlow/PyTorch over large files. In this paper, we propose a simple but novel two-level data shuffling strategy for SGD, namely CorgiPile. CorgiPile can avoid full data shuffle while maintaining comparable convergence rate of SGD as if a full shuffle were performed. We provide a theoretical analysis of CorgiPile on its convergence behavior. We further integrate CorgiPile into PostgreSQL by introducing three new physical operators with optimizations. Our experimental results show that CorgiPile can achieve comparable convergence rate to the full shuffle based SGD, and 1.6-12.8X faster than two state-of-the-art in-DB ML systems, Apache MADlib and Bismarck, on both HDD and SSD.