Magnet: push-based shuffle service for large-scale data processing

Over the past decade, Apache Spark has become a popular compute engine for large scale data processing. Similar to other compute engines based on the MapReduce compute paradigm, the shuffle operation, namely the all-to-all transfer of the intermediate data, plays an important role in Spark. At LinkedIn, with the rapid growth of the data size and scale of the Spark deployment, the shuffle operation is becoming a bottleneck of further scaling the infrastructure. This has led to overall job slowness and even failures for long running jobs. This not only impacts developer productivity for addressing such slowness and failures, but also results in high operational cost of infrastructure.

In this work, we describe the main bottlenecks impacting shuffle scalability. We propose Magnet, a novel shuffle mechanism that can scale to handle petabytes of daily shuffled data and clusters with thousands of nodes. Magnet is designed to work with both on-prem and cloud-based cluster deployments. It addresses a key shuffle scalability bottleneck by merging fragmented intermediate shuffle data into large blocks. Magnet provides further improvements by co-locating merged blocks with the reduce tasks. Our benchmarks show that Magnet significantly improves shuffle performance independent of the underlying hardware. Magnet reduces the end-to-end runtime of Linkedln's production Spark jobs by nearly 30%. Furthermore, Magnet improves user productivity by removing the shuffle related tuning burden from users.