Shuttle2.0: Enhancing Spark and Flink Shuffle with Distributed Sorting and Adaptive Broadcast

Shuttle is OPPO's open‑source high‑availability, high‑performance Spark Remote Shuffle Service (RSS) released in April 2022, which has been stable and improves compute efficiency.

Shuttle2.0 extends Shuttle to support Flink by addressing the differences between Spark's pull‑based and Flink's push‑based shuffle, providing two integration schemes and a unified data model for partition, broadcast, and event streams.

Two integration schemes are described: Scheme 1 where the Shuttle Worker handles both data aggregation and downstream push, incurring high pressure and data copies; Scheme 2 where the Shuttle Worker only aggregates and stores data, while a Reader plugin pushes data to Reduce tasks, reducing pressure and improving stability.

A unified stream‑batch shuffle data model is introduced, ensuring order of partition, broadcast, and event data by assigning monotonically increasing sequence numbers to packets.

Shuttle2.0 also pipelines shuffle and sort, turning the serial process into a pipeline and moving sorting to distributed memory, which eliminates disk spill during the ShuffleRead phase.

Performance tests on a 20‑node cluster (48 cores, 384 GB RAM per node) with 1 TB TPCH data show Shuttle Sorter improves over Shuttle RSS by 11.7 % and over native Spark by 20.9 %; distributed sorting gains 36 % over native Spark and 23 % over RSS.

Adaptive BroadcastJoin moves broadcast data to Shuttle, avoiding driver OOM, and uses HashJoin when appropriate; extensive tests demonstrate significant speedups for large‑scale broadcast joins.

The article concludes with future directions for Shuttle as a universal data‑flow accelerator for various compute engines, including machine‑learning workloads.