  Shuffle is one of the most expensive communication primitives in distributed
data processing and is difficult to scale. Prior work addresses the scalability
challenges of shuffle by building monolithic shuffle systems. These systems are
costly to develop, and they are tightly integrated with batch processing
frameworks that offer only high-level APIs such as SQL. New applications, such
as ML training, require more flexibility and finer-grained interoperability
with shuffle. They are often unable to leverage existing shuffle optimizations.
  We propose an extensible shuffle architecture. We present Exoshuffle, a
library for distributed shuffle that offers competitive performance and
scalability as well as greater flexibility than monolithic shuffle systems. We
design an architecture that decouples the shuffle control plane from the data
plane without sacrificing performance. We build Exoshuffle on Ray, a
distributed futures system for data and ML applications, and demonstrate that
we can: (1) rewrite previous shuffle optimizations as application-level
libraries with an order of magnitude less code, (2) achieve shuffle performance
and scalability competitive with monolithic shuffle systems, and break the
CloudSort record as the world's most cost-efficient sorting system, and (3)
enable new applications such as ML training to easily leverage scalable
shuffle.

Distributed, Parallel, and Cluster Computing