Distributed training proves to be a highly eﬀective approach for accelerating model
training, particularly for machine learning tasks that exceed the memory capacity of
a single accelerator, such as GPU, TPU, and more. In distributed training, the task
of training the model is divided and allocated to multiple working nodes. These nodes
concurrently execute local training tasks and collectively contribute to developing the
original task.
Data parallelism. Data parallelism (DP) is the most straightforward approach for
distributed training and has been inherently supported by famous machine learning
frameworks like TensorFlow and PyTorch. In the paradigm of data parallelism, the ini-
tial dataset is divided into multiple partitions, and diﬀerent data partitions are trained
in parallel by multiple accelerators. However, DP has memory redundancies across all
data partitions, and model states including model parameters, gradients, and optimizer
Beyond Efficiency