Communication-Efficient Language Model Training Scales Reliably and
  Robustly: Scaling Laws for DiLoCo
As we scale to more massive machine learning models, the frequent
synchronization demands inherent in data-parallel approaches create significant
slowdowns, posing a critical challenge to further scaling. Recent work develops
an approach (DiLoCo) that relaxes synchronization demands without compromising
model quality. However, these works do not carefully analyze how DiLoCo's
behavior changes with model size. In this work, we study the scaling law
behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
how algorithmic factors, including number of model replicas, hyperparameters,
and token budget affect training in ways that can be accurately predicted via
scaling laws. We find that DiLoCo scales both predictably and robustly with
model size. When well-tuned, DiLoCo scales better than data-parallel training
with model size, and can outperform data-parallel training even at small model
sizes. Our results showcase a more general set of benefits of DiLoCo than
previously documented, including increased optimal batch sizes, improved
downstream generalization with scale, and improved evaluation loss for a fixed
token budget.
67d456e2f624d5a396490fa3
Zachary Charles
extracted_pending
2025-03-14T16:18:43.760Z
65f1c0bf22ea2a37a28b6466
/avatars/d9a11158bad338ada0d044476d8681c1.svg
Zachary Charles
user
zcharles8
67d456e2f624d5a396490fa4
Gabriel Teston
claimed_verified
2025-03-16T21:13:45.133Z
63177ee2879073cf83e8b2ab
/avatars/e1b7974fd48c273616c13390a8052bf0.svg
Gabriel Teston
user
Gabriel-Teston
67d456e2f624d5a396490fa5
Lucio Dery
67d456e2f624d5a396490fa6
Keith Rush
claimed_verified
2025-03-16T21:13:47.205Z
65f1cf3aea7a16c6295f4737
/avatars/905106897916e87e114ccc2d33e8e1c2.svg
Keith Rush
user
jkr26
67d456e2f624d5a396490fa7
Nova Fallen
67d456e2f624d5a396490fa8
Zachary Garrett
claimed_verified
2025-03-16T21:13:43.057Z
65f1be9109a3517bd1c8e17b
/avatars/bc4112a3f749fe969ebb633b2e0cbc14.svg
Zachary Garrett
user
ZacharyGarrett
67d456e2f624d5a396490fa9
Arthur Szlam
67d456e2f624d5a396490faa
Arthur Douillard
Communication-Efficient Language Model Training Scales Reliably and
  Robustly: Scaling Laws for DiLoCo
As we scale to more massive machine learning models, the frequent
synchronization demands inherent in data-parallel approaches create significant
slowdowns, posing a critical challenge to further scaling. Recent work develops
an approach (DiLoCo) that relaxes synchronization demands without compromising
model quality. However, these works do not carefully analyze how DiLoCo's
behavior changes with model size. In this work, we study the scaling law
behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
how algorithmic factors, including number of model replicas, hyperparameters,
and token budget affect training in ways that can be accurately predicted via
scaling laws. We find that DiLoCo scales both predictably and robustly with
model size. When well-tuned, DiLoCo scales better than data-parallel training
with model size, and can outperform data-parallel training even at small model
sizes. Our results showcase a more general set of benefits of DiLoCo than
previously documented, including increased optimal batch sizes, improved
downstream generalization with scale, and improved evaluation loss for a fixed
token budget.
DiLoCo scales predictably and robustly with model size, outperforming data-parallel training and offering benefits like increased batch sizes and improved generalization.
data-parallel approaches synchronization demands DiLoCo scaling law LLMs model replicas hyperparameters token budget optimal batch sizes downstream generalization evaluation loss