title: Communication-Efficient Language Model Training Scales Reliably and
    Robustly: Scaling Laws for DiLoCo
summary: As we scale to more massive machine learning models, the frequent
  synchronization demands inherent in data-parallel approaches create significant
  slowdowns, posing a critical challenge to further scaling. Recent work develops
  an approach (DiLoCo) that relaxes synchronization demands without compromising
  model quality. However, these works do not carefully analyze how DiLoCo's
  behavior changes with model size. In this work, we study the scaling law
  behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
  how algorithmic factors, including number of model replicas, hyperparameters,
  and token budget affect training in ways that can be accurately predicted via
  scaling laws. We find that DiLoCo scales both predictably and robustly with
  model size. When well-tuned, DiLoCo scales better than data-parallel training
  with model size, and can outperform data-parallel training even at small model
  sizes. Our results showcase a more general set of benefits of DiLoCo than
  previously documented, including increased optimal batch sizes, improved
  downstream generalization with scale, and improved evaluation loss for a fixed
  token budget.
paper_authors._id: 67d456e2f624d5a396490fa3
paper_authors.name: Zachary Charles
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-14T16:18:43.760Z
paper_authors._id: 65f1c0bf22ea2a37a28b6466
paper_authors.avatarUrl: /avatars/d9a11158bad338ada0d044476d8681c1.svg
paper_authors.fullname: Zachary Charles
paper_authors.type: user
paper_authors.user: zcharles8
paper_authors._id: 67d456e2f624d5a396490fa4
paper_authors.name: Gabriel Teston
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:13:45.133Z
paper_authors._id: 63177ee2879073cf83e8b2ab
paper_authors.avatarUrl: /avatars/e1b7974fd48c273616c13390a8052bf0.svg
paper_authors.fullname: Gabriel Teston
paper_authors.type: user
paper_authors.user: Gabriel-Teston
paper_authors._id: 67d456e2f624d5a396490fa5
paper_authors.name: Lucio Dery
paper_authors._id: 67d456e2f624d5a396490fa6
paper_authors.name: Keith Rush
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:13:47.205Z
paper_authors._id: 65f1cf3aea7a16c6295f4737
paper_authors.avatarUrl: /avatars/905106897916e87e114ccc2d33e8e1c2.svg
paper_authors.fullname: Keith Rush
paper_authors.type: user
paper_authors.user: jkr26
paper_authors._id: 67d456e2f624d5a396490fa7
paper_authors.name: Nova Fallen
paper_authors._id: 67d456e2f624d5a396490fa8
paper_authors.name: Zachary Garrett
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:13:43.057Z
paper_authors._id: 65f1be9109a3517bd1c8e17b
paper_authors.avatarUrl: /avatars/bc4112a3f749fe969ebb633b2e0cbc14.svg
paper_authors.fullname: Zachary Garrett
paper_authors.type: user
paper_authors.user: ZacharyGarrett
paper_authors._id: 67d456e2f624d5a396490fa9
paper_authors.name: Arthur Szlam
paper_authors._id: 67d456e2f624d5a396490faa
paper_authors.name: Arthur Douillard
paper_title: Communication-Efficient Language Model Training Scales Reliably and
    Robustly: Scaling Laws for DiLoCo
paper_summary: As we scale to more massive machine learning models, the frequent
  synchronization demands inherent in data-parallel approaches create significant
  slowdowns, posing a critical challenge to further scaling. Recent work develops
  an approach (DiLoCo) that relaxes synchronization demands without compromising
  model quality. However, these works do not carefully analyze how DiLoCo's
  behavior changes with model size. In this work, we study the scaling law
  behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
  how algorithmic factors, including number of model replicas, hyperparameters,
  and token budget affect training in ways that can be accurately predicted via
  scaling laws. We find that DiLoCo scales both predictably and robustly with
  model size. When well-tuned, DiLoCo scales better than data-parallel training
  with model size, and can outperform data-parallel training even at small model
  sizes. Our results showcase a more general set of benefits of DiLoCo than
  previously documented, including increased optimal batch sizes, improved
  downstream generalization with scale, and improved evaluation loss for a fixed
  token budget.
paper_ai_summary: DiLoCo scales predictably and robustly with model size, outperforming data-parallel training and offering benefits like increased batch sizes and improved generalization.
paper_ai_keywords: data-parallel approaches synchronization demands DiLoCo scaling law LLMs model replicas hyperparameters token budget optimal batch sizes downstream generalization evaluation loss