title: Eager Updates For Overlapped Communication and Computation in DiLoCo
summary: Distributed optimization methods such as DiLoCo have been shown to be
  effective in training very large models across multiple distributed workers,
  such as datacenters. These methods split updates into two parts: an inner
  optimization phase, where the workers independently execute multiple
  optimization steps on their own local data, and an outer optimization step,
  where the inner updates are synchronized. While such approaches require orders
  of magnitude less communication than standard data-parallel training, in
  settings where the workers are datacenters, even the limited communication
  requirements of these approaches can still cause significant slow downs due to
  the blocking necessary at each outer optimization step. In this paper, we
  investigate techniques to mitigate this issue by overlapping communication with
  computation in a manner that allows the outer optimization step to fully
  overlap with the inner optimization phase. We show that a particular variant,
  dubbed eager updates, provides competitive performance with standard DiLoCo in
  settings with low bandwidth between workers.
paper_authors._id: 67b5bcd091132877cf330179
paper_authors.name: Satyen Kale
paper_authors._id: 67b5bcd091132877cf33017a
paper_authors.name: Arthur Douillard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T14:42:51.668Z
paper_authors._id: 622792366303bf1dc304f49f
paper_authors.avatarUrl: /avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
paper_authors.fullname: Arthur Douillard
paper_authors.type: user
paper_authors.user: ArthurDouillard
paper_authors._id: 67b5bcd091132877cf33017b
paper_authors.name: Yanislav Donchev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T14:42:57.488Z
paper_authors._id: 61796df92357d28cdf952511
paper_authors.avatarUrl: /avatars/01821f364cf78c95aed1986d91d40610.svg
paper_authors.fullname: Yanislav Donchev
paper_authors.type: user
paper_authors.user: yannidd
paper_title: Eager Updates For Overlapped Communication and Computation in DiLoCo
paper_summary: Distributed optimization methods such as DiLoCo have been shown to be
  effective in training very large models across multiple distributed workers,
  such as datacenters. These methods split updates into two parts: an inner
  optimization phase, where the workers independently execute multiple
  optimization steps on their own local data, and an outer optimization step,
  where the inner updates are synchronized. While such approaches require orders
  of magnitude less communication than standard data-parallel training, in
  settings where the workers are datacenters, even the limited communication
  requirements of these approaches can still cause significant slow downs due to
  the blocking necessary at each outer optimization step. In this paper, we
  investigate techniques to mitigate this issue by overlapping communication with
  computation in a manner that allows the outer optimization step to fully
  overlap with the inner optimization phase. We show that a particular variant,
  dubbed eager updates, provides competitive performance with standard DiLoCo in
  settings with low bandwidth between workers.
paper_ai_keywords: DiLoCo distributed optimization methods inner optimization phase outer optimization step standard data-parallel training communication eager updates