title: Streaming DiLoCo with overlapping communication: Towards a Distributed
    Free Lunch
summary: Training of large language models (LLMs) is typically distributed across a
  large number of accelerators to reduce training time. Since internal states and
  parameter gradients need to be exchanged at each and every single gradient
  step, all devices need to be co-located using low-latency high-bandwidth
  communication links to support the required high volume of exchanged bits.
  Recently, distributed algorithms like DiLoCo have relaxed such co-location
  constraint: accelerators can be grouped into ``workers'', where
  synchronizations between workers only occur infrequently. This in turn means
  that workers can afford being connected by lower bandwidth communication links
  without affecting learning quality. However, in these methods, communication
  across workers still requires the same peak bandwidth as before, as the
  synchronizations require all parameters to be exchanged across all workers. In
  this paper, we improve DiLoCo in three ways. First, we synchronize only subsets
  of parameters in sequence, rather than all at once, which greatly reduces peak
  bandwidth. Second, we allow workers to continue training while synchronizing,
  which decreases wall clock time. Third, we quantize the data exchanged by
  workers, which further reduces bandwidth across workers. By properly combining
  these modifications, we show experimentally that we can distribute training of
  billion-scale parameters and reach similar quality as before, but reducing
  required bandwidth by two orders of magnitude.
paper_authors._id: 679ca01ecad2402cec0a939a
paper_authors.name: Arthur Douillard
paper_authors._id: 679ca01ecad2402cec0a939b
paper_authors.name: Yanislav Donchev
paper_authors._id: 679ca01ecad2402cec0a939c
paper_authors.name: Keith Rush
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:21:00.258Z
paper_authors._id: 65f1cf3aea7a16c6295f4737
paper_authors.avatarUrl: /avatars/905106897916e87e114ccc2d33e8e1c2.svg
paper_authors.fullname: Keith Rush
paper_authors.type: user
paper_authors.user: jkr26
paper_authors._id: 679ca01ecad2402cec0a939d
paper_authors.name: Satyen Kale
paper_authors._id: 679ca01ecad2402cec0a939e
paper_authors.name: Zachary Charles
paper_authors._id: 679ca01ecad2402cec0a939f
paper_authors.name: Zachary Garrett
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:20:51.929Z
paper_authors._id: 65f1be9109a3517bd1c8e17b
paper_authors.avatarUrl: /avatars/bc4112a3f749fe969ebb633b2e0cbc14.svg
paper_authors.fullname: Zachary Garrett
paper_authors.type: user
paper_authors.user: ZacharyGarrett
paper_authors._id: 679ca01ecad2402cec0a93a0
paper_authors.name: Gabriel Teston
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:20:57.817Z
paper_authors._id: 63177ee2879073cf83e8b2ab
paper_authors.avatarUrl: /avatars/e1b7974fd48c273616c13390a8052bf0.svg
paper_authors.fullname: Gabriel Teston
paper_authors.type: user
paper_authors.user: Gabriel-Teston
paper_authors._id: 679ca01ecad2402cec0a93a1
paper_authors.name: Dave Lacey
paper_authors._id: 679ca01ecad2402cec0a93a2
paper_authors.name: Ross McIlroy
paper_authors._id: 679ca01ecad2402cec0a93a3
paper_authors.name: Jiajun Shen
paper_authors._id: 679ca01ecad2402cec0a93a4
paper_authors.name: Alexandre Ramé
paper_authors._id: 679ca01ecad2402cec0a93a5
paper_authors.name: Arthur Szlam
paper_authors._id: 679ca01ecad2402cec0a93a6
paper_authors.name: Marc'Aurelio Ranzato
paper_authors._id: 679ca01ecad2402cec0a93a7
paper_authors.name: Paul Barham
paper_title: Streaming DiLoCo with overlapping communication: Towards a Distributed
    Free Lunch
paper_summary: Training of large language models (LLMs) is typically distributed across a
  large number of accelerators to reduce training time. Since internal states and
  parameter gradients need to be exchanged at each and every single gradient
  step, all devices need to be co-located using low-latency high-bandwidth
  communication links to support the required high volume of exchanged bits.
  Recently, distributed algorithms like DiLoCo have relaxed such co-location
  constraint: accelerators can be grouped into ``workers'', where
  synchronizations between workers only occur infrequently. This in turn means
  that workers can afford being connected by lower bandwidth communication links
  without affecting learning quality. However, in these methods, communication
  across workers still requires the same peak bandwidth as before, as the
  synchronizations require all parameters to be exchanged across all workers. In
  this paper, we improve DiLoCo in three ways. First, we synchronize only subsets
  of parameters in sequence, rather than all at once, which greatly reduces peak
  bandwidth. Second, we allow workers to continue training while synchronizing,
  which decreases wall clock time. Third, we quantize the data exchanged by
  workers, which further reduces bandwidth across workers. By properly combining
  these modifications, we show experimentally that we can distribute training of
  billion-scale parameters and reach similar quality as before, but reducing
  required bandwidth by two orders of magnitude.
paper_ai_keywords: large language models (LLMs) accelerators gradient step low-latency high-bandwidth communication links DiLoCo workers synchronizations lower bandwidth communication links peak bandwidth parameter subsets quantization