title: DiLoCo: Distributed Low-Communication Training of Language Models
summary: Large language models (LLM) have become a critical component in many
  applications of machine learning. However, standard approaches to training LLM
  require a large number of tightly interconnected accelerators, with devices
  exchanging gradients and other intermediate states at each optimization step.
  While it is difficult to build and maintain a single computing cluster hosting
  many accelerators, it might be easier to find several computing clusters each
  hosting a smaller number of devices. In this work, we propose a distributed
  optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
  training of language models on islands of devices that are poorly connected.
  The approach is a variant of federated averaging, where the number of inner
  steps is large, the inner optimizer is AdamW, and the outer optimizer is
  Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
  workers performs as well as fully synchronous optimization while communicating
  500 times less. DiLoCo exhibits great robustness to the data distribution of
  each worker. It is also robust to resources becoming unavailable over time, and
  vice versa, it can seamlessly leverage resources that become available during
  training.
paper_authors._id: 655440f639bce75e6cc78a38
paper_authors.name: Arthur Douillard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:34:49.342Z
paper_authors._id: 622792366303bf1dc304f49f
paper_authors.avatarUrl: /avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
paper_authors.fullname: Arthur Douillard
paper_authors.type: user
paper_authors.user: ArthurDouillard
paper_authors._id: 655440f639bce75e6cc78a39
paper_authors.name: Qixuan Feng
paper_authors._id: 655440f639bce75e6cc78a3a
paper_authors.name: Andrei A. Rusu
paper_authors._id: 655440f639bce75e6cc78a3b
paper_authors.name: Rachita Chhaparia
paper_authors._id: 655440f639bce75e6cc78a3c
paper_authors.name: Yani Donchev
paper_authors._id: 655440f639bce75e6cc78a3d
paper_authors.name: Adhiguna Kuncoro
paper_authors._id: 655440f639bce75e6cc78a3e
paper_authors.name: Marc'Aurelio Ranzato
paper_authors._id: 655440f639bce75e6cc78a3f
paper_authors.name: Arthur Szlam
paper_authors._id: 655440f639bce75e6cc78a40
paper_authors.name: Jiajun Shen
paper_title: DiLoCo: Distributed Low-Communication Training of Language Models
paper_summary: Large language models (LLM) have become a critical component in many
  applications of machine learning. However, standard approaches to training LLM
  require a large number of tightly interconnected accelerators, with devices
  exchanging gradients and other intermediate states at each optimization step.
  While it is difficult to build and maintain a single computing cluster hosting
  many accelerators, it might be easier to find several computing clusters each
  hosting a smaller number of devices. In this work, we propose a distributed
  optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
  training of language models on islands of devices that are poorly connected.
  The approach is a variant of federated averaging, where the number of inner
  steps is large, the inner optimizer is AdamW, and the outer optimizer is
  Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
  workers performs as well as fully synchronous optimization while communicating
  500 times less. DiLoCo exhibits great robustness to the data distribution of
  each worker. It is also robust to resources becoming unavailable over time, and
  vice versa, it can seamlessly leverage resources that become available during
  training.
paper_ai_summary: A distributed optimization algorithm called DiLoCo allows for effective training of language models across poorly connected devices with reduced communication, maintaining performance comparable to fully synchronous optimization.
paper_ai_keywords: distributed optimization DiLoCo federated averaging AdamW Nesterov momentum C4 dataset