title: DiPaCo: Distributed Path Composition
summary: Progress in machine learning (ML) has been fueled by scaling neural network
  models. This scaling has been enabled by ever more heroic feats of engineering,
  necessary for accommodating ML approaches that require high bandwidth
  communication between devices working in parallel. In this work, we propose a
  co-designed modular architecture and training approach for ML models, dubbed
  DIstributed PAth COmposition (DiPaCo). During training, DiPaCo distributes
  computation by paths through a set of shared modules. Together with a Local-SGD
  inspired optimization (DiLoCo) that keeps modules in sync with drastically
  reduced communication, Our approach facilitates training across poorly
  connected and heterogeneous workers, with a design that ensures robustness to
  worker failures and preemptions. At inference time, only a single path needs to
  be executed for each input, without the need for any model compression. We
  consider this approach as a first prototype towards a new paradigm of
  large-scale learning, one that is less synchronous and more modular. Our
  experiments on the widely used C4 benchmark show that, for the same amount of
  training steps but less wall-clock time, DiPaCo exceeds the performance of a 1
  billion-parameter dense transformer language model by choosing one of 256
  possible paths, each with a size of 150 million parameters.
paper_authors._id: 65f91b1d16c9738732d57e3b
paper_authors.name: Arthur Douillard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:15:34.136Z
paper_authors._id: 622792366303bf1dc304f49f
paper_authors.avatarUrl: /avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
paper_authors.fullname: Arthur Douillard
paper_authors.type: user
paper_authors.user: ArthurDouillard
paper_authors._id: 65f91b1d16c9738732d57e3c
paper_authors.name: Qixuan Feng
paper_authors._id: 65f91b1d16c9738732d57e3d
paper_authors.name: Andrei A. Rusu
paper_authors._id: 65f91b1d16c9738732d57e3e
paper_authors.name: Adhiguna Kuncoro
paper_authors._id: 65f91b1d16c9738732d57e3f
paper_authors.name: Yani Donchev
paper_authors._id: 65f91b1d16c9738732d57e40
paper_authors.name: Rachita Chhaparia
paper_authors._id: 65f91b1d16c9738732d57e41
paper_authors.name: Ionel Gog
paper_authors._id: 65f91b1d16c9738732d57e42
paper_authors.name: Marc'Aurelio Ranzato
paper_authors._id: 65f91b1d16c9738732d57e43
paper_authors.name: Jiajun Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:17:20.313Z
paper_authors._id: 61e11136fc27c0f5e3641eb2
paper_authors.avatarUrl: /avatars/8d63d2f2a17754d1df66947071f3e9a0.svg
paper_authors.fullname: Jiajun Shen
paper_authors.type: user
paper_authors.user: jiajunshen
paper_authors._id: 65f91b1d16c9738732d57e44
paper_authors.name: Arthur Szlam
paper_title: DiPaCo: Distributed Path Composition
paper_summary: Progress in machine learning (ML) has been fueled by scaling neural network
  models. This scaling has been enabled by ever more heroic feats of engineering,
  necessary for accommodating ML approaches that require high bandwidth
  communication between devices working in parallel. In this work, we propose a
  co-designed modular architecture and training approach for ML models, dubbed
  DIstributed PAth COmposition (DiPaCo). During training, DiPaCo distributes
  computation by paths through a set of shared modules. Together with a Local-SGD
  inspired optimization (DiLoCo) that keeps modules in sync with drastically
  reduced communication, Our approach facilitates training across poorly
  connected and heterogeneous workers, with a design that ensures robustness to
  worker failures and preemptions. At inference time, only a single path needs to
  be executed for each input, without the need for any model compression. We
  consider this approach as a first prototype towards a new paradigm of
  large-scale learning, one that is less synchronous and more modular. Our
  experiments on the widely used C4 benchmark show that, for the same amount of
  training steps but less wall-clock time, DiPaCo exceeds the performance of a 1
  billion-parameter dense transformer language model by choosing one of 256
  possible paths, each with a size of 150 million parameters.
paper_ai_keywords: neural network models DIstributed PAth COmposition (DiPaCo) Local-SGD inspired optimization (DiLoCo) poorly connected and heterogeneous workers robustness to worker failures and preemptions C4 benchmark dense transformer language model path-based training model compression