DiPaCo: Distributed Path Composition
Progress in machine learning (ML) has been fueled by scaling neural network
models. This scaling has been enabled by ever more heroic feats of engineering,
necessary for accommodating ML approaches that require high bandwidth
communication between devices working in parallel. In this work, we propose a
co-designed modular architecture and training approach for ML models, dubbed
DIstributed PAth COmposition (DiPaCo). During training, DiPaCo distributes
computation by paths through a set of shared modules. Together with a Local-SGD
inspired optimization (DiLoCo) that keeps modules in sync with drastically
reduced communication, Our approach facilitates training across poorly
connected and heterogeneous workers, with a design that ensures robustness to
worker failures and preemptions. At inference time, only a single path needs to
be executed for each input, without the need for any model compression. We
consider this approach as a first prototype towards a new paradigm of
large-scale learning, one that is less synchronous and more modular. Our
experiments on the widely used C4 benchmark show that, for the same amount of
training steps but less wall-clock time, DiPaCo exceeds the performance of a 1
billion-parameter dense transformer language model by choosing one of 256
possible paths, each with a size of 150 million parameters.
65f91b1d16c9738732d57e3b
Arthur Douillard
admin_assigned
2024-03-19T10:15:34.136Z
622792366303bf1dc304f49f
/avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
Arthur Douillard
user
ArthurDouillard
65f91b1d16c9738732d57e3c
Qixuan Feng
65f91b1d16c9738732d57e3d
Andrei A. Rusu
65f91b1d16c9738732d57e3e
Adhiguna Kuncoro
65f91b1d16c9738732d57e3f
Yani Donchev
65f91b1d16c9738732d57e40
Rachita Chhaparia
65f91b1d16c9738732d57e41
Ionel Gog
65f91b1d16c9738732d57e42
Marc'Aurelio Ranzato
65f91b1d16c9738732d57e43
Jiajun Shen
admin_assigned
2024-03-19T10:17:20.313Z
61e11136fc27c0f5e3641eb2
/avatars/8d63d2f2a17754d1df66947071f3e9a0.svg
Jiajun Shen
user
jiajunshen
65f91b1d16c9738732d57e44
Arthur Szlam
DiPaCo: Distributed Path Composition
Progress in machine learning (ML) has been fueled by scaling neural network
models. This scaling has been enabled by ever more heroic feats of engineering,
necessary for accommodating ML approaches that require high bandwidth
communication between devices working in parallel. In this work, we propose a
co-designed modular architecture and training approach for ML models, dubbed
DIstributed PAth COmposition (DiPaCo). During training, DiPaCo distributes
computation by paths through a set of shared modules. Together with a Local-SGD
inspired optimization (DiLoCo) that keeps modules in sync with drastically
reduced communication, Our approach facilitates training across poorly
connected and heterogeneous workers, with a design that ensures robustness to
worker failures and preemptions. At inference time, only a single path needs to
be executed for each input, without the need for any model compression. We
consider this approach as a first prototype towards a new paradigm of
large-scale learning, one that is less synchronous and more modular. Our
experiments on the widely used C4 benchmark show that, for the same amount of
training steps but less wall-clock time, DiPaCo exceeds the performance of a 1
billion-parameter dense transformer language model by choosing one of 256
possible paths, each with a size of 150 million parameters.
A modular architecture and training approach, DiPaCo, improves large-scale neural network training by distributing computation and optimizing communication across heterogeneous workers, achieving performance exceeding a billion-parameter dense transformer.
DIstributed PAth COmposition (DiPaCo) Local-SGD inspired optimization (DiLoCo) C4 benchmark modular architecture large-scale learning