DiLoCo: Distributed Low-Communication Training of Language Models
Large language models (LLM) have become a critical component in many
applications of machine learning. However, standard approaches to training LLM
require a large number of tightly interconnected accelerators, with devices
exchanging gradients and other intermediate states at each optimization step.
While it is difficult to build and maintain a single computing cluster hosting
many accelerators, it might be easier to find several computing clusters each
hosting a smaller number of devices. In this work, we propose a distributed
optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
training of language models on islands of devices that are poorly connected.
The approach is a variant of federated averaging, where the number of inner
steps is large, the inner optimizer is AdamW, and the outer optimizer is
Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
workers performs as well as fully synchronous optimization while communicating
500 times less. DiLoCo exhibits great robustness to the data distribution of
each worker. It is also robust to resources becoming unavailable over time, and
vice versa, it can seamlessly leverage resources that become available during
training.
655440f639bce75e6cc78a38
Arthur Douillard
claimed_verified
2024-06-25T07:34:49.342Z
622792366303bf1dc304f49f
/avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
Arthur Douillard
user
ArthurDouillard
655440f639bce75e6cc78a39
Qixuan Feng
655440f639bce75e6cc78a3a
Andrei A. Rusu
655440f639bce75e6cc78a3b
Rachita Chhaparia
655440f639bce75e6cc78a3c
Yani Donchev
655440f639bce75e6cc78a3d
Adhiguna Kuncoro
655440f639bce75e6cc78a3e
Marc'Aurelio Ranzato
655440f639bce75e6cc78a3f
Arthur Szlam
655440f639bce75e6cc78a40
Jiajun Shen
DiLoCo: Distributed Low-Communication Training of Language Models
Large language models (LLM) have become a critical component in many
applications of machine learning. However, standard approaches to training LLM
require a large number of tightly interconnected accelerators, with devices
exchanging gradients and other intermediate states at each optimization step.
While it is difficult to build and maintain a single computing cluster hosting
many accelerators, it might be easier to find several computing clusters each
hosting a smaller number of devices. In this work, we propose a distributed
optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
training of language models on islands of devices that are poorly connected.
The approach is a variant of federated averaging, where the number of inner
steps is large, the inner optimizer is AdamW, and the outer optimizer is
Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
workers performs as well as fully synchronous optimization while communicating
500 times less. DiLoCo exhibits great robustness to the data distribution of
each worker. It is also robust to resources becoming unavailable over time, and
vice versa, it can seamlessly leverage resources that become available during
training.
A distributed optimization algorithm called DiLoCo allows for effective training of language models across poorly connected devices with reduced communication, maintaining performance comparable to fully synchronous optimization.
distributed optimization DiLoCo federated averaging AdamW Nesterov momentum C4 dataset