NoLoCo: No-all-reduce Low Communication Training Method for Large Models
Training large language models is generally done via optimization methods on
clusters containing tens of thousands of accelerators, communicating over a
high-bandwidth interconnect. Scaling up these clusters is expensive and can
become impractical, imposing limits on the size of models that can be trained.
Several recent studies have proposed training methods that are less
communication intensive, avoiding the need for a highly connected compute
cluster. These state-of-the-art low communication training methods still employ
a synchronization step for model parameters, which, when performed over all
model replicas, can become costly on a low-bandwidth network.
  In this work, we propose a novel optimization method, NoLoCo, that does not
explicitly synchronize all model parameters during training and, as a result,
does not require any collective communication. NoLoCo implicitly synchronizes
model weights via a novel variant of the Nesterov momentum optimizer by
partially averaging model weights with a randomly selected other one. We
provide both a theoretical convergence analysis for our proposed optimizer as
well as empirical results from language model training.
  We benchmark NoLoCo on a wide range of accelerator counts and model sizes,
between 125M to 6.8B parameters. Our method requires significantly less
communication overhead than fully sharded data parallel training or even widely
used low communication training method, DiLoCo. The synchronization step itself
is estimated to be one magnitude faster than the all-reduce used in DiLoCo for
few hundred accelerators training over the internet. We also do not have any
global blocking communication that reduces accelerator idling time. Compared to
DiLoCo, we also observe up to 4% faster convergence rate with wide range of
model sizes and accelerator counts.
684c72743b733ba3336872cb
Jari Kolehmainen
extracted_pending
2025-06-13T18:48:21.254Z
6760ac9b53b5d7148556d5ef
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kkodSQK4l5zkh-xAXe-K3.png
Jari Kolehmainen
user
jkolehm
684c72743b733ba3336872cc
Nikolay Blagoev
684c72743b733ba3336872cd
John Donaghy
claimed_verified
2025-06-14T06:24:24.420Z
67e8761654cc82de7b632159
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9anqjWVQEBFRG7lyX-rcA.png
john donaghy
user
johnnyd-gensyn
684c72743b733ba3336872ce
Oğuzhan Ersoy
claimed_verified
2025-06-14T06:24:22.513Z
673c712c8a9ba30a57741a8c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c712c8a9ba30a57741a8c/khBSYmO5V1BM8Q_uPLLWB.jpeg
Oguzhan Ersoy
user
oguzer
684c72743b733ba3336872cf
Christopher Nies
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
Training large language models is generally done via optimization methods on
clusters containing tens of thousands of accelerators, communicating over a
high-bandwidth interconnect. Scaling up these clusters is expensive and can
become impractical, imposing limits on the size of models that can be trained.
Several recent studies have proposed training methods that are less
communication intensive, avoiding the need for a highly connected compute
cluster. These state-of-the-art low communication training methods still employ
a synchronization step for model parameters, which, when performed over all
model replicas, can become costly on a low-bandwidth network.
  In this work, we propose a novel optimization method, NoLoCo, that does not
explicitly synchronize all model parameters during training and, as a result,
does not require any collective communication. NoLoCo implicitly synchronizes
model weights via a novel variant of the Nesterov momentum optimizer by
partially averaging model weights with a randomly selected other one. We
provide both a theoretical convergence analysis for our proposed optimizer as
well as empirical results from language model training.
  We benchmark NoLoCo on a wide range of accelerator counts and model sizes,
between 125M to 6.8B parameters. Our method requires significantly less
communication overhead than fully sharded data parallel training or even widely
used low communication training method, DiLoCo. The synchronization step itself
is estimated to be one magnitude faster than the all-reduce used in DiLoCo for
few hundred accelerators training over the internet. We also do not have any
global blocking communication that reduces accelerator idling time. Compared to
DiLoCo, we also observe up to 4% faster convergence rate with wide range of
model sizes and accelerator counts.
NoLoCo is a novel optimization method that eliminates explicit parameter synchronization and reduces communication overhead during the training of large language models, achieving faster convergence rates and reduced idling time compared to existing methods.
NoLoCo Nesterov momentum optimizer parameter synchronization collective communication communication overhead empirical results language model training theoretical convergence analysis fully sharded data parallel training DiLoCo convergence rate accelerator idling time