DeMo: Decoupled Momentum Optimization
Training large neural networks typically requires sharing gradients between
accelerators through specialized high-speed interconnects. Drawing from the
signal processing principles of frequency decomposition and energy compaction,
we demonstrate that synchronizing full optimizer states and model parameters
during training is unnecessary. By decoupling momentum updates and allowing
controlled divergence in optimizer states across accelerators, we achieve
improved convergence compared to state-of-the-art optimizers. We introduce
{De}coupled {Mo}mentum (DeMo), a fused optimizer and data
parallel algorithm that reduces inter-accelerator communication requirements by
several orders of magnitude. This enables training of large neural networks
even with limited network bandwidth and heterogeneous hardware. Our method is
topology-agnostic and architecture-independent and supports scalable
clock-synchronous distributed training with negligible compute and memory
overhead. Empirical results show that models trained with DeMo match or exceed
the performance of equivalent models trained with AdamW, while eliminating the
need for high-speed interconnects when pre-training large scale foundation
models. An open source reference PyTorch implementation is published on GitHub
at https://github.com/bloc97/DeMo
674e13bd8ac9fe84b87b94cc
Bowen Peng
claimed_verified
2025-01-06T20:49:00.709Z
62cf262026c94b143173ef65
https://cdn-avatars.huggingface.co/v1/production/uploads/62cf262026c94b143173ef65/FFcC-XhG5hMjApSvfk_he.png
Bowen Peng
user
bloc97
674e13bd8ac9fe84b87b94cd
Jeffrey Quesnelle
claimed_verified
2024-12-05T09:28:56.192Z
630581db99870e13d3e0006f
https://cdn-avatars.huggingface.co/v1/production/uploads/1676652577978-630581db99870e13d3e0006f.jpeg
Jeffrey Quesnelle
user
emozilla
674e13bd8ac9fe84b87b94ce
Diederik P. Kingma
DeMo: Decoupled Momentum Optimization
Training large neural networks typically requires sharing gradients between
accelerators through specialized high-speed interconnects. Drawing from the
signal processing principles of frequency decomposition and energy compaction,
we demonstrate that synchronizing full optimizer states and model parameters
during training is unnecessary. By decoupling momentum updates and allowing
controlled divergence in optimizer states across accelerators, we achieve
improved convergence compared to state-of-the-art optimizers. We introduce
{De}coupled {Mo}mentum (DeMo), a fused optimizer and data
parallel algorithm that reduces inter-accelerator communication requirements by
several orders of magnitude. This enables training of large neural networks
even with limited network bandwidth and heterogeneous hardware. Our method is
topology-agnostic and architecture-independent and supports scalable
clock-synchronous distributed training with negligible compute and memory
overhead. Empirical results show that models trained with DeMo match or exceed
the performance of equivalent models trained with AdamW, while eliminating the
need for high-speed interconnects when pre-training large scale foundation
models. An open source reference PyTorch implementation is published on GitHub
at https://github.com/bloc97/DeMo
Decoupled Momentum (DeMo) optimizes large neural network training by reducing inter-accelerator communication, improving convergence, and enabling distributed training with limited bandwidth and hardware diversity.
decoupled momentum DeMo optimizer states data parallel algorithm inter-accelerator communication convergence AdamW scalable clock-synchronous distributed training topology-agnostic architecture-independent