title: DeMo: Decoupled Momentum Optimization
summary: Training large neural networks typically requires sharing gradients between
  accelerators through specialized high-speed interconnects. Drawing from the
  signal processing principles of frequency decomposition and energy compaction,
  we demonstrate that synchronizing full optimizer states and model parameters
  during training is unnecessary. By decoupling momentum updates and allowing
  controlled divergence in optimizer states across accelerators, we achieve
  improved convergence compared to state-of-the-art optimizers. We introduce
  {De}coupled {Mo}mentum (DeMo), a fused optimizer and data
  parallel algorithm that reduces inter-accelerator communication requirements by
  several orders of magnitude. This enables training of large neural networks
  even with limited network bandwidth and heterogeneous hardware. Our method is
  topology-agnostic and architecture-independent and supports scalable
  clock-synchronous distributed training with negligible compute and memory
  overhead. Empirical results show that models trained with DeMo match or exceed
  the performance of equivalent models trained with AdamW, while eliminating the
  need for high-speed interconnects when pre-training large scale foundation
  models. An open source reference PyTorch implementation is published on GitHub
  at https://github.com/bloc97/DeMo
paper_authors._id: 674e13bd8ac9fe84b87b94cc
paper_authors.name: Bowen Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-06T20:49:00.709Z
paper_authors._id: 62cf262026c94b143173ef65
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cf262026c94b143173ef65/FFcC-XhG5hMjApSvfk_he.png
paper_authors.fullname: Bowen Peng
paper_authors.type: user
paper_authors.user: bloc97
paper_authors._id: 674e13bd8ac9fe84b87b94cd
paper_authors.name: Jeffrey Quesnelle
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:56.192Z
paper_authors._id: 630581db99870e13d3e0006f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676652577978-630581db99870e13d3e0006f.jpeg
paper_authors.fullname: Jeffrey Quesnelle
paper_authors.type: user
paper_authors.user: emozilla
paper_authors._id: 674e13bd8ac9fe84b87b94ce
paper_authors.name: Diederik P. Kingma
paper_title: DeMo: Decoupled Momentum Optimization
paper_summary: Training large neural networks typically requires sharing gradients between
  accelerators through specialized high-speed interconnects. Drawing from the
  signal processing principles of frequency decomposition and energy compaction,
  we demonstrate that synchronizing full optimizer states and model parameters
  during training is unnecessary. By decoupling momentum updates and allowing
  controlled divergence in optimizer states across accelerators, we achieve
  improved convergence compared to state-of-the-art optimizers. We introduce
  {De}coupled {Mo}mentum (DeMo), a fused optimizer and data
  parallel algorithm that reduces inter-accelerator communication requirements by
  several orders of magnitude. This enables training of large neural networks
  even with limited network bandwidth and heterogeneous hardware. Our method is
  topology-agnostic and architecture-independent and supports scalable
  clock-synchronous distributed training with negligible compute and memory
  overhead. Empirical results show that models trained with DeMo match or exceed
  the performance of equivalent models trained with AdamW, while eliminating the
  need for high-speed interconnects when pre-training large scale foundation
  models. An open source reference PyTorch implementation is published on GitHub
  at https://github.com/bloc97/DeMo
paper_ai_keywords: decoupled momentum frequency decomposition energy compaction DeMo data parallel algorithm inter-accelerator communication heterogeneous hardware topology-agnostic architecture-independent scalable clock-synchronous distributed training compute and memory overhead AdamW pre-training large scale foundation models