title: Memory-Efficient LLM Training with Online Subspace Descent
summary: Recently, a wide range of memory-efficient LLM training algorithms have
  gained substantial popularity. These methods leverage the low-rank structure of
  gradients to project optimizer states into a subspace using projection matrix
  found by singular value decomposition (SVD). However, convergence of these
  algorithms is highly dependent on the update rules of their projection matrix.
  In this work, we provide the first convergence guarantee for arbitrary
  update rules of projection matrix. This guarantee is generally applicable to
  optimizers that can be analyzed with Hamiltonian Descent, including most common
  ones, such as LION, Adam. Inspired by our theoretical understanding, we propose
  Online Subspace Descent, a new family of subspace descent optimizer without
  SVD. Instead of updating the projection matrix with eigenvectors, Online
  Subspace Descent updates the projection matrix with online PCA. Online Subspace
  Descent is flexible and introduces only minimum overhead to training. We show
  that for the task of pretraining LLaMA models ranging from 60M to 7B parameters
  on the C4 dataset, Online Subspace Descent achieves lower perplexity and better
  downstream tasks performance than state-of-the-art low-rank training methods
  across different settings and narrows the gap with full-rank baselines.
paper_authors._id: 66cbe5ef50ce52b61b221506
paper_authors.name: Kaizhao Liang
paper_authors._id: 66cbe5ef50ce52b61b221507
paper_authors.name: Bo Liu
paper_authors._id: 66cbe5ef50ce52b61b221508
paper_authors.name: Lizhang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:52:00.398Z
paper_authors._id: 657ac67b5b9c86c08f2c073d
paper_authors.avatarUrl: /avatars/a6770776bba02413123d53eb020240b4.svg
paper_authors.fullname: 'Tim' Lizhang Chen
paper_authors.type: user
paper_authors.user: lzchenut
paper_authors._id: 66cbe5ef50ce52b61b221509
paper_authors.name: Qiang Liu
paper_title: Memory-Efficient LLM Training with Online Subspace Descent
paper_summary: Recently, a wide range of memory-efficient LLM training algorithms have
  gained substantial popularity. These methods leverage the low-rank structure of
  gradients to project optimizer states into a subspace using projection matrix
  found by singular value decomposition (SVD). However, convergence of these
  algorithms is highly dependent on the update rules of their projection matrix.
  In this work, we provide the first convergence guarantee for arbitrary
  update rules of projection matrix. This guarantee is generally applicable to
  optimizers that can be analyzed with Hamiltonian Descent, including most common
  ones, such as LION, Adam. Inspired by our theoretical understanding, we propose
  Online Subspace Descent, a new family of subspace descent optimizer without
  SVD. Instead of updating the projection matrix with eigenvectors, Online
  Subspace Descent updates the projection matrix with online PCA. Online Subspace
  Descent is flexible and introduces only minimum overhead to training. We show
  that for the task of pretraining LLaMA models ranging from 60M to 7B parameters
  on the C4 dataset, Online Subspace Descent achieves lower perplexity and better
  downstream tasks performance than state-of-the-art low-rank training methods
  across different settings and narrows the gap with full-rank baselines.
paper_ai_summary: A new family of memory-efficient optimizers called Online Subspace Descent provides convergence guarantees and outperforms existing methods for pretraining large language models.
paper_ai_keywords: memory-efficient LLM training low-rank structure gradients projection matrix singular value decomposition Hamiltonian Descent LION Adam Online Subspace Descent online PCA perplexity downstream tasks performance full-rank baselines