Memory-Efficient LLM Training with Online Subspace Descent
Recently, a wide range of memory-efficient LLM training algorithms have
gained substantial popularity. These methods leverage the low-rank structure of
gradients to project optimizer states into a subspace using projection matrix
found by singular value decomposition (SVD). However, convergence of these
algorithms is highly dependent on the update rules of their projection matrix.
In this work, we provide the first convergence guarantee for arbitrary
update rules of projection matrix. This guarantee is generally applicable to
optimizers that can be analyzed with Hamiltonian Descent, including most common
ones, such as LION, Adam. Inspired by our theoretical understanding, we propose
Online Subspace Descent, a new family of subspace descent optimizer without
SVD. Instead of updating the projection matrix with eigenvectors, Online
Subspace Descent updates the projection matrix with online PCA. Online Subspace
Descent is flexible and introduces only minimum overhead to training. We show
that for the task of pretraining LLaMA models ranging from 60M to 7B parameters
on the C4 dataset, Online Subspace Descent achieves lower perplexity and better
downstream tasks performance than state-of-the-art low-rank training methods
across different settings and narrows the gap with full-rank baselines.
66cbe5ef50ce52b61b221506
Kaizhao Liang
66cbe5ef50ce52b61b221507
Bo Liu
66cbe5ef50ce52b61b221508
Lizhang Chen
claimed_verified
2024-12-02T08:52:00.398Z
657ac67b5b9c86c08f2c073d
/avatars/a6770776bba02413123d53eb020240b4.svg
'Tim' Lizhang Chen
user
lzchenut
66cbe5ef50ce52b61b221509
Qiang Liu
Memory-Efficient LLM Training with Online Subspace Descent
Recently, a wide range of memory-efficient LLM training algorithms have
gained substantial popularity. These methods leverage the low-rank structure of
gradients to project optimizer states into a subspace using projection matrix
found by singular value decomposition (SVD). However, convergence of these
algorithms is highly dependent on the update rules of their projection matrix.
In this work, we provide the first convergence guarantee for arbitrary
update rules of projection matrix. This guarantee is generally applicable to
optimizers that can be analyzed with Hamiltonian Descent, including most common
ones, such as LION, Adam. Inspired by our theoretical understanding, we propose
Online Subspace Descent, a new family of subspace descent optimizer without
SVD. Instead of updating the projection matrix with eigenvectors, Online
Subspace Descent updates the projection matrix with online PCA. Online Subspace
Descent is flexible and introduces only minimum overhead to training. We show
that for the task of pretraining LLaMA models ranging from 60M to 7B parameters
on the C4 dataset, Online Subspace Descent achieves lower perplexity and better
downstream tasks performance than state-of-the-art low-rank training methods
across different settings and narrows the gap with full-rank baselines.
A new family of memory-efficient optimizers called Online Subspace Descent provides convergence guarantees and outperforms existing methods for pretraining large language models.
memory-efficient LLM training low-rank structure gradients projection matrix singular value decomposition Hamiltonian Descent LION Adam Online Subspace Descent online PCA perplexity downstream tasks performance full-rank baselines