From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from
  Low-Rank Gradients
Modern Large Language Models (LLMs) are composed of matrices with billions of
elements, making their storage and processing quite demanding in terms of
computational resources and memory usage. Being significantly large, such
matrices can often be expressed in low-rank format with potential to relax
resource requirements. Unlike prior works which focus on developing novel
matrix decomposition algorithms, in this work we first study the emergence of
low-rank structures across matrices within different layers of LLMs and
establish a consequential relationship between the gradient dynamics and
emerging low-rank expressiveness of matrices. Our findings reveal that
different layers exhibit varying levels of converged low-rank structure,
necessitating a non-uniform rank reduction across them to minimize performance
drop due to compression. In view of that, we present Weight Low-Rank Projection
(WeLore) that unifies weight compression and memory-efficient fine-tuning as
ONE, in a data-agnostic and one-shot way. WeLore capitalizes the heavy-tail
distribution of singular values to identify a suitable rank reduction ratio for
matrices within LLMs. Going beyond only as a compression technique, WeLore
categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank
Components (N-LRCs) based on their ability to express themselves as low-rank.
Our gradient perspective and extensive experiments illustrate that LRCs tend to
have better finetuning capabilities and can closely mimic (sometimes
outperform) the training loss trajectory and performance of full-finetuning
with notable memory and compute footprint reduction. For example, finetuning a
50\% compressed LLaMa-2 7B model using only a fraction of parameters in LRCs
(WeLore) can outperform its full finetuning with ~3x better throughput and
~0.6x GPU requirement. Our codes are available at
https://github.com/VITA-Group/welore
669733c98031bf92b4192002
Ajay Jaiswal
669733c98031bf92b4192003
Lu Yin
669733c98031bf92b4192004
Zhenyu Zhang
claimed_verified
2024-07-19T11:23:49.067Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
669733c98031bf92b4192005
Shiwei Liu
claimed_verified
2025-03-03T11:15:32.199Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
669733c98031bf92b4192006
Jiawei Zhao
669733c98031bf92b4192007
Yuandong Tian
669733c98031bf92b4192008
Zhangyang Wang
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from
  Low-Rank Gradients
Modern Large Language Models (LLMs) are composed of matrices with billions of
elements, making their storage and processing quite demanding in terms of
computational resources and memory usage. Being significantly large, such
matrices can often be expressed in low-rank format with potential to relax
resource requirements. Unlike prior works which focus on developing novel
matrix decomposition algorithms, in this work we first study the emergence of
low-rank structures across matrices within different layers of LLMs and
establish a consequential relationship between the gradient dynamics and
emerging low-rank expressiveness of matrices. Our findings reveal that
different layers exhibit varying levels of converged low-rank structure,
necessitating a non-uniform rank reduction across them to minimize performance
drop due to compression. In view of that, we present Weight Low-Rank Projection
(WeLore) that unifies weight compression and memory-efficient fine-tuning as
ONE, in a data-agnostic and one-shot way. WeLore capitalizes the heavy-tail
distribution of singular values to identify a suitable rank reduction ratio for
matrices within LLMs. Going beyond only as a compression technique, WeLore
categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank
Components (N-LRCs) based on their ability to express themselves as low-rank.
Our gradient perspective and extensive experiments illustrate that LRCs tend to
have better finetuning capabilities and can closely mimic (sometimes
outperform) the training loss trajectory and performance of full-finetuning
with notable memory and compute footprint reduction. For example, finetuning a
50\% compressed LLaMa-2 7B model using only a fraction of parameters in LRCs
(WeLore) can outperform its full finetuning with ~3x better throughput and
~0.6x GPU requirement. Our codes are available at
https://github.com/VITA-Group/welore
WeLore, a unified weight compression and memory-efficient fine-tuning technique for large language models, identifies and compresses low-rank components while preserving performance.
Large Language Models (LLMs) low-rank format matrix decomposition gradient dynamics low-rank expressiveness non-uniform rank reduction Weight Low-Rank Projection (WeLore) heavy-tail distribution singular values Low-rank Components (LRCs) Non-Low-rank Components (N-LRCs) full-finetuning throughput GPU requirement