title: From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from
    Low-Rank Gradients
summary: Modern Large Language Models (LLMs) are composed of matrices with billions of
  elements, making their storage and processing quite demanding in terms of
  computational resources and memory usage. Being significantly large, such
  matrices can often be expressed in low-rank format with potential to relax
  resource requirements. Unlike prior works which focus on developing novel
  matrix decomposition algorithms, in this work we first study the emergence of
  low-rank structures across matrices within different layers of LLMs and
  establish a consequential relationship between the gradient dynamics and
  emerging low-rank expressiveness of matrices. Our findings reveal that
  different layers exhibit varying levels of converged low-rank structure,
  necessitating a non-uniform rank reduction across them to minimize performance
  drop due to compression. In view of that, we present Weight Low-Rank Projection
  (WeLore) that unifies weight compression and memory-efficient fine-tuning as
  ONE, in a data-agnostic and one-shot way. WeLore capitalizes the heavy-tail
  distribution of singular values to identify a suitable rank reduction ratio for
  matrices within LLMs. Going beyond only as a compression technique, WeLore
  categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank
  Components (N-LRCs) based on their ability to express themselves as low-rank.
  Our gradient perspective and extensive experiments illustrate that LRCs tend to
  have better finetuning capabilities and can closely mimic (sometimes
  outperform) the training loss trajectory and performance of full-finetuning
  with notable memory and compute footprint reduction. For example, finetuning a
  50\% compressed LLaMa-2 7B model using only a fraction of parameters in LRCs
  (WeLore) can outperform its full finetuning with ~3x better throughput and
  ~0.6x GPU requirement. Our codes are available at
  https://github.com/VITA-Group/welore
paper_authors._id: 669733c98031bf92b4192002
paper_authors.name: Ajay Jaiswal
paper_authors._id: 669733c98031bf92b4192003
paper_authors.name: Lu Yin
paper_authors._id: 669733c98031bf92b4192004
paper_authors.name: Zhenyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:49.067Z
paper_authors._id: 649c888f67fd6c6aa97e5f85
paper_authors.avatarUrl: /avatars/9967b729916d1128773102797fed1673.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: Kyriection
paper_authors._id: 669733c98031bf92b4192005
paper_authors.name: Shiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T11:15:32.199Z
paper_authors._id: 65b04d2291e63920a7898c9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Shiweiliuiiiiiii
paper_authors._id: 669733c98031bf92b4192006
paper_authors.name: Jiawei Zhao
paper_authors._id: 669733c98031bf92b4192007
paper_authors.name: Yuandong Tian
paper_authors._id: 669733c98031bf92b4192008
paper_authors.name: Zhangyang Wang
paper_title: From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from
    Low-Rank Gradients
paper_summary: Modern Large Language Models (LLMs) are composed of matrices with billions of
  elements, making their storage and processing quite demanding in terms of
  computational resources and memory usage. Being significantly large, such
  matrices can often be expressed in low-rank format with potential to relax
  resource requirements. Unlike prior works which focus on developing novel
  matrix decomposition algorithms, in this work we first study the emergence of
  low-rank structures across matrices within different layers of LLMs and
  establish a consequential relationship between the gradient dynamics and
  emerging low-rank expressiveness of matrices. Our findings reveal that
  different layers exhibit varying levels of converged low-rank structure,
  necessitating a non-uniform rank reduction across them to minimize performance
  drop due to compression. In view of that, we present Weight Low-Rank Projection
  (WeLore) that unifies weight compression and memory-efficient fine-tuning as
  ONE, in a data-agnostic and one-shot way. WeLore capitalizes the heavy-tail
  distribution of singular values to identify a suitable rank reduction ratio for
  matrices within LLMs. Going beyond only as a compression technique, WeLore
  categorizes weight matrices into Low-rank Components (LRCs) and Non-Low-rank
  Components (N-LRCs) based on their ability to express themselves as low-rank.
  Our gradient perspective and extensive experiments illustrate that LRCs tend to
  have better finetuning capabilities and can closely mimic (sometimes
  outperform) the training loss trajectory and performance of full-finetuning
  with notable memory and compute footprint reduction. For example, finetuning a
  50\% compressed LLaMa-2 7B model using only a fraction of parameters in LRCs
  (WeLore) can outperform its full finetuning with ~3x better throughput and
  ~0.6x GPU requirement. Our codes are available at
  https://github.com/VITA-Group/welore
paper_ai_summary: WeLore, a unified weight compression and memory-efficient fine-tuning technique for large language models, identifies and compresses low-rank components while preserving performance.
paper_ai_keywords: Large Language Models (LLMs) low-rank format matrix decomposition gradient dynamics low-rank expressiveness non-uniform rank reduction Weight Low-Rank Projection (WeLore) heavy-tail distribution singular values Low-rank Components (LRCs) Non-Low-rank Components (N-LRCs) full-finetuning throughput GPU requirement