title: EoRA: Training-free Compensation for Compressed LLM with Eigenspace
    Low-Rank Approximation
summary: In this work, we re-formulate the model compression problem into the
  customized compensation problem: Given a compressed model, we aim to introduce
  residual low-rank paths to compensate for compression errors under customized
  requirements from users (e.g., tasks, compression ratios), resulting in greater
  flexibility in adjusting overall capacity without being constrained by specific
  compression formats. However, naively applying SVD to derive residual paths
  causes suboptimal utilization of the low-rank representation capacity. Instead,
  we propose Training-free Eigenspace Low-Rank Approximation (EoRA), a method
  that directly minimizes compression-induced errors without requiring
  gradient-based training, achieving fast optimization in minutes using a small
  amount of calibration data. EoRA projects compression errors into the
  eigenspace of input activations, leveraging eigenvalues to effectively
  prioritize the reconstruction of high-importance error components. Moreover,
  EoRA can be seamlessly integrated with fine-tuning and quantization to further
  improve effectiveness and efficiency. EoRA consistently outperforms previous
  methods in compensating errors for compressed LLaMA2/3 models on various tasks,
  such as language generation, commonsense reasoning, and math reasoning tasks
  (e.g., 31.31%/12.88% and 9.69% improvements on ARC-Easy/ARC-Challenge and
  MathQA when compensating LLaMA3-8B that is quantized to 4-bit and pruned to 2:4
  sparsity). EoRA offers a scalable, training-free solution to compensate for
  compression errors, making it a powerful tool to deploy LLMs in various
  capacity and efficiency requirements.
paper_authors._id: 672115f045a90e94a76d3c5c
paper_authors.name: Shih-Yang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:55.372Z
paper_authors._id: 62b58c68a1bae3c711c41321
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b58c68a1bae3c711c41321/FGQ1ifsPpmRi8P0ElxV5J.png
paper_authors.fullname: LIU Shih-yang
paper_authors.type: user
paper_authors.user: sliuau
paper_authors._id: 672115f045a90e94a76d3c5d
paper_authors.name: Huck Yang
paper_authors._id: 672115f045a90e94a76d3c5e
paper_authors.name: Chein-Yi Wang
paper_authors._id: 672115f045a90e94a76d3c5f
paper_authors.name: Nai Chit Fung
paper_authors._id: 672115f045a90e94a76d3c60
paper_authors.name: Hongxu Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:06:01.693Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 672115f045a90e94a76d3c61
paper_authors.name: Charbel Sakr
paper_authors._id: 672115f045a90e94a76d3c62
paper_authors.name: Saurav Muralidharan
paper_authors._id: 672115f045a90e94a76d3c63
paper_authors.name: Kwang-Ting Cheng
paper_authors._id: 672115f045a90e94a76d3c64
paper_authors.name: Jan Kautz
paper_authors._id: 672115f045a90e94a76d3c65
paper_authors.name: Yu-Chiang Frank Wang
paper_authors._id: 672115f045a90e94a76d3c66
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:05:00.219Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 672115f045a90e94a76d3c67
paper_authors.name: Min-Hung Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:57.332Z
paper_authors._id: 64ae22dd1aee69ece065cdcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
paper_authors.fullname: Min-Hung Chen
paper_authors.type: user
paper_authors.user: cmhungsteve
paper_title: EoRA: Training-free Compensation for Compressed LLM with Eigenspace
    Low-Rank Approximation
paper_summary: In this work, we re-formulate the model compression problem into the
  customized compensation problem: Given a compressed model, we aim to introduce
  residual low-rank paths to compensate for compression errors under customized
  requirements from users (e.g., tasks, compression ratios), resulting in greater
  flexibility in adjusting overall capacity without being constrained by specific
  compression formats. However, naively applying SVD to derive residual paths
  causes suboptimal utilization of the low-rank representation capacity. Instead,
  we propose Training-free Eigenspace Low-Rank Approximation (EoRA), a method
  that directly minimizes compression-induced errors without requiring
  gradient-based training, achieving fast optimization in minutes using a small
  amount of calibration data. EoRA projects compression errors into the
  eigenspace of input activations, leveraging eigenvalues to effectively
  prioritize the reconstruction of high-importance error components. Moreover,
  EoRA can be seamlessly integrated with fine-tuning and quantization to further
  improve effectiveness and efficiency. EoRA consistently outperforms previous
  methods in compensating errors for compressed LLaMA2/3 models on various tasks,
  such as language generation, commonsense reasoning, and math reasoning tasks
  (e.g., 31.31%/12.88% and 9.69% improvements on ARC-Easy/ARC-Challenge and
  MathQA when compensating LLaMA3-8B that is quantized to 4-bit and pruned to 2:4
  sparsity). EoRA offers a scalable, training-free solution to compensate for
  compression errors, making it a powerful tool to deploy LLMs in various
  capacity and efficiency requirements.
paper_ai_summary: EoRA, a training-free method, compensates compression errors in deep learning models by projecting them into the eigenspace of input activations, achieving superior performance across various tasks and compression techniques compared to previous methods.
paper_ai_keywords: model compression customized compensation residual low-rank paths low-rank approximation SVD eigenvalues input activations fine-tuning quantization LLaMA2/3 language generation commonsense reasoning math reasoning ARC-Easy ARC-Challenge MathQA