EoRA: Training-free Compensation for Compressed LLM with Eigenspace
  Low-Rank Approximation
In this work, we re-formulate the model compression problem into the
customized compensation problem: Given a compressed model, we aim to introduce
residual low-rank paths to compensate for compression errors under customized
requirements from users (e.g., tasks, compression ratios), resulting in greater
flexibility in adjusting overall capacity without being constrained by specific
compression formats. However, naively applying SVD to derive residual paths
causes suboptimal utilization of the low-rank representation capacity. Instead,
we propose Training-free Eigenspace Low-Rank Approximation (EoRA), a method
that directly minimizes compression-induced errors without requiring
gradient-based training, achieving fast optimization in minutes using a small
amount of calibration data. EoRA projects compression errors into the
eigenspace of input activations, leveraging eigenvalues to effectively
prioritize the reconstruction of high-importance error components. Moreover,
EoRA can be seamlessly integrated with fine-tuning and quantization to further
improve effectiveness and efficiency. EoRA consistently outperforms previous
methods in compensating errors for compressed LLaMA2/3 models on various tasks,
such as language generation, commonsense reasoning, and math reasoning tasks
(e.g., 31.31%/12.88% and 9.69% improvements on ARC-Easy/ARC-Challenge and
MathQA when compensating LLaMA3-8B that is quantized to 4-bit and pruned to 2:4
sparsity). EoRA offers a scalable, training-free solution to compensate for
compression errors, making it a powerful tool to deploy LLMs in various
capacity and efficiency requirements.
672115f045a90e94a76d3c5c
Shih-Yang Liu
claimed_verified
2024-10-30T09:52:55.372Z
62b58c68a1bae3c711c41321
https://cdn-avatars.huggingface.co/v1/production/uploads/62b58c68a1bae3c711c41321/FGQ1ifsPpmRi8P0ElxV5J.png
LIU Shih-yang
user
sliuau
672115f045a90e94a76d3c5d
Huck Yang
672115f045a90e94a76d3c5e
Chein-Yi Wang
672115f045a90e94a76d3c5f
Nai Chit Fung
672115f045a90e94a76d3c60
Hongxu Yin
claimed_verified
2025-03-27T09:06:01.693Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
672115f045a90e94a76d3c61
Charbel Sakr
672115f045a90e94a76d3c62
Saurav Muralidharan
672115f045a90e94a76d3c63
Kwang-Ting Cheng
672115f045a90e94a76d3c64
Jan Kautz
672115f045a90e94a76d3c65
Yu-Chiang Frank Wang
672115f045a90e94a76d3c66
Pavlo Molchanov
claimed_verified
2025-04-20T15:05:00.219Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
672115f045a90e94a76d3c67
Min-Hung Chen
claimed_verified
2024-10-30T09:52:57.332Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
EoRA: Training-free Compensation for Compressed LLM with Eigenspace
  Low-Rank Approximation
In this work, we re-formulate the model compression problem into the
customized compensation problem: Given a compressed model, we aim to introduce
residual low-rank paths to compensate for compression errors under customized
requirements from users (e.g., tasks, compression ratios), resulting in greater
flexibility in adjusting overall capacity without being constrained by specific
compression formats. However, naively applying SVD to derive residual paths
causes suboptimal utilization of the low-rank representation capacity. Instead,
we propose Training-free Eigenspace Low-Rank Approximation (EoRA), a method
that directly minimizes compression-induced errors without requiring
gradient-based training, achieving fast optimization in minutes using a small
amount of calibration data. EoRA projects compression errors into the
eigenspace of input activations, leveraging eigenvalues to effectively
prioritize the reconstruction of high-importance error components. Moreover,
EoRA can be seamlessly integrated with fine-tuning and quantization to further
improve effectiveness and efficiency. EoRA consistently outperforms previous
methods in compensating errors for compressed LLaMA2/3 models on various tasks,
such as language generation, commonsense reasoning, and math reasoning tasks
(e.g., 31.31%/12.88% and 9.69% improvements on ARC-Easy/ARC-Challenge and
MathQA when compensating LLaMA3-8B that is quantized to 4-bit and pruned to 2:4
sparsity). EoRA offers a scalable, training-free solution to compensate for
compression errors, making it a powerful tool to deploy LLMs in various
capacity and efficiency requirements.
EoRA, a training-free method, compensates compression errors in deep learning models by projecting them into the eigenspace of input activations, achieving superior performance across various tasks and compression techniques compared to previous methods.
model compression customized compensation residual low-rank paths low-rank approximation SVD eigenvalues input activations fine-tuning quantization LLaMA2/3 language generation commonsense reasoning math reasoning ARC-Easy ARC-Challenge MathQA