SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Fine-tuning LLMs is both computationally and memory-intensive. While
parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the
number of trainable parameters and lower memory usage, they do not decrease
computational cost. In some cases, they may even slow down fine-tuning. In this
paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning
through contextual sparsity. We propose a lightweight, training-free SVD
sparsity estimator that dynamically selects a sparse subset of weights for loss
and gradient computation. Also, we systematically analyze and address
sensitivity across layers, tokens, and training steps. Our experimental results
show that SparseLoRA reduces computational cost by up to 2.2 times and a
measured speedup of up to 1.6 times while maintaining accuracy across various
downstream tasks, including commonsense and arithmetic reasoning, code
generation, and instruction following.
6858a7f0c0c8e29df8ea3c06
Samir Khaki
claimed_verified
2025-07-01T10:14:44.911Z
64b38bc2a248169796fec4fa
/avatars/1371021f2ef0ce2197cc13627c9e03c9.svg
Samir Khaki
user
Skhaki
6858a7f0c0c8e29df8ea3c07
Xiuyu Li
claimed_verified
2025-07-03T07:41:46.329Z
644570ba2d91b15b4c7f6311
/avatars/d5e66012066d0c330b8f23718b1499d8.svg
Xiuyu Li
user
xiuyul
6858a7f0c0c8e29df8ea3c08
Junxian Guo
6858a7f0c0c8e29df8ea3c09
Ligeng Zhu
6858a7f0c0c8e29df8ea3c0a
Chenfeng Xu
6858a7f0c0c8e29df8ea3c0b
Konstantinos N. Plataniotis
6858a7f0c0c8e29df8ea3c0c
Amir Yazdanbakhsh
6858a7f0c0c8e29df8ea3c0d
Kurt Keutzer
6858a7f0c0c8e29df8ea3c0e
Song Han
6858a7f0c0c8e29df8ea3c0f
Zhijian Liu
claimed_verified
2025-07-01T10:14:42.873Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Fine-tuning LLMs is both computationally and memory-intensive. While
parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the
number of trainable parameters and lower memory usage, they do not decrease
computational cost. In some cases, they may even slow down fine-tuning. In this
paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning
through contextual sparsity. We propose a lightweight, training-free SVD
sparsity estimator that dynamically selects a sparse subset of weights for loss
and gradient computation. Also, we systematically analyze and address
sensitivity across layers, tokens, and training steps. Our experimental results
show that SparseLoRA reduces computational cost by up to 2.2 times and a
measured speedup of up to 1.6 times while maintaining accuracy across various
downstream tasks, including commonsense and arithmetic reasoning, code
generation, and instruction following.
SparseLoRA reduces computational cost and speeds up fine-tuning of LLMs by dynamically selecting a sparse subset of weights for loss and gradient computation.
QLoRA DoRA parameter-efficient fine-tuning SparseLoRA contextual sparsity SVD sparsity estimator computational cost commonsense reasoning arithmetic reasoning code generation instruction following