SALT: Singular Value Adaptation with Low-Rank Transformation
The complex nature of medical image segmentation calls for models that are
specifically designed to capture detailed, domain-specific features. Large
foundation models offer considerable flexibility, yet the cost of fine-tuning
these models remains a significant barrier. Parameter-Efficient Fine-Tuning
(PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
weights with low-rank matrices but may suffer from underfitting when the chosen
rank is insufficient to capture domain-specific nuances. Conversely, full-rank
Singular Value Decomposition (SVD) based methods provide comprehensive updates
by modifying all singular values, yet they often lack flexibility and exhibit
variable performance across datasets. We propose SALT (Singular Value
Adaptation with Low-Rank Transformation), a method that selectively adapts the
most influential singular values using trainable scale and shift parameters
while complementing this with a low-rank update for the remaining subspace.
This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
effective adaptation without relying on increasing model size or depth.
Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
in low-resource settings. The code for SALT is available at:
https://github.com/BioMedIA-MBZUAI/SALT
67dd0d7c68dc6463747e6cac
Abdelrahman Elsayed
claimed_verified
2025-03-24T08:53:23.059Z
66f7e7a78d215c6331aabee2
/avatars/f5a8e4894d889cf4c55ec4913381504b.svg
Abdelrahman Amr El Sayed
user
pythn
67dd0d7c68dc6463747e6cad
Sarim Hashmi
claimed_verified
2025-03-21T11:37:54.986Z
62676a94dacab364889bb36c
/avatars/0ead41b44957eb30564ea685ed22781a.svg
SARIM HASHMI
user
Sarim-Hash
67dd0d7c68dc6463747e6cae
Mohammed Elseiagy
claimed_verified
2025-03-31T08:15:43.955Z
670a9e9d92e1f8429b4dff14
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FqzdtSimXHTZEgSKYxRPA.png
Mohammed Elseiagy
user
MohammedElseiagy
67dd0d7c68dc6463747e6caf
Hu Wang
67dd0d7c68dc6463747e6cb0
Mohammad Yaqub
67dd0d7c68dc6463747e6cb1
Ibrahim Almakky
SALT: Singular Value Adaptation with Low-Rank Transformation
The complex nature of medical image segmentation calls for models that are
specifically designed to capture detailed, domain-specific features. Large
foundation models offer considerable flexibility, yet the cost of fine-tuning
these models remains a significant barrier. Parameter-Efficient Fine-Tuning
(PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
weights with low-rank matrices but may suffer from underfitting when the chosen
rank is insufficient to capture domain-specific nuances. Conversely, full-rank
Singular Value Decomposition (SVD) based methods provide comprehensive updates
by modifying all singular values, yet they often lack flexibility and exhibit
variable performance across datasets. We propose SALT (Singular Value
Adaptation with Low-Rank Transformation), a method that selectively adapts the
most influential singular values using trainable scale and shift parameters
while complementing this with a low-rank update for the remaining subspace.
This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
effective adaptation without relying on increasing model size or depth.
Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
in low-resource settings. The code for SALT is available at:
https://github.com/BioMedIA-MBZUAI/SALT
SALT combines LoRA and SVD to efficiently fine-tune medical image segmentation models, achieving state-of-the-art performance with limited trainable parameters.
Parameter-Efficient Fine-Tuning (PEFT) Low-Rank Adaptation (LoRA) Singular Value Decomposition (SVD) SALT (Singular Value Adaptation with Low-Rank Transformation) Dice medical datasets