title: SALT: Singular Value Adaptation with Low-Rank Transformation
summary: The complex nature of medical image segmentation calls for models that are
  specifically designed to capture detailed, domain-specific features. Large
  foundation models offer considerable flexibility, yet the cost of fine-tuning
  these models remains a significant barrier. Parameter-Efficient Fine-Tuning
  (PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
  weights with low-rank matrices but may suffer from underfitting when the chosen
  rank is insufficient to capture domain-specific nuances. Conversely, full-rank
  Singular Value Decomposition (SVD) based methods provide comprehensive updates
  by modifying all singular values, yet they often lack flexibility and exhibit
  variable performance across datasets. We propose SALT (Singular Value
  Adaptation with Low-Rank Transformation), a method that selectively adapts the
  most influential singular values using trainable scale and shift parameters
  while complementing this with a low-rank update for the remaining subspace.
  This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
  effective adaptation without relying on increasing model size or depth.
  Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
  to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
  Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
  in low-resource settings. The code for SALT is available at:
  https://github.com/BioMedIA-MBZUAI/SALT
paper_authors._id: 67dd0d7c68dc6463747e6cac
paper_authors.name: Abdelrahman Elsayed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:53:23.059Z
paper_authors._id: 66f7e7a78d215c6331aabee2
paper_authors.avatarUrl: /avatars/f5a8e4894d889cf4c55ec4913381504b.svg
paper_authors.fullname: Abdelrahman Amr El Sayed
paper_authors.type: user
paper_authors.user: pythn
paper_authors._id: 67dd0d7c68dc6463747e6cad
paper_authors.name: Sarim Hashmi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:37:54.986Z
paper_authors._id: 62676a94dacab364889bb36c
paper_authors.avatarUrl: /avatars/0ead41b44957eb30564ea685ed22781a.svg
paper_authors.fullname: SARIM HASHMI
paper_authors.type: user
paper_authors.user: Sarim-Hash
paper_authors._id: 67dd0d7c68dc6463747e6cae
paper_authors.name: Mohammed Elseiagy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:15:43.955Z
paper_authors._id: 670a9e9d92e1f8429b4dff14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FqzdtSimXHTZEgSKYxRPA.png
paper_authors.fullname: Mohammed Elseiagy
paper_authors.type: user
paper_authors.user: MohammedElseiagy
paper_authors._id: 67dd0d7c68dc6463747e6caf
paper_authors.name: Hu Wang
paper_authors._id: 67dd0d7c68dc6463747e6cb0
paper_authors.name: Mohammad Yaqub
paper_authors._id: 67dd0d7c68dc6463747e6cb1
paper_authors.name: Ibrahim Almakky
paper_title: SALT: Singular Value Adaptation with Low-Rank Transformation
paper_summary: The complex nature of medical image segmentation calls for models that are
  specifically designed to capture detailed, domain-specific features. Large
  foundation models offer considerable flexibility, yet the cost of fine-tuning
  these models remains a significant barrier. Parameter-Efficient Fine-Tuning
  (PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
  weights with low-rank matrices but may suffer from underfitting when the chosen
  rank is insufficient to capture domain-specific nuances. Conversely, full-rank
  Singular Value Decomposition (SVD) based methods provide comprehensive updates
  by modifying all singular values, yet they often lack flexibility and exhibit
  variable performance across datasets. We propose SALT (Singular Value
  Adaptation with Low-Rank Transformation), a method that selectively adapts the
  most influential singular values using trainable scale and shift parameters
  while complementing this with a low-rank update for the remaining subspace.
  This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
  effective adaptation without relying on increasing model size or depth.
  Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
  to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
  Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
  in low-resource settings. The code for SALT is available at:
  https://github.com/BioMedIA-MBZUAI/SALT
paper_ai_summary: SALT combines LoRA and SVD to efficiently fine-tune medical image segmentation models, achieving state-of-the-art performance with limited trainable parameters.
paper_ai_keywords: Parameter-Efficient Fine-Tuning (PEFT) Low-Rank Adaptation (LoRA) Singular Value Decomposition (SVD) SALT (Singular Value Adaptation with Low-Rank Transformation) Dice medical datasets