title: Trans-LoRA: towards data-free Transferable Parameter
    Efficient Finetuning
summary: Low-rank adapters (LoRA) and their variants are popular parameter-efficient
  fine-tuning (PEFT) techniques that closely match full model fine-tune
  performance while requiring only a small number of additional parameters. These
  additional LoRA parameters are specific to the base model being adapted. When
  the base model needs to be deprecated and replaced with a new one, all the
  associated LoRA modules need to be re-trained. Such re-training requires access
  to the data used to train the LoRA for the original base model. This is
  especially problematic for commercial cloud applications where the LoRA modules
  and the base models are hosted by service providers who may not be allowed to
  host proprietary client task data. To address this challenge, we propose
  Trans-LoRA -- a novel method for lossless, nearly data-free transfer
  of LoRAs across base models. Our approach relies on synthetic data to transfer
  LoRA modules. Using large language models, we design a synthetic data generator
  to approximate the data-generating process of the observed task data
  subset. Training on the resulting synthetic dataset transfers LoRA modules to
  new models. We show the effectiveness of our approach using both LLama and
  Gemma model families. Our approach achieves lossless (mostly improved) LoRA
  transfer between models within and across different base model families, and
  even between different PEFT methods, on a wide variety of tasks.
paper_authors._id: 66554b81ac73f7759fa84cad
paper_authors.name: Runqian Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T10:36:17.540Z
paper_authors._id: 656fd5c7e542334e136d88f6
paper_authors.avatarUrl: /avatars/56f960d6b0d64f1cbb7f9f5ba69995e1.svg
paper_authors.fullname: Ray Wang
paper_authors.type: user
paper_authors.user: raywang4
paper_authors._id: 66554b81ac73f7759fa84cae
paper_authors.name: Soumya Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-30T07:24:21.234Z
paper_authors._id: 65d11bc5bf849f3b79571771
paper_authors.avatarUrl: /avatars/af8d2c02e2ecc39a319a0741aa830422.svg
paper_authors.fullname: Soumya Ghosh
paper_authors.type: user
paper_authors.user: leproquark
paper_authors._id: 66554b81ac73f7759fa84caf
paper_authors.name: David Cox
paper_authors._id: 66554b81ac73f7759fa84cb0
paper_authors.name: Diego Antognini
paper_authors._id: 66554b81ac73f7759fa84cb1
paper_authors.name: Aude Oliva
paper_authors._id: 66554b81ac73f7759fa84cb2
paper_authors.name: Rogerio Feris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:26:55.838Z
paper_authors._id: 6426053ca5ec4a5cbc516567
paper_authors.avatarUrl: /avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
paper_authors.fullname: Rogerio Feris
paper_authors.type: user
paper_authors.user: rferis
paper_authors._id: 66554b81ac73f7759fa84cb3
paper_authors.name: Leonid Karlinsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:27:01.773Z
paper_authors._id: 62558f7cebfc059817b06690
paper_authors.avatarUrl: /avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
paper_authors.fullname: Leonid Karlinsky
paper_authors.type: user
paper_authors.user: leokarlin
paper_title: Trans-LoRA: towards data-free Transferable Parameter
    Efficient Finetuning
paper_summary: Low-rank adapters (LoRA) and their variants are popular parameter-efficient
  fine-tuning (PEFT) techniques that closely match full model fine-tune
  performance while requiring only a small number of additional parameters. These
  additional LoRA parameters are specific to the base model being adapted. When
  the base model needs to be deprecated and replaced with a new one, all the
  associated LoRA modules need to be re-trained. Such re-training requires access
  to the data used to train the LoRA for the original base model. This is
  especially problematic for commercial cloud applications where the LoRA modules
  and the base models are hosted by service providers who may not be allowed to
  host proprietary client task data. To address this challenge, we propose
  Trans-LoRA -- a novel method for lossless, nearly data-free transfer
  of LoRAs across base models. Our approach relies on synthetic data to transfer
  LoRA modules. Using large language models, we design a synthetic data generator
  to approximate the data-generating process of the observed task data
  subset. Training on the resulting synthetic dataset transfers LoRA modules to
  new models. We show the effectiveness of our approach using both LLama and
  Gemma model families. Our approach achieves lossless (mostly improved) LoRA
  transfer between models within and across different base model families, and
  even between different PEFT methods, on a wide variety of tasks.
paper_ai_summary: Trans-LoRA facilitates data-free transfer of LoRA modules across base models using synthetic data generated by large language models, enabling lossless or improved performance across various tasks and model families.
paper_ai_keywords: Low-rank adapters LoRA parameter-efficient fine-tuning PEFT base models re-training synthetic data data-free transfer LLama Gemma