title: One Initialization to Rule them All: Fine-tuning via Explained Variance
    Adaptation
summary: Foundation models (FMs) are pre-trained on large-scale datasets and then
  fine-tuned on a downstream task for a specific application. The most successful
  and most commonly used fine-tuning method is to update the pre-trained weights
  via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are
  usually initialized at random with a uniform rank distribution across model
  weights. Recent works focus on weight-driven initialization or learning of
  adaptive ranks during training. Both approaches have only been investigated in
  isolation, resulting in slow convergence or a uniform rank distribution, in
  turn leading to sub-optimal performance. We propose to enhance LoRA by
  initializing the new weights in a data-driven manner by computing singular
  value decomposition on minibatches of activation vectors. Then, we initialize
  the LoRA matrices with the obtained right-singular vectors and re-distribute
  ranks among all weight matrices to explain the maximal amount of variance and
  continue the standard LoRA fine-tuning procedure. This results in our new
  method Explained Variance Adaptation (EVA). We apply EVA to a variety of
  fine-tuning tasks ranging from language generation and understanding to image
  classification and reinforcement learning. EVA exhibits faster convergence than
  competitors and attains the highest average score across a multitude of tasks
  per domain.
paper_authors._id: 6707636dbe29abe3768fc949
paper_authors.name: Fabian Paischer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:13.044Z
paper_authors._id: 648826f845a9218318e0272c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648826f845a9218318e0272c/1zUQFA7TdQ8WC9Chskx6b.jpeg
paper_authors.fullname: Fabian Paischer
paper_authors.type: user
paper_authors.user: paischer101
paper_authors._id: 6707636dbe29abe3768fc94a
paper_authors.name: Lukas Hauzenberger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:58:29.988Z
paper_authors._id: 6041ff7ff84ebe399f1c85ea
paper_authors.avatarUrl: /avatars/a5e2306f3cd27e0ea1c30eeb81f870fa.svg
paper_authors.fullname: Lukas
paper_authors.type: user
paper_authors.user: sirluk
paper_authors._id: 6707636dbe29abe3768fc94b
paper_authors.name: Thomas Schmied
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:14:26.041Z
paper_authors._id: 64c3849269b1a6796052eac7
paper_authors.avatarUrl: /avatars/9f0c832d5b51b659c7bb83074f02a648.svg
paper_authors.fullname: Thomas Schmied
paper_authors.type: user
paper_authors.user: thomasschmied
paper_authors._id: 6707636dbe29abe3768fc94c
paper_authors.name: Benedikt Alkin
paper_authors._id: 6707636dbe29abe3768fc94d
paper_authors.name: Marc Peter Deisenroth
paper_authors._id: 6707636dbe29abe3768fc94e
paper_authors.name: Sepp Hochreiter
paper_title: One Initialization to Rule them All: Fine-tuning via Explained Variance
    Adaptation
paper_summary: Foundation models (FMs) are pre-trained on large-scale datasets and then
  fine-tuned on a downstream task for a specific application. The most successful
  and most commonly used fine-tuning method is to update the pre-trained weights
  via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are
  usually initialized at random with a uniform rank distribution across model
  weights. Recent works focus on weight-driven initialization or learning of
  adaptive ranks during training. Both approaches have only been investigated in
  isolation, resulting in slow convergence or a uniform rank distribution, in
  turn leading to sub-optimal performance. We propose to enhance LoRA by
  initializing the new weights in a data-driven manner by computing singular
  value decomposition on minibatches of activation vectors. Then, we initialize
  the LoRA matrices with the obtained right-singular vectors and re-distribute
  ranks among all weight matrices to explain the maximal amount of variance and
  continue the standard LoRA fine-tuning procedure. This results in our new
  method Explained Variance Adaptation (EVA). We apply EVA to a variety of
  fine-tuning tasks ranging from language generation and understanding to image
  classification and reinforcement learning. EVA exhibits faster convergence than
  competitors and attains the highest average score across a multitude of tasks
  per domain.
paper_ai_summary: Explained Variance Adaptation (EVA) enhances LoRA by data-driven initialization and rank redistribution to achieve faster convergence and superior performance across various fine-tuning tasks.
paper_ai_keywords: Foundation models fine-tuning low-rank adaptation (LoRA) weight-driven initialization adaptive ranks singular value decomposition activation vectors right-singular vectors Explained Variance Adaptation (EVA) language generation understanding image classification reinforcement learning