One Initialization to Rule them All: Fine-tuning via Explained Variance
  Adaptation
Foundation models (FMs) are pre-trained on large-scale datasets and then
fine-tuned on a downstream task for a specific application. The most successful
and most commonly used fine-tuning method is to update the pre-trained weights
via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are
usually initialized at random with a uniform rank distribution across model
weights. Recent works focus on weight-driven initialization or learning of
adaptive ranks during training. Both approaches have only been investigated in
isolation, resulting in slow convergence or a uniform rank distribution, in
turn leading to sub-optimal performance. We propose to enhance LoRA by
initializing the new weights in a data-driven manner by computing singular
value decomposition on minibatches of activation vectors. Then, we initialize
the LoRA matrices with the obtained right-singular vectors and re-distribute
ranks among all weight matrices to explain the maximal amount of variance and
continue the standard LoRA fine-tuning procedure. This results in our new
method Explained Variance Adaptation (EVA). We apply EVA to a variety of
fine-tuning tasks ranging from language generation and understanding to image
classification and reinforcement learning. EVA exhibits faster convergence than
competitors and attains the highest average score across a multitude of tasks
per domain.
6707636dbe29abe3768fc949
Fabian Paischer
claimed_verified
2024-10-10T08:00:13.044Z
648826f845a9218318e0272c
https://cdn-avatars.huggingface.co/v1/production/uploads/648826f845a9218318e0272c/1zUQFA7TdQ8WC9Chskx6b.jpeg
Fabian Paischer
user
paischer101
6707636dbe29abe3768fc94a
Lukas Hauzenberger
claimed_verified
2024-10-10T08:58:29.988Z
6041ff7ff84ebe399f1c85ea
/avatars/a5e2306f3cd27e0ea1c30eeb81f870fa.svg
Lukas
user
sirluk
6707636dbe29abe3768fc94b
Thomas Schmied
admin_assigned
2024-10-10T09:14:26.041Z
64c3849269b1a6796052eac7
/avatars/9f0c832d5b51b659c7bb83074f02a648.svg
Thomas Schmied
user
thomasschmied
6707636dbe29abe3768fc94c
Benedikt Alkin
6707636dbe29abe3768fc94d
Marc Peter Deisenroth
6707636dbe29abe3768fc94e
Sepp Hochreiter
One Initialization to Rule them All: Fine-tuning via Explained Variance
  Adaptation
Foundation models (FMs) are pre-trained on large-scale datasets and then
fine-tuned on a downstream task for a specific application. The most successful
and most commonly used fine-tuning method is to update the pre-trained weights
via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are
usually initialized at random with a uniform rank distribution across model
weights. Recent works focus on weight-driven initialization or learning of
adaptive ranks during training. Both approaches have only been investigated in
isolation, resulting in slow convergence or a uniform rank distribution, in
turn leading to sub-optimal performance. We propose to enhance LoRA by
initializing the new weights in a data-driven manner by computing singular
value decomposition on minibatches of activation vectors. Then, we initialize
the LoRA matrices with the obtained right-singular vectors and re-distribute
ranks among all weight matrices to explain the maximal amount of variance and
continue the standard LoRA fine-tuning procedure. This results in our new
method Explained Variance Adaptation (EVA). We apply EVA to a variety of
fine-tuning tasks ranging from language generation and understanding to image
classification and reinforcement learning. EVA exhibits faster convergence than
competitors and attains the highest average score across a multitude of tasks
per domain.
Explained Variance Adaptation (EVA) enhances LoRA by data-driven initialization and rank redistribution to achieve faster convergence and superior performance across various fine-tuning tasks.
Foundation models fine-tuning low-rank adaptation (LoRA) weight-driven initialization adaptive ranks singular value decomposition activation vectors right-singular vectors Explained Variance Adaptation (EVA) language generation understanding image classification reinforcement learning