One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
<|reference_start|>One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation: Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned on a downstream task for a specific application. The most successful and most commonly used fine-tuning method is to update the pre-trained weights via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are usually initialized at random with a uniform rank distribution across model weights. Recent works focus on weight-driven initialization or learning of adaptive ranks during training. Both approaches have only been investigated in isolation, resulting in slow convergence or a uniform rank distribution, in turn leading to sub-optimal performance. We propose to enhance LoRA by initializing the new weights in a data-driven manner by computing singular value decomposition on minibatches of activation vectors. Then, we initialize the LoRA matrices with the obtained right-singular vectors and re-distribute ranks among all weight matrices to explain the maximal amount of variance and continue the standard LoRA fine-tuning procedure. This results in our new method Explained Variance Adaptation (EVA). We apply EVA to a variety of fine-tuning tasks ranging from language generation and understanding to image classification and reinforcement learning. EVA exhibits faster convergence than competitors and attains the highest average score across a multitude of tasks per domain.<|reference_end|>
@article{paischer2024one,
    title={One Initialization to Rule them All: Fine-tuning via Explained Variance
  Adaptation},
    author={Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin,
  Marc Peter Deisenroth, Sepp Hochreiter},
    journal={arXiv preprint arXiv:2410.07170},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07170},
    primaryClass={cs.LG cs.AI cs.CL stat.ML}
}