title: RandLoRA: Full-rank parameter-efficient fine-tuning of large models
summary: Low-Rank Adaptation (LoRA) and its variants have shown impressive results in
  reducing the number of trainable parameters and memory requirements of large
  transformer networks while maintaining fine-tuning performance. However, the
  low-rank nature of the weight update inherently limits the representation power
  of fine-tuned models, potentially compromising performance on complex tasks.
  This raises a critical question: when a performance gap between LoRA and
  standard fine-tuning is observed, is it due to the reduced number of trainable
  parameters or the rank deficiency? This paper aims to answer this question by
  introducing RandLoRA, a parameter-efficient method that performs full-rank
  updates using a learned linear combinations of low-rank, non-trainable random
  matrices. Our method limits the number of trainable parameters by restricting
  optimization to diagonal scaling matrices applied to the fixed random matrices.
  This allows us to effectively overcome the low-rank limitations while
  maintaining parameter and memory efficiency during training. Through extensive
  experimentation across vision, language, and vision-language benchmarks, we
  systematically evaluate the limitations of LoRA and existing random basis
  methods. Our findings reveal that full-rank updates are beneficial across
  vision and language tasks individually, and even more so for vision-language
  tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the
  performance gap between standard fine-tuning and LoRA, demonstrating its
  efficacy.
paper_authors._id: 67a1a4278b6584b24ff98eaf
paper_authors.name: Paul Albert
paper_authors._id: 67a1a4278b6584b24ff98eb0
paper_authors.name: Frederic Z. Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:15:29.195Z
paper_authors._id: 668ca1b3ed63008dfa687990
paper_authors.avatarUrl: /avatars/b3ae0e7ff60c60db605288c9cf5ae6f3.svg
paper_authors.fullname: Fred Zhang
paper_authors.type: user
paper_authors.user: fredzzhang
paper_authors._id: 67a1a4278b6584b24ff98eb1
paper_authors.name: Hemanth Saratchandran
paper_authors._id: 67a1a4278b6584b24ff98eb2
paper_authors.name: Cristian Rodriguez-Opazo
paper_authors._id: 67a1a4278b6584b24ff98eb3
paper_authors.name: Anton van den Hengel
paper_authors._id: 67a1a4278b6584b24ff98eb4
paper_authors.name: Ehsan Abbasnejad
paper_title: RandLoRA: Full-rank parameter-efficient fine-tuning of large models
paper_summary: Low-Rank Adaptation (LoRA) and its variants have shown impressive results in
  reducing the number of trainable parameters and memory requirements of large
  transformer networks while maintaining fine-tuning performance. However, the
  low-rank nature of the weight update inherently limits the representation power
  of fine-tuned models, potentially compromising performance on complex tasks.
  This raises a critical question: when a performance gap between LoRA and
  standard fine-tuning is observed, is it due to the reduced number of trainable
  parameters or the rank deficiency? This paper aims to answer this question by
  introducing RandLoRA, a parameter-efficient method that performs full-rank
  updates using a learned linear combinations of low-rank, non-trainable random
  matrices. Our method limits the number of trainable parameters by restricting
  optimization to diagonal scaling matrices applied to the fixed random matrices.
  This allows us to effectively overcome the low-rank limitations while
  maintaining parameter and memory efficiency during training. Through extensive
  experimentation across vision, language, and vision-language benchmarks, we
  systematically evaluate the limitations of LoRA and existing random basis
  methods. Our findings reveal that full-rank updates are beneficial across
  vision and language tasks individually, and even more so for vision-language
  tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the
  performance gap between standard fine-tuning and LoRA, demonstrating its
  efficacy.
paper_ai_keywords: Low-Rank Adaptation (LoRA) trainable parameters memory requirements transformer networks fine-tuning weight update representation power RandLoRA parameter-efficient method full-rank updates learned linear combinations low-rank matrices non-trainable random matrices diagonal scaling matrices optimization fixed random matrices performance gap standard fine-tuning RandLoRA vision language vision-language benchmarks performance gap standard fine-tuning RandLoRA vision-language tasks