title: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
summary: We present Generalized LoRA (GLoRA), an advanced approach for universal
  parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
  GLoRA employs a generalized prompt module to optimize pre-trained model weights
  and adjust intermediate activations, providing more flexibility and capability
  across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
  parameter adaptation by employing a scalable, modular, layer-wise structure
  search that learns individual adapter of each layer. Originating from a unified
  mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
  learning and domain generalization abilities, as it adjusts to new tasks
  through additional dimensions on weights and activations. Comprehensive
  experiments demonstrate that GLoRA outperforms all previous methods in natural,
  specialized, and structured benchmarks, achieving superior accuracy with fewer
  parameters and computations on various datasets. Furthermore, our structural
  re-parameterization design ensures that GLoRA incurs no extra inference cost,
  rendering it a practical solution for resource-limited applications. Code is
  available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.
paper_authors._id: 648938c3de81ed071f971cf2
paper_authors.name: Arnav Chavan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T09:29:11.295Z
paper_authors._id: 648979dd308f19bc7c42899f
paper_authors.avatarUrl: /avatars/6b7bde1d453f0328fcd5a86cfd9eb3b2.svg
paper_authors.fullname: Arnav Chavan
paper_authors.type: user
paper_authors.user: Arnav0400
paper_authors._id: 648938c3de81ed071f971cf3
paper_authors.name: Zhuang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T19:38:50.005Z
paper_authors._id: 647d3a9383cdb93baf76dce2
paper_authors.avatarUrl: /avatars/56efca912a66ef9d4f2f3679aac31bae.svg
paper_authors.fullname: Zhuang Liu
paper_authors.type: user
paper_authors.user: liuzhuang13
paper_authors._id: 648938c3de81ed071f971cf4
paper_authors.name: Deepak Gupta
paper_authors._id: 648938c3de81ed071f971cf5
paper_authors.name: Eric Xing
paper_authors._id: 648938c3de81ed071f971cf6
paper_authors.name: Zhiqiang Shen
paper_title: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
paper_summary: We present Generalized LoRA (GLoRA), an advanced approach for universal
  parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
  GLoRA employs a generalized prompt module to optimize pre-trained model weights
  and adjust intermediate activations, providing more flexibility and capability
  across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
  parameter adaptation by employing a scalable, modular, layer-wise structure
  search that learns individual adapter of each layer. Originating from a unified
  mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
  learning and domain generalization abilities, as it adjusts to new tasks
  through additional dimensions on weights and activations. Comprehensive
  experiments demonstrate that GLoRA outperforms all previous methods in natural,
  specialized, and structured benchmarks, achieving superior accuracy with fewer
  parameters and computations on various datasets. Furthermore, our structural
  re-parameterization design ensures that GLoRA incurs no extra inference cost,
  rendering it a practical solution for resource-limited applications. Code is
  available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.
paper_ai_keywords: Generalized LoRA (GLoRA) Low-Rank Adaptation (LoRA) pre-efficient fine-tuning prompt module pre-trained model weights intermediate activations layer-wise structure scalable modular adapter unified mathematical formulation transfer learning few-shot learning domain generalization re-parameterization design inference cost