title: Selective Aggregation for Low-Rank Adaptation in Federated Learning
summary: We investigate LoRA in federated learning through the lens of the asymmetry
  analysis of the learned A and B matrices. In doing so, we uncover that A
  matrices are responsible for learning general knowledge, while B matrices
  focus on capturing client-specific knowledge. Based on this finding, we
  introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two
  low-rank trainable matrices A and B to model the weight update, but only
  A matrices are shared with the server for aggregation. Moreover, we delve
  into the relationship between the learned A and B matrices in other LoRA
  variants, such as rsLoRA and VeRA, revealing a consistent pattern.
  Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting
  in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm
  for integrating LoRA with FL, offering guidance for future work on subsequent
  LoRA variants combined with FL. Extensive experimental results on natural
  language understanding and generation tasks demonstrate the effectiveness of
  the proposed method.
paper_authors._id: 66fdfde460a069010d11c007
paper_authors.name: Pengxin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T08:31:28.860Z
paper_authors._id: 668f440894dfc0ed1a7006ed
paper_authors.avatarUrl: /avatars/fa0d328300b03bcbbf9b3a7532f28458.svg
paper_authors.fullname: Pengxin Guo
paper_authors.type: user
paper_authors.user: gpx333
paper_authors._id: 66fdfde460a069010d11c008
paper_authors.name: Shuang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:31:15.344Z
paper_authors._id: 66712ff09c609c2484ce4aa0
paper_authors.avatarUrl: /avatars/717b96ddef8a4c19ce07ea1fd9e9fd66.svg
paper_authors.fullname: Shuang Zeng
paper_authors.type: user
paper_authors.user: stevezs
paper_authors._id: 66fdfde460a069010d11c009
paper_authors.name: Yanran Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:31:22.188Z
paper_authors._id: 656f698c80ff527c44e3c33b
paper_authors.avatarUrl: /avatars/19ea552ed0bb36260ab0f6e41421f9b3.svg
paper_authors.fullname: Yanran Wang
paper_authors.type: user
paper_authors.user: yanranw1
paper_authors._id: 66fdfde460a069010d11c00a
paper_authors.name: Huijie Fan
paper_authors._id: 66fdfde460a069010d11c00b
paper_authors.name: Feifei Wang
paper_authors._id: 66fdfde460a069010d11c00c
paper_authors.name: Liangqiong Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:32:14.110Z
paper_authors._id: 663058bc2653ec94f4a6235f
paper_authors.avatarUrl: /avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg
paper_authors.fullname: Liangqiong Qu
paper_authors.type: user
paper_authors.user: Liangqiong-QU
paper_title: Selective Aggregation for Low-Rank Adaptation in Federated Learning
paper_summary: We investigate LoRA in federated learning through the lens of the asymmetry
  analysis of the learned A and B matrices. In doing so, we uncover that A
  matrices are responsible for learning general knowledge, while B matrices
  focus on capturing client-specific knowledge. Based on this finding, we
  introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two
  low-rank trainable matrices A and B to model the weight update, but only
  A matrices are shared with the server for aggregation. Moreover, we delve
  into the relationship between the learned A and B matrices in other LoRA
  variants, such as rsLoRA and VeRA, revealing a consistent pattern.
  Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting
  in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm
  for integrating LoRA with FL, offering guidance for future work on subsequent
  LoRA variants combined with FL. Extensive experimental results on natural
  language understanding and generation tasks demonstrate the effectiveness of
  the proposed method.
paper_ai_keywords: LoRA federated learning asymmetry analysis $A$ matrices $B$ matrices general knowledge client-specific knowledge Federated Share-A Low-Rank Adaptation (FedSA-LoRA) low-rank trainable matrices weight update rsLoRA VeRA FL natural language understanding natural language generation