title: FedSVD: Adaptive Orthogonalization for Private Federated Learning with
    LoRA
summary: Low-Rank Adaptation (LoRA), which introduces a product of two trainable
  low-rank matrices into frozen pre-trained weights, is widely used for efficient
  fine-tuning of language models in federated learning (FL). However, when
  combined with differentially private stochastic gradient descent (DP-SGD), LoRA
  faces substantial noise amplification: DP-SGD perturbs per-sample gradients,
  and the matrix multiplication of the LoRA update (BA) intensifies this
  effect. Freezing one matrix (e.g., A) reduces the noise but restricts model
  expressiveness, often resulting in suboptimal adaptation. To address this, we
  propose FedSVD, a simple yet effective method that introduces a global
  reparameterization based on singular value decomposition (SVD). In our
  approach, each client optimizes only the B matrix and transmits it to the
  server. The server aggregates the B matrices, computes the product BA using
  the previous A, and refactorizes the result via SVD. This yields a new
  adaptive A composed of the orthonormal right singular vectors of BA, and an
  updated B containing the remaining SVD components. This reparameterization
  avoids quadratic noise amplification, while allowing A to better capture the
  principal directions of the aggregate updates. Moreover, the orthonormal
  structure of A bounds the gradient norms of B and preserves more signal
  under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD
  consistently improves stability and performance across a variety of privacy
  settings and benchmarks, outperforming relevant baselines under both private
  and non-private regimes.
paper_authors._id: 682bfcec8081928badd176e7
paper_authors.name: Seanie Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:22.246Z
paper_authors._id: 64ad5f59b7e4b2c1ce47eb43
paper_authors.avatarUrl: /avatars/1f13ebe21a90d8c99920aa2c8cd9ac45.svg
paper_authors.fullname: Seanie Lee
paper_authors.type: user
paper_authors.user: Seanie-lee
paper_authors._id: 682bfcec8081928badd176e8
paper_authors.name: Sangwoo Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:03:34.792Z
paper_authors._id: 638716c14e00d7fc0902fef4
paper_authors.avatarUrl: /avatars/5fa8152f8c0e4e600d1a64802c3e0103.svg
paper_authors.fullname: Sangwoo Park
paper_authors.type: user
paper_authors.user: Sangsang
paper_authors._id: 682bfcec8081928badd176e9
paper_authors.name: Dong Bok Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:31:04.999Z
paper_authors._id: 64f000769e7770db74d44bba
paper_authors.avatarUrl: /avatars/d015820380ffb823b1b35df64dcd3457.svg
paper_authors.fullname: Dong-Bok Lee
paper_authors.type: user
paper_authors.user: dongboklee
paper_authors._id: 682bfcec8081928badd176ea
paper_authors.name: Dominik Wagner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:31:33.334Z
paper_authors._id: 6311ba6f05cc08a1408d910a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
paper_authors.fullname: Dominik Wagner
paper_authors.type: user
paper_authors.user: dwgnr
paper_authors._id: 682bfcec8081928badd176eb
paper_authors.name: Haebin Seong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:31:39.425Z
paper_authors._id: 63a9379e2e05ca32e352d93b
paper_authors.avatarUrl: /avatars/6cda37befc873a92ed6d5dcba507954a.svg
paper_authors.fullname: Haebin Seong
paper_authors.type: user
paper_authors.user: hbseong
paper_authors._id: 682bfcec8081928badd176ec
paper_authors.name: Tobias Bocklet
paper_authors._id: 682bfcec8081928badd176ed
paper_authors.name: Juho Lee
paper_authors._id: 682bfcec8081928badd176ee
paper_authors.name: Sung Ju Hwang
paper_title: FedSVD: Adaptive Orthogonalization for Private Federated Learning with
    LoRA
paper_summary: Low-Rank Adaptation (LoRA), which introduces a product of two trainable
  low-rank matrices into frozen pre-trained weights, is widely used for efficient
  fine-tuning of language models in federated learning (FL). However, when
  combined with differentially private stochastic gradient descent (DP-SGD), LoRA
  faces substantial noise amplification: DP-SGD perturbs per-sample gradients,
  and the matrix multiplication of the LoRA update (BA) intensifies this
  effect. Freezing one matrix (e.g., A) reduces the noise but restricts model
  expressiveness, often resulting in suboptimal adaptation. To address this, we
  propose FedSVD, a simple yet effective method that introduces a global
  reparameterization based on singular value decomposition (SVD). In our
  approach, each client optimizes only the B matrix and transmits it to the
  server. The server aggregates the B matrices, computes the product BA using
  the previous A, and refactorizes the result via SVD. This yields a new
  adaptive A composed of the orthonormal right singular vectors of BA, and an
  updated B containing the remaining SVD components. This reparameterization
  avoids quadratic noise amplification, while allowing A to better capture the
  principal directions of the aggregate updates. Moreover, the orthonormal
  structure of A bounds the gradient norms of B and preserves more signal
  under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD
  consistently improves stability and performance across a variety of privacy
  settings and benchmarks, outperforming relevant baselines under both private
  and non-private regimes.
paper_ai_summary: FedSVD introduces a method for stable and effective fine-tuning of pre-trained language models in federated learning with differential privacy by using singular value decomposition to handle noise and maintain performance.
paper_ai_keywords: Low-Rank Adaptation LoRA federated learning FL differentially private stochastic gradient descent DP-SGD singular value decomposition SVD matrix multiplication global reparameterization orthonormal right singular vectors gradient norms