title: Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large
    Language Models
summary: Fine-tuning large language models (LLMs) on devices is attracting increasing
  interest. Recent works have fused low-rank adaptation (LoRA) techniques with
  federated fine-tuning to mitigate challenges associated with device model sizes
  and data scarcity. Still, the heterogeneity of computational resources remains
  a critical bottleneck: while higher-rank modules generally enhance performance,
  varying device capabilities constrain LoRA's feasible rank range. Existing
  approaches attempting to resolve this issue either lack analytical
  justification or impose additional computational overhead, leaving a wide gap
  for an efficient and theoretically-grounded solution. To address these
  challenges, we propose federated sketching LoRA (FSLoRA), which leverages a
  sketching mechanism to enable devices to selectively update submatrices of
  global LoRA modules maintained by the server. By adjusting the sketching
  ratios, which determine the ranks of the submatrices on the devices, FSLoRA
  flexibly adapts to device-specific communication and computational constraints.
  We provide a rigorous convergence analysis of FSLoRA that characterizes how the
  sketching ratios affect the convergence rate. Through comprehensive experiments
  on multiple datasets and LLM models, we demonstrate FSLoRA's superior
  performance compared to various baselines.
paper_authors._id: 67a2a05be5b870d51558fc00
paper_authors.name: Wenzhi Fang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-04T23:18:52.300Z
paper_authors._id: 671ff4124b2e5a664aae01e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8PQkyFF-fc9W2K8uArMXn.png
paper_authors.fullname: Wenzhi Fang
paper_authors.type: user
paper_authors.user: wenzhifang
paper_authors._id: 67a2a05be5b870d51558fc01
paper_authors.name: Dong-Jun Han
paper_authors._id: 67a2a05be5b870d51558fc02
paper_authors.name: Liangqi Yuan
paper_authors._id: 67a2a05be5b870d51558fc03
paper_authors.name: Seyyedali Hosseinalipour
paper_authors._id: 67a2a05be5b870d51558fc04
paper_authors.name: Christopher G. Brinton
paper_title: Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large
    Language Models
paper_summary: Fine-tuning large language models (LLMs) on devices is attracting increasing
  interest. Recent works have fused low-rank adaptation (LoRA) techniques with
  federated fine-tuning to mitigate challenges associated with device model sizes
  and data scarcity. Still, the heterogeneity of computational resources remains
  a critical bottleneck: while higher-rank modules generally enhance performance,
  varying device capabilities constrain LoRA's feasible rank range. Existing
  approaches attempting to resolve this issue either lack analytical
  justification or impose additional computational overhead, leaving a wide gap
  for an efficient and theoretically-grounded solution. To address these
  challenges, we propose federated sketching LoRA (FSLoRA), which leverages a
  sketching mechanism to enable devices to selectively update submatrices of
  global LoRA modules maintained by the server. By adjusting the sketching
  ratios, which determine the ranks of the submatrices on the devices, FSLoRA
  flexibly adapts to device-specific communication and computational constraints.
  We provide a rigorous convergence analysis of FSLoRA that characterizes how the
  sketching ratios affect the convergence rate. Through comprehensive experiments
  on multiple datasets and LLM models, we demonstrate FSLoRA's superior
  performance compared to various baselines.
paper_ai_summary: Federated sketching low-rank adaptation enables efficient fine-tuning of large language models on devices with varying computational resources.
paper_ai_keywords: low-rank adaptation federated fine-tuning sketching mechanism FSLoRA submatrices convergence analysis