Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large
  Language Models
Fine-tuning large language models (LLMs) on devices is attracting increasing
interest. Recent works have fused low-rank adaptation (LoRA) techniques with
federated fine-tuning to mitigate challenges associated with device model sizes
and data scarcity. Still, the heterogeneity of computational resources remains
a critical bottleneck: while higher-rank modules generally enhance performance,
varying device capabilities constrain LoRA's feasible rank range. Existing
approaches attempting to resolve this issue either lack analytical
justification or impose additional computational overhead, leaving a wide gap
for an efficient and theoretically-grounded solution. To address these
challenges, we propose federated sketching LoRA (FSLoRA), which leverages a
sketching mechanism to enable devices to selectively update submatrices of
global LoRA modules maintained by the server. By adjusting the sketching
ratios, which determine the ranks of the submatrices on the devices, FSLoRA
flexibly adapts to device-specific communication and computational constraints.
We provide a rigorous convergence analysis of FSLoRA that characterizes how the
sketching ratios affect the convergence rate. Through comprehensive experiments
on multiple datasets and LLM models, we demonstrate FSLoRA's superior
performance compared to various baselines.
67a2a05be5b870d51558fc00
Wenzhi Fang
extracted_pending
2025-02-04T23:18:52.300Z
671ff4124b2e5a664aae01e1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8PQkyFF-fc9W2K8uArMXn.png
Wenzhi Fang
user
wenzhifang
67a2a05be5b870d51558fc01
Dong-Jun Han
67a2a05be5b870d51558fc02
Liangqi Yuan
67a2a05be5b870d51558fc03
Seyyedali Hosseinalipour
67a2a05be5b870d51558fc04
Christopher G. Brinton
Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large
  Language Models
Fine-tuning large language models (LLMs) on devices is attracting increasing
interest. Recent works have fused low-rank adaptation (LoRA) techniques with
federated fine-tuning to mitigate challenges associated with device model sizes
and data scarcity. Still, the heterogeneity of computational resources remains
a critical bottleneck: while higher-rank modules generally enhance performance,
varying device capabilities constrain LoRA's feasible rank range. Existing
approaches attempting to resolve this issue either lack analytical
justification or impose additional computational overhead, leaving a wide gap
for an efficient and theoretically-grounded solution. To address these
challenges, we propose federated sketching LoRA (FSLoRA), which leverages a
sketching mechanism to enable devices to selectively update submatrices of
global LoRA modules maintained by the server. By adjusting the sketching
ratios, which determine the ranks of the submatrices on the devices, FSLoRA
flexibly adapts to device-specific communication and computational constraints.
We provide a rigorous convergence analysis of FSLoRA that characterizes how the
sketching ratios affect the convergence rate. Through comprehensive experiments
on multiple datasets and LLM models, we demonstrate FSLoRA's superior
performance compared to various baselines.
Federated sketching low-rank adaptation enables efficient fine-tuning of large language models on devices with varying computational resources.
low-rank adaptation federated fine-tuning sketching mechanism FSLoRA submatrices convergence analysis