title: Ferret: Federated Full-Parameter Tuning at Scale for Large Language
    Models
summary: Large Language Models (LLMs) have become indispensable in numerous real-world
  applications. Unfortunately, fine-tuning these models at scale, especially in
  federated settings where data privacy and communication efficiency are
  critical, presents significant challenges. Existing methods often resort to
  parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
  this typically comes at the cost of model accuracy. To address these
  limitations, we propose federated full-parameter tuning at scale for LLMs
  (Ferret), the first first-order method with shared randomness to enable
  scalable full-parameter tuning of LLMs across decentralized data sources while
  maintaining competitive model accuracy. Ferret accomplishes this through three
  aspects: (1) it employs widely applied first-order methods for efficient local
  updates; (2) it projects these updates into a low-dimensional space to
  considerably reduce communication overhead; and (3) it reconstructs local
  updates from this low-dimensional space with shared randomness to facilitate
  effective full-parameter global aggregation, ensuring fast convergence and
  competitive final performance. Our rigorous theoretical analyses and insights
  along with extensive experiments, show that Ferret significantly enhances the
  scalability of existing federated full-parameter tuning approaches by achieving
  high computational efficiency, reduced communication overhead, and fast
  convergence, all while maintaining competitive model accuracy. Our
  implementation is available at https://github.com/allen4747/Ferret.
paper_authors._id: 66e4dfaa5c06b7719c362746
paper_authors.name: Yao Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:02:53.062Z
paper_authors._id: 66123816d7dfcea8ae55a751
paper_authors.avatarUrl: /avatars/3f24468b63e4babd7d9a0c926ca01b23.svg
paper_authors.fullname: Shu Yao
paper_authors.type: user
paper_authors.user: ZCODE0
paper_authors._id: 66e4dfaa5c06b7719c362747
paper_authors.name: Wenyang Hu
paper_authors._id: 66e4dfaa5c06b7719c362748
paper_authors.name: See-Kiong Ng
paper_authors._id: 66e4dfaa5c06b7719c362749
paper_authors.name: Bryan Kian Hsiang Low
paper_authors._id: 66e4dfaa5c06b7719c36274a
paper_authors.name: Fei Richard Yu
paper_title: Ferret: Federated Full-Parameter Tuning at Scale for Large Language
    Models
paper_summary: Large Language Models (LLMs) have become indispensable in numerous real-world
  applications. Unfortunately, fine-tuning these models at scale, especially in
  federated settings where data privacy and communication efficiency are
  critical, presents significant challenges. Existing methods often resort to
  parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
  this typically comes at the cost of model accuracy. To address these
  limitations, we propose federated full-parameter tuning at scale for LLMs
  (Ferret), the first first-order method with shared randomness to enable
  scalable full-parameter tuning of LLMs across decentralized data sources while
  maintaining competitive model accuracy. Ferret accomplishes this through three
  aspects: (1) it employs widely applied first-order methods for efficient local
  updates; (2) it projects these updates into a low-dimensional space to
  considerably reduce communication overhead; and (3) it reconstructs local
  updates from this low-dimensional space with shared randomness to facilitate
  effective full-parameter global aggregation, ensuring fast convergence and
  competitive final performance. Our rigorous theoretical analyses and insights
  along with extensive experiments, show that Ferret significantly enhances the
  scalability of existing federated full-parameter tuning approaches by achieving
  high computational efficiency, reduced communication overhead, and fast
  convergence, all while maintaining competitive model accuracy. Our
  implementation is available at https://github.com/allen4747/Ferret.
paper_ai_keywords: parameter-efficient fine-tuning (PEFT) federated full-parameter tuning first-order methods low-dimensional space shared randomness full-parameter global aggregation computational efficiency communication overhead fast convergence competitive model accuracy