Ferret: Federated Full-Parameter Tuning at Scale for Large Language
  Models
Large Language Models (LLMs) have become indispensable in numerous real-world
applications. Unfortunately, fine-tuning these models at scale, especially in
federated settings where data privacy and communication efficiency are
critical, presents significant challenges. Existing methods often resort to
parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
this typically comes at the cost of model accuracy. To address these
limitations, we propose federated full-parameter tuning at scale for LLMs
(Ferret), the first first-order method with shared randomness to enable
scalable full-parameter tuning of LLMs across decentralized data sources while
maintaining competitive model accuracy. Ferret accomplishes this through three
aspects: (1) it employs widely applied first-order methods for efficient local
updates; (2) it projects these updates into a low-dimensional space to
considerably reduce communication overhead; and (3) it reconstructs local
updates from this low-dimensional space with shared randomness to facilitate
effective full-parameter global aggregation, ensuring fast convergence and
competitive final performance. Our rigorous theoretical analyses and insights
along with extensive experiments, show that Ferret significantly enhances the
scalability of existing federated full-parameter tuning approaches by achieving
high computational efficiency, reduced communication overhead, and fast
convergence, all while maintaining competitive model accuracy. Our
implementation is available at https://github.com/allen4747/Ferret.
66e4dfaa5c06b7719c362746
Yao Shu
claimed_verified
2024-09-16T07:02:53.062Z
66123816d7dfcea8ae55a751
/avatars/3f24468b63e4babd7d9a0c926ca01b23.svg
Shu Yao
user
ZCODE0
66e4dfaa5c06b7719c362747
Wenyang Hu
66e4dfaa5c06b7719c362748
See-Kiong Ng
66e4dfaa5c06b7719c362749
Bryan Kian Hsiang Low
66e4dfaa5c06b7719c36274a
Fei Richard Yu
Ferret: Federated Full-Parameter Tuning at Scale for Large Language
  Models
Large Language Models (LLMs) have become indispensable in numerous real-world
applications. Unfortunately, fine-tuning these models at scale, especially in
federated settings where data privacy and communication efficiency are
critical, presents significant challenges. Existing methods often resort to
parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
this typically comes at the cost of model accuracy. To address these
limitations, we propose federated full-parameter tuning at scale for LLMs
(Ferret), the first first-order method with shared randomness to enable
scalable full-parameter tuning of LLMs across decentralized data sources while
maintaining competitive model accuracy. Ferret accomplishes this through three
aspects: (1) it employs widely applied first-order methods for efficient local
updates; (2) it projects these updates into a low-dimensional space to
considerably reduce communication overhead; and (3) it reconstructs local
updates from this low-dimensional space with shared randomness to facilitate
effective full-parameter global aggregation, ensuring fast convergence and
competitive final performance. Our rigorous theoretical analyses and insights
along with extensive experiments, show that Ferret significantly enhances the
scalability of existing federated full-parameter tuning approaches by achieving
high computational efficiency, reduced communication overhead, and fast
convergence, all while maintaining competitive model accuracy. Our
implementation is available at https://github.com/allen4747/Ferret.
Ferret is a federated full-parameter tuning method for LLMs that achieves high efficiency, reduced communication overhead, and competitive accuracy using first-order methods and shared randomness.
Large Language Models (LLMs) federated settings parameter-efficient fine-tuning (PEFT) federated full-parameter tuning Ferret first-order methods low-dimensional space shared randomness full-parameter global aggregation computational efficiency communication overhead