title: Federated Full-Parameter Tuning of Billion-Sized Language Models with
    Communication Cost under 18 Kilobytes
summary: Pre-trained large language models (LLMs) require fine-tuning to improve their
  responsiveness to natural language instructions. Federated learning (FL) offers
  a way to perform fine-tuning using the abundant data on end devices without
  compromising data privacy. Most existing federated fine-tuning methods for LLMs
  rely on parameter-efficient fine-tuning techniques, which may not reach the
  performance heights possible with full-parameter tuning. However, the
  communication overhead associated with full-parameter tuning is prohibitively
  high for both servers and clients. This work introduces FedKSeed, a novel
  approach that employs zeroth-order optimization (ZOO) with a set of random
  seeds. It enables federated full-parameter tuning of billion-sized LLMs
  directly on devices. Our method significantly reduces transmission requirements
  between the server and clients to just a few scalar gradients and random seeds,
  amounting to only a few thousand bytes. Building on this, we develop a strategy
  to assess the significance of ZOO perturbations for FL, allowing for
  probability-differentiated seed sampling. This prioritizes perturbations that
  have a greater impact on model accuracy. Experiments across six scenarios with
  different LLMs, datasets and data partitions demonstrate that our approach
  outperforms existing federated LLM fine-tuning methods in terms of both
  communication efficiency and new task generalization.
paper_authors._id: 6577f13432150025aca5762b
paper_authors.name: Zhen Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:57:15.132Z
paper_authors._id: 649014b91d71e55664838d2d
paper_authors.avatarUrl: /avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: zhenqincn
paper_authors._id: 6577f13432150025aca5762c
paper_authors.name: Daoyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:57:31.994Z
paper_authors._id: 6530ca70c1b8f265a431fc42
paper_authors.avatarUrl: /avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
paper_authors.fullname: Daoyuan Chen
paper_authors.type: user
paper_authors.user: yxdyc
paper_authors._id: 6577f13432150025aca5762d
paper_authors.name: Bingchen Qian
paper_authors._id: 6577f13432150025aca5762e
paper_authors.name: Bolin Ding
paper_authors._id: 6577f13432150025aca5762f
paper_authors.name: Yaliang Li
paper_authors._id: 6577f13432150025aca57630
paper_authors.name: Shuiguang Deng
paper_title: Federated Full-Parameter Tuning of Billion-Sized Language Models with
    Communication Cost under 18 Kilobytes
paper_summary: Pre-trained large language models (LLMs) require fine-tuning to improve their
  responsiveness to natural language instructions. Federated learning (FL) offers
  a way to perform fine-tuning using the abundant data on end devices without
  compromising data privacy. Most existing federated fine-tuning methods for LLMs
  rely on parameter-efficient fine-tuning techniques, which may not reach the
  performance heights possible with full-parameter tuning. However, the
  communication overhead associated with full-parameter tuning is prohibitively
  high for both servers and clients. This work introduces FedKSeed, a novel
  approach that employs zeroth-order optimization (ZOO) with a set of random
  seeds. It enables federated full-parameter tuning of billion-sized LLMs
  directly on devices. Our method significantly reduces transmission requirements
  between the server and clients to just a few scalar gradients and random seeds,
  amounting to only a few thousand bytes. Building on this, we develop a strategy
  to assess the significance of ZOO perturbations for FL, allowing for
  probability-differentiated seed sampling. This prioritizes perturbations that
  have a greater impact on model accuracy. Experiments across six scenarios with
  different LLMs, datasets and data partitions demonstrate that our approach
  outperforms existing federated LLM fine-tuning methods in terms of both
  communication efficiency and new task generalization.
paper_ai_summary: FedKSeed uses zeroth-order optimization with random seeds to enable efficient, full-parameter federated fine-tuning of large language models on client devices.
paper_ai_keywords: pre-trained LLMs federated learning parameter-efficient fine-tuning zeroth-order optimization FedKSeed scalar gradients new task generalization