title: Fine-Tuning Language Models with Just Forward Passes
summary: Fine-tuning language models (LMs) has yielded success on diverse downstream
  tasks, but as LMs grow in size, backpropagation requires a prohibitively large
  amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients
  using only two forward passes but are theorized to be catastrophically slow for
  optimizing large models. In this work, we propose a memory-efficient
  zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate
  in-place, thereby fine-tuning LMs with the same memory footprint as inference.
  For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter
  model, whereas fine-tuning with backpropagation can train only a 2.7B LM with
  the same budget. We conduct comprehensive experiments across model types
  (masked and autoregressive LMs), model scales (up to 66B), and downstream tasks
  (classification, multiple-choice, and generation). Our results demonstrate that
  (1) MeZO significantly outperforms in-context learning and linear probing; (2)
  MeZO achieves comparable performance to fine-tuning with backpropagation across
  multiple tasks, with up to 12x memory reduction; (3) MeZO is compatible with
  both full-parameter and parameter-efficient tuning techniques such as LoRA and
  prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives
  (e.g., maximizing accuracy or F1). We support our empirical findings with
  theoretical insights, highlighting how adequate pre-training and task prompts
  enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting
  otherwise.
paper_authors._id: 64758d9433192631bad32b3e
paper_authors.name: Sadhika Malladi
paper_authors._id: 64758d9433192631bad32b3f
paper_authors.name: Tianyu Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:58:01.706Z
paper_authors._id: 607eadfc048fe4c6ab48b6ee
paper_authors.avatarUrl: /avatars/7064f3b5e50bc58d32cfb58d0dd28a79.svg
paper_authors.fullname: Tianyu Gao
paper_authors.type: user
paper_authors.user: gaotianyu1350
paper_authors._id: 64758d9433192631bad32b40
paper_authors.name: Eshaan Nichani
paper_authors._id: 64758d9433192631bad32b41
paper_authors.name: Alex Damian
paper_authors.statusLastChangedAt: 2023-05-31T08:59:57.567Z
paper_authors._id: 64758d9433192631bad32b42
paper_authors.name: Jason D. Lee
paper_authors._id: 64758d9433192631bad32b43
paper_authors.name: Danqi Chen
paper_authors._id: 64758d9433192631bad32b44
paper_authors.name: Sanjeev Arora
paper_title: Fine-Tuning Language Models with Just Forward Passes
paper_summary: Fine-tuning language models (LMs) has yielded success on diverse downstream
  tasks, but as LMs grow in size, backpropagation requires a prohibitively large
  amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients
  using only two forward passes but are theorized to be catastrophically slow for
  optimizing large models. In this work, we propose a memory-efficient
  zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate
  in-place, thereby fine-tuning LMs with the same memory footprint as inference.
  For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter
  model, whereas fine-tuning with backpropagation can train only a 2.7B LM with
  the same budget. We conduct comprehensive experiments across model types
  (masked and autoregressive LMs), model scales (up to 66B), and downstream tasks
  (classification, multiple-choice, and generation). Our results demonstrate that
  (1) MeZO significantly outperforms in-context learning and linear probing; (2)
  MeZO achieves comparable performance to fine-tuning with backpropagation across
  multiple tasks, with up to 12x memory reduction; (3) MeZO is compatible with
  both full-parameter and parameter-efficient tuning techniques such as LoRA and
  prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives
  (e.g., maximizing accuracy or F1). We support our empirical findings with
  theoretical insights, highlighting how adequate pre-training and task prompts
  enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting
  otherwise.
paper_ai_keywords: zeroth-order (ZO) methods ZO-SGD memory-efficient in-place inference backpropagation fine-tuning language models (LMs) masked LMs autoregressive LMs in-context learning linear probing LoRA prefix tuning non-differentiable objectives pre-training task prompts