title: Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
summary: As the size of large language models grows exponentially, GPU memory has
  become a bottleneck for adapting these models to downstream tasks. In this
  paper, we aim to push the limits of memory-efficient training by minimizing
  memory usage on model weights, gradients, and optimizer states, within a
  unified framework. Our idea is to eliminate both gradients and optimizer states
  using zeroth-order optimization, which approximates gradients by perturbing
  weights during forward passes to identify gradient directions. To minimize
  memory usage on weights, we employ model quantization, e.g., converting from
  bfloat16 to int4. However, directly applying zeroth-order optimization to
  quantized weights is infeasible due to the precision gap between discrete
  weights and continuous gradients, which would otherwise require de-quantization
  and re-quantization. To overcome this challenge, we propose Quantized
  Zeroth-order Optimization (QZO), a novel approach that perturbs the continuous
  quantization scale for gradient estimation and uses a directional derivative
  clipping method to stabilize training. QZO is orthogonal to both scalar-based
  and codebook-based post-training quantization methods. Compared to
  full-parameter fine-tuning in bfloat16, QZO can reduce the total memory cost by
  more than 18times for 4-bit LLMs, and enables fine-tuning Llama-2-13B and
  Stable Diffusion 3.5 Large within a single 24GB GPU.
paper_authors._id: 682d70c598e3ea9be315ed85
paper_authors.name: Sifeng Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:38:00.611Z
paper_authors._id: 66fd024b2f3b4b95701b3055
paper_authors.avatarUrl: /avatars/7efa17e0ba1b8e2f5afabeba59ef7a6f.svg
paper_authors.fullname: Sifeng SHANG
paper_authors.type: user
paper_authors.user: sifengshang
paper_authors._id: 682d70c598e3ea9be315ed86
paper_authors.name: Jiayi Zhou
paper_authors._id: 682d70c598e3ea9be315ed87
paper_authors.name: Chenyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:38:15.674Z
paper_authors._id: 644b40a9a49c03b1b68b7e48
paper_authors.avatarUrl: /avatars/0fbe390e34a45441905f2c55c17ca1ea.svg
paper_authors.fullname: lin
paper_authors.type: user
paper_authors.user: chenyulin
paper_authors._id: 682d70c598e3ea9be315ed88
paper_authors.name: Minxian Li
paper_authors._id: 682d70c598e3ea9be315ed89
paper_authors.name: Kaiyang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:38:21.725Z
paper_authors._id: 62ac6656de8bfbb93094b8fd
paper_authors.avatarUrl: /avatars/1d8f445b6d5f9d43ddab81056bcb141e.svg
paper_authors.fullname: Kaiyang Zhou
paper_authors.type: user
paper_authors.user: kaiyangzhou
paper_title: Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
paper_summary: As the size of large language models grows exponentially, GPU memory has
  become a bottleneck for adapting these models to downstream tasks. In this
  paper, we aim to push the limits of memory-efficient training by minimizing
  memory usage on model weights, gradients, and optimizer states, within a
  unified framework. Our idea is to eliminate both gradients and optimizer states
  using zeroth-order optimization, which approximates gradients by perturbing
  weights during forward passes to identify gradient directions. To minimize
  memory usage on weights, we employ model quantization, e.g., converting from
  bfloat16 to int4. However, directly applying zeroth-order optimization to
  quantized weights is infeasible due to the precision gap between discrete
  weights and continuous gradients, which would otherwise require de-quantization
  and re-quantization. To overcome this challenge, we propose Quantized
  Zeroth-order Optimization (QZO), a novel approach that perturbs the continuous
  quantization scale for gradient estimation and uses a directional derivative
  clipping method to stabilize training. QZO is orthogonal to both scalar-based
  and codebook-based post-training quantization methods. Compared to
  full-parameter fine-tuning in bfloat16, QZO can reduce the total memory cost by
  more than 18times for 4-bit LLMs, and enables fine-tuning Llama-2-13B and
  Stable Diffusion 3.5 Large within a single 24GB GPU.
paper_ai_summary: Quantized Zeroth-order Optimization (QZO) enables memory-efficient fine-tuning of large language models by minimizing memory usage on model weights, gradients, and optimizer states.
paper_ai_keywords: zeroth-order optimization model quantization bfloat16 int4 Quantized Zeroth-order Optimization QZO directional derivative clipping full-parameter fine-tuning Llama-2-13B Stable Diffusion 3.5 Large