Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
As the size of large language models grows exponentially, GPU memory has
become a bottleneck for adapting these models to downstream tasks. In this
paper, we aim to push the limits of memory-efficient training by minimizing
memory usage on model weights, gradients, and optimizer states, within a
unified framework. Our idea is to eliminate both gradients and optimizer states
using zeroth-order optimization, which approximates gradients by perturbing
weights during forward passes to identify gradient directions. To minimize
memory usage on weights, we employ model quantization, e.g., converting from
bfloat16 to int4. However, directly applying zeroth-order optimization to
quantized weights is infeasible due to the precision gap between discrete
weights and continuous gradients, which would otherwise require de-quantization
and re-quantization. To overcome this challenge, we propose Quantized
Zeroth-order Optimization (QZO), a novel approach that perturbs the continuous
quantization scale for gradient estimation and uses a directional derivative
clipping method to stabilize training. QZO is orthogonal to both scalar-based
and codebook-based post-training quantization methods. Compared to
full-parameter fine-tuning in bfloat16, QZO can reduce the total memory cost by
more than 18times for 4-bit LLMs, and enables fine-tuning Llama-2-13B and
Stable Diffusion 3.5 Large within a single 24GB GPU.
682d70c598e3ea9be315ed85
Sifeng Shang
admin_assigned
2025-05-21T10:38:00.611Z
66fd024b2f3b4b95701b3055
/avatars/7efa17e0ba1b8e2f5afabeba59ef7a6f.svg
Sifeng SHANG
user
sifengshang
682d70c598e3ea9be315ed86
Jiayi Zhou
682d70c598e3ea9be315ed87
Chenyu Lin
admin_assigned
2025-05-21T10:38:15.674Z
644b40a9a49c03b1b68b7e48
/avatars/0fbe390e34a45441905f2c55c17ca1ea.svg
lin
user
chenyulin
682d70c598e3ea9be315ed88
Minxian Li
682d70c598e3ea9be315ed89
Kaiyang Zhou
admin_assigned
2025-05-21T10:38:21.725Z
62ac6656de8bfbb93094b8fd
/avatars/1d8f445b6d5f9d43ddab81056bcb141e.svg
Kaiyang Zhou
user
kaiyangzhou
Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
As the size of large language models grows exponentially, GPU memory has
become a bottleneck for adapting these models to downstream tasks. In this
paper, we aim to push the limits of memory-efficient training by minimizing
memory usage on model weights, gradients, and optimizer states, within a
unified framework. Our idea is to eliminate both gradients and optimizer states
using zeroth-order optimization, which approximates gradients by perturbing
weights during forward passes to identify gradient directions. To minimize
memory usage on weights, we employ model quantization, e.g., converting from
bfloat16 to int4. However, directly applying zeroth-order optimization to
quantized weights is infeasible due to the precision gap between discrete
weights and continuous gradients, which would otherwise require de-quantization
and re-quantization. To overcome this challenge, we propose Quantized
Zeroth-order Optimization (QZO), a novel approach that perturbs the continuous
quantization scale for gradient estimation and uses a directional derivative
clipping method to stabilize training. QZO is orthogonal to both scalar-based
and codebook-based post-training quantization methods. Compared to
full-parameter fine-tuning in bfloat16, QZO can reduce the total memory cost by
more than 18times for 4-bit LLMs, and enables fine-tuning Llama-2-13B and
Stable Diffusion 3.5 Large within a single 24GB GPU.
Quantized Zeroth-order Optimization (QZO) enables memory-efficient fine-tuning of large language models by minimizing memory usage on model weights, gradients, and optimizer states.
zeroth-order optimization model quantization bfloat16 int4 Quantized Zeroth-order Optimization QZO directional derivative clipping full-parameter fine-tuning Llama-2-13B Stable Diffusion 3.5 Large