Efficient Personalization of Quantized Diffusion Model without
  Backpropagation
Diffusion models have shown remarkable performance in image synthesis, but
they demand extensive computational and memory resources for training,
fine-tuning and inference. Although advanced quantization techniques have
successfully minimized memory usage for inference, training and fine-tuning
these quantized models still require large memory possibly due to
dequantization for accurate computation of gradients and/or backpropagation for
gradient-based algorithms. However, memory-efficient fine-tuning is
particularly desirable for applications such as personalization that often must
be run on edge devices like mobile phones with private data. In this work, we
address this challenge by quantizing a diffusion model with personalization via
Textual Inversion and by leveraging a zeroth-order optimization on
personalization tokens without dequantization so that it does not require
gradient and activation storage for backpropagation that consumes considerable
memory. Since a gradient estimation using zeroth-order optimization is quite
noisy for a single or a few images in personalization, we propose to denoise
the estimated gradient by projecting it onto a subspace that is constructed
with the past history of the tokens, dubbed Subspace Gradient. In addition, we
investigated the influence of text embedding in image generation, leading to
our proposed time steps sampling, dubbed Partial Uniform Timestep Sampling for
sampling with effective diffusion timesteps. Our method achieves comparable
performance to prior methods in image and text alignment scores for
personalizing Stable Diffusion with only forward passes while reducing training
memory demand up to 8.2times.
67db9f06842d8b6642a5eeaf
Hoigi Seo
claimed_verified
2025-03-20T10:45:07.668Z
633e6f07309a99325095dd42
/avatars/57b91a488ac1745b3c0509c04eb6ad93.svg
Hoigi Seo
user
Agorium
67db9f06842d8b6642a5eeb0
Wongi Jeong
claimed_verified
2025-07-23T08:38:17.090Z
63a1684281173de5e4caf63b
/avatars/68ac507ffa3ff7cb6899b8c8d002c01f.svg
wongi
user
ignow
67db9f06842d8b6642a5eeb1
Kyungryeol Lee
admin_assigned
2025-03-20T13:23:58.734Z
67076244679d36bc0e3eda5b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0yE8cHzBuz9QA7sE8rQkf.png
Kyungryeol Lee
user
hirussell
67db9f06842d8b6642a5eeb2
Se Young Chun
Efficient Personalization of Quantized Diffusion Model without
  Backpropagation
Diffusion models have shown remarkable performance in image synthesis, but
they demand extensive computational and memory resources for training,
fine-tuning and inference. Although advanced quantization techniques have
successfully minimized memory usage for inference, training and fine-tuning
these quantized models still require large memory possibly due to
dequantization for accurate computation of gradients and/or backpropagation for
gradient-based algorithms. However, memory-efficient fine-tuning is
particularly desirable for applications such as personalization that often must
be run on edge devices like mobile phones with private data. In this work, we
address this challenge by quantizing a diffusion model with personalization via
Textual Inversion and by leveraging a zeroth-order optimization on
personalization tokens without dequantization so that it does not require
gradient and activation storage for backpropagation that consumes considerable
memory. Since a gradient estimation using zeroth-order optimization is quite
noisy for a single or a few images in personalization, we propose to denoise
the estimated gradient by projecting it onto a subspace that is constructed
with the past history of the tokens, dubbed Subspace Gradient. In addition, we
investigated the influence of text embedding in image generation, leading to
our proposed time steps sampling, dubbed Partial Uniform Timestep Sampling for
sampling with effective diffusion timesteps. Our method achieves comparable
performance to prior methods in image and text alignment scores for
personalizing Stable Diffusion with only forward passes while reducing training
memory demand up to 8.2times.
A diffusion model is quantized for personalization using zeroth-order optimization and Subspace Gradient denoising, reducing memory demand while maintaining image and text alignment performance with forward passes only.
diffusion models Textual Inversion zeroth-order optimization personalization tokens gradient estimation Subspace Gradient denoising diffusion timesteps Partial Uniform Timestep Sampling Stable Diffusion memory-efficient fine-tuning image synthesis