title: Efficient Personalization of Quantized Diffusion Model without
    Backpropagation
summary: Diffusion models have shown remarkable performance in image synthesis, but
  they demand extensive computational and memory resources for training,
  fine-tuning and inference. Although advanced quantization techniques have
  successfully minimized memory usage for inference, training and fine-tuning
  these quantized models still require large memory possibly due to
  dequantization for accurate computation of gradients and/or backpropagation for
  gradient-based algorithms. However, memory-efficient fine-tuning is
  particularly desirable for applications such as personalization that often must
  be run on edge devices like mobile phones with private data. In this work, we
  address this challenge by quantizing a diffusion model with personalization via
  Textual Inversion and by leveraging a zeroth-order optimization on
  personalization tokens without dequantization so that it does not require
  gradient and activation storage for backpropagation that consumes considerable
  memory. Since a gradient estimation using zeroth-order optimization is quite
  noisy for a single or a few images in personalization, we propose to denoise
  the estimated gradient by projecting it onto a subspace that is constructed
  with the past history of the tokens, dubbed Subspace Gradient. In addition, we
  investigated the influence of text embedding in image generation, leading to
  our proposed time steps sampling, dubbed Partial Uniform Timestep Sampling for
  sampling with effective diffusion timesteps. Our method achieves comparable
  performance to prior methods in image and text alignment scores for
  personalizing Stable Diffusion with only forward passes while reducing training
  memory demand up to 8.2times.
paper_authors._id: 67db9f06842d8b6642a5eeaf
paper_authors.name: Hoigi Seo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:45:07.668Z
paper_authors._id: 633e6f07309a99325095dd42
paper_authors.avatarUrl: /avatars/57b91a488ac1745b3c0509c04eb6ad93.svg
paper_authors.fullname: Hoigi Seo
paper_authors.type: user
paper_authors.user: Agorium
paper_authors._id: 67db9f06842d8b6642a5eeb0
paper_authors.name: Wongi Jeong
paper_authors._id: 67db9f06842d8b6642a5eeb1
paper_authors.name: Kyungryeol Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:23:58.734Z
paper_authors._id: 67076244679d36bc0e3eda5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0yE8cHzBuz9QA7sE8rQkf.png
paper_authors.fullname: Kyungryeol Lee
paper_authors.type: user
paper_authors.user: hirussell
paper_authors._id: 67db9f06842d8b6642a5eeb2
paper_authors.name: Se Young Chun
paper_title: Efficient Personalization of Quantized Diffusion Model without
    Backpropagation
paper_summary: Diffusion models have shown remarkable performance in image synthesis, but
  they demand extensive computational and memory resources for training,
  fine-tuning and inference. Although advanced quantization techniques have
  successfully minimized memory usage for inference, training and fine-tuning
  these quantized models still require large memory possibly due to
  dequantization for accurate computation of gradients and/or backpropagation for
  gradient-based algorithms. However, memory-efficient fine-tuning is
  particularly desirable for applications such as personalization that often must
  be run on edge devices like mobile phones with private data. In this work, we
  address this challenge by quantizing a diffusion model with personalization via
  Textual Inversion and by leveraging a zeroth-order optimization on
  personalization tokens without dequantization so that it does not require
  gradient and activation storage for backpropagation that consumes considerable
  memory. Since a gradient estimation using zeroth-order optimization is quite
  noisy for a single or a few images in personalization, we propose to denoise
  the estimated gradient by projecting it onto a subspace that is constructed
  with the past history of the tokens, dubbed Subspace Gradient. In addition, we
  investigated the influence of text embedding in image generation, leading to
  our proposed time steps sampling, dubbed Partial Uniform Timestep Sampling for
  sampling with effective diffusion timesteps. Our method achieves comparable
  performance to prior methods in image and text alignment scores for
  personalizing Stable Diffusion with only forward passes while reducing training
  memory demand up to 8.2times.
paper_ai_summary: A diffusion model is quantized for personalization using zeroth-order optimization and Subspace Gradient denoising, reducing memory demand while maintaining image and text alignment performance with forward passes only.
paper_ai_keywords: diffusion models Textual Inversion zeroth-order optimization personalization tokens gradient estimation Subspace Gradient denoising diffusion timesteps Partial Uniform Timestep Sampling Stable Diffusion memory-efficient fine-tuning image synthesis