title: Efficient Distillation of Classifier-Free Guidance using Adapters
summary: While classifier-free guidance (CFG) is essential for conditional diffusion
  models, it doubles the number of neural function evaluations (NFEs) per
  inference step. To mitigate this inefficiency, we introduce adapter guidance
  distillation (AGD), a novel approach that simulates CFG in a single forward
  pass. AGD leverages lightweight adapters to approximate CFG, effectively
  doubling the sampling speed while maintaining or even improving sample quality.
  Unlike prior guidance distillation methods that tune the entire model, AGD
  keeps the base model frozen and only trains minimal additional parameters
  (sim2%) to significantly reduce the resource requirement of the distillation
  phase. Additionally, this approach preserves the original model weights and
  enables the adapters to be seamlessly combined with other checkpoints derived
  from the same base model. We also address a key mismatch between training and
  inference in existing guidance distillation methods by training on CFG-guided
  trajectories instead of standard diffusion trajectories. Through extensive
  experiments, we show that AGD achieves comparable or superior FID to CFG across
  multiple architectures with only half the NFEs. Notably, our method enables the
  distillation of large models (sim2.6B parameters) on a single consumer GPU
  with 24 GB of VRAM, making it more accessible than previous approaches that
  require multiple high-end GPUs. We will publicly release the implementation of
  our method.
paper_authors._id: 67d010ecdf1cf0ec0b961fd1
paper_authors.name: Cristian Perez Jensen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T10:57:59.848Z
paper_authors._id: 65e1863062dc883757896f98
paper_authors.avatarUrl: /avatars/125c56f4c605a09485e0b17e76897bdf.svg
paper_authors.fullname: Cristian Perez Jensen
paper_authors.type: user
paper_authors.user: cristianpjensen
paper_authors._id: 67d010ecdf1cf0ec0b961fd2
paper_authors.name: Seyedmorteza Sadat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T10:37:28.600Z
paper_authors._id: 63b4b02a103617b0a5b0ee2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b4b02a103617b0a5b0ee2e/LasBC-pCnPJ6XuCt6qqcp.jpeg
paper_authors.fullname: Seyedmorteza Sadat
paper_authors.type: user
paper_authors.user: msadat97
paper_title: Efficient Distillation of Classifier-Free Guidance using Adapters
paper_summary: While classifier-free guidance (CFG) is essential for conditional diffusion
  models, it doubles the number of neural function evaluations (NFEs) per
  inference step. To mitigate this inefficiency, we introduce adapter guidance
  distillation (AGD), a novel approach that simulates CFG in a single forward
  pass. AGD leverages lightweight adapters to approximate CFG, effectively
  doubling the sampling speed while maintaining or even improving sample quality.
  Unlike prior guidance distillation methods that tune the entire model, AGD
  keeps the base model frozen and only trains minimal additional parameters
  (sim2%) to significantly reduce the resource requirement of the distillation
  phase. Additionally, this approach preserves the original model weights and
  enables the adapters to be seamlessly combined with other checkpoints derived
  from the same base model. We also address a key mismatch between training and
  inference in existing guidance distillation methods by training on CFG-guided
  trajectories instead of standard diffusion trajectories. Through extensive
  experiments, we show that AGD achieves comparable or superior FID to CFG across
  multiple architectures with only half the NFEs. Notably, our method enables the
  distillation of large models (sim2.6B parameters) on a single consumer GPU
  with 24 GB of VRAM, making it more accessible than previous approaches that
  require multiple high-end GPUs. We will publicly release the implementation of
  our method.
paper_ai_keywords: classifier-free guidance (CFG) diffusion models neural function evaluations (NFEs) adapter guidance distillation (AGD) lightweight adapters sampling speed sample quality guidance distillation methods base model minimal additional parameters resource requirement training trajectories diffusion trajectories FID large models consumer GPU VRAM