title: Unconditional Priors Matter! Improving Conditional Generation of
    Fine-Tuned Diffusion Models
summary: Classifier-Free Guidance (CFG) is a fundamental technique in training
  conditional diffusion models. The common practice for CFG-based training is to
  use a single network to learn both conditional and unconditional noise
  prediction, with a small dropout rate for conditioning. However, we observe
  that the joint learning of unconditional noise with limited bandwidth in
  training results in poor priors for the unconditional case. More importantly,
  these poor unconditional noise predictions become a serious reason for
  degrading the quality of conditional generation. Inspired by the fact that most
  CFG-based conditional models are trained by fine-tuning a base model with
  better unconditional generation, we first show that simply replacing the
  unconditional noise in CFG with that predicted by the base model can
  significantly improve conditional generation. Furthermore, we show that a
  diffusion model other than the one the fine-tuned model was trained on can be
  used for unconditional noise replacement. We experimentally verify our claim
  with a range of CFG-based conditional models for both image and video
  generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and
  InstructPix2Pix.
paper_authors._id: 67e4ae6787c92169aa3caa74
paper_authors.name: Prin Phunyaphibarn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:03:34.055Z
paper_authors._id: 66435efdc26b490acc85079b
paper_authors.avatarUrl: /avatars/16e0ee25734516d4295abe0fcc0e26a9.svg
paper_authors.fullname: Prin Phunyaphibarn
paper_authors.type: user
paper_authors.user: prinphunya
paper_authors._id: 67e4ae6787c92169aa3caa75
paper_authors.name: Phillip Y. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:03:32.144Z
paper_authors._id: 6342796a0875f2c99cfd313b
paper_authors.avatarUrl: /avatars/98575092404c4197b20c929a6499a015.svg
paper_authors.fullname: Yuseung "Phillip" Lee
paper_authors.type: user
paper_authors.user: phillipinseoul
paper_authors._id: 67e4ae6787c92169aa3caa76
paper_authors.name: Jaihoon Kim
paper_authors._id: 67e4ae6787c92169aa3caa77
paper_authors.name: Minhyuk Sung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:47:11.255Z
paper_authors._id: 631f432b5ba8c026340a7890
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f432b5ba8c026340a7890/9PK7A_TRMpugwYjCsNBf1.jpeg
paper_authors.fullname: Minhyuk Sung
paper_authors.type: user
paper_authors.user: Minhyuk
paper_title: Unconditional Priors Matter! Improving Conditional Generation of
    Fine-Tuned Diffusion Models
paper_summary: Classifier-Free Guidance (CFG) is a fundamental technique in training
  conditional diffusion models. The common practice for CFG-based training is to
  use a single network to learn both conditional and unconditional noise
  prediction, with a small dropout rate for conditioning. However, we observe
  that the joint learning of unconditional noise with limited bandwidth in
  training results in poor priors for the unconditional case. More importantly,
  these poor unconditional noise predictions become a serious reason for
  degrading the quality of conditional generation. Inspired by the fact that most
  CFG-based conditional models are trained by fine-tuning a base model with
  better unconditional generation, we first show that simply replacing the
  unconditional noise in CFG with that predicted by the base model can
  significantly improve conditional generation. Furthermore, we show that a
  diffusion model other than the one the fine-tuned model was trained on can be
  used for unconditional noise replacement. We experimentally verify our claim
  with a range of CFG-based conditional models for both image and video
  generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and
  InstructPix2Pix.
paper_ai_summary: Replacing unconditional noise predictions in Classifier-Free Guidance with those from a pre-trained model improves conditional generation quality for diffusion models.
paper_ai_keywords: Classifier-Free Guidance conditional diffusion models noise prediction dropout rate priors fine-tuning base model image generation video generation Zero-1-to-3 Versatile Diffusion DiT DynamiCrafter InstructPix2Pix