Unconditional Priors Matter! Improving Conditional Generation of
  Fine-Tuned Diffusion Models
Classifier-Free Guidance (CFG) is a fundamental technique in training
conditional diffusion models. The common practice for CFG-based training is to
use a single network to learn both conditional and unconditional noise
prediction, with a small dropout rate for conditioning. However, we observe
that the joint learning of unconditional noise with limited bandwidth in
training results in poor priors for the unconditional case. More importantly,
these poor unconditional noise predictions become a serious reason for
degrading the quality of conditional generation. Inspired by the fact that most
CFG-based conditional models are trained by fine-tuning a base model with
better unconditional generation, we first show that simply replacing the
unconditional noise in CFG with that predicted by the base model can
significantly improve conditional generation. Furthermore, we show that a
diffusion model other than the one the fine-tuned model was trained on can be
used for unconditional noise replacement. We experimentally verify our claim
with a range of CFG-based conditional models for both image and video
generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and
InstructPix2Pix.
67e4ae6787c92169aa3caa74
Prin Phunyaphibarn
claimed_verified
2025-03-27T09:03:34.055Z
66435efdc26b490acc85079b
/avatars/16e0ee25734516d4295abe0fcc0e26a9.svg
Prin Phunyaphibarn
user
prinphunya
67e4ae6787c92169aa3caa75
Phillip Y. Lee
claimed_verified
2025-03-27T09:03:32.144Z
6342796a0875f2c99cfd313b
/avatars/98575092404c4197b20c929a6499a015.svg
Yuseung "Phillip" Lee
user
phillipinseoul
67e4ae6787c92169aa3caa76
Jaihoon Kim
67e4ae6787c92169aa3caa77
Minhyuk Sung
admin_assigned
2025-03-27T09:47:11.255Z
631f432b5ba8c026340a7890
https://cdn-avatars.huggingface.co/v1/production/uploads/631f432b5ba8c026340a7890/9PK7A_TRMpugwYjCsNBf1.jpeg
Minhyuk Sung
user
Minhyuk
Unconditional Priors Matter! Improving Conditional Generation of
  Fine-Tuned Diffusion Models
Classifier-Free Guidance (CFG) is a fundamental technique in training
conditional diffusion models. The common practice for CFG-based training is to
use a single network to learn both conditional and unconditional noise
prediction, with a small dropout rate for conditioning. However, we observe
that the joint learning of unconditional noise with limited bandwidth in
training results in poor priors for the unconditional case. More importantly,
these poor unconditional noise predictions become a serious reason for
degrading the quality of conditional generation. Inspired by the fact that most
CFG-based conditional models are trained by fine-tuning a base model with
better unconditional generation, we first show that simply replacing the
unconditional noise in CFG with that predicted by the base model can
significantly improve conditional generation. Furthermore, we show that a
diffusion model other than the one the fine-tuned model was trained on can be
used for unconditional noise replacement. We experimentally verify our claim
with a range of CFG-based conditional models for both image and video
generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and
InstructPix2Pix.
Replacing unconditional noise predictions in Classifier-Free Guidance with those from a pre-trained model improves conditional generation quality for diffusion models.
Classifier-Free Guidance conditional diffusion models noise prediction dropout rate priors fine-tuning base model image generation video generation Zero-1-to-3 Versatile Diffusion DiT DynamiCrafter InstructPix2Pix