Improving Diffusion-Based Image Synthesis with Context Prediction
Diffusion models are a new class of generative models, and have dramatically
promoted image generation with unprecedented quality and diversity. Existing
diffusion models mainly try to reconstruct input image from a corrupted one
with a pixel-wise or feature-wise constraint along spatial axes. However, such
point-based reconstruction may fail to make each predicted pixel/feature fully
preserve its neighborhood context, impairing diffusion-based image synthesis.
As a powerful source of automatic supervisory signal, context has been well
studied for learning representations. Inspired by this, we for the first time
propose ConPreDiff to improve diffusion-based image synthesis with context
prediction. We explicitly reinforce each point to predict its neighborhood
context (i.e., multi-stride features/tokens/pixels) with a context decoder at
the end of diffusion denoising blocks in training stage, and remove the decoder
for inference. In this way, each point can better reconstruct itself by
preserving its semantic connections with neighborhood context. This new
paradigm of ConPreDiff can generalize to arbitrary discrete and continuous
diffusion backbones without introducing extra parameters in sampling procedure.
Extensive experiments are conducted on unconditional image generation,
text-to-image generation and image inpainting tasks. Our ConPreDiff
consistently outperforms previous methods and achieves a new SOTA text-to-image
generation results on MS-COCO, with a zero-shot FID score of 6.21.
659798bf6a30bf8860e96a41
Ling Yang
claimed_verified
2024-01-23T13:49:09.721Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
659798bf6a30bf8860e96a42
Jingwei Liu
659798bf6a30bf8860e96a43
Shenda Hong
659798bf6a30bf8860e96a44
Zhilong Zhang
659798bf6a30bf8860e96a45
Zhilin Huang
659798bf6a30bf8860e96a46
Zheming Cai
659798bf6a30bf8860e96a47
Wentao Zhang
659798bf6a30bf8860e96a48
Bin Cui
Improving Diffusion-Based Image Synthesis with Context Prediction
Diffusion models are a new class of generative models, and have dramatically
promoted image generation with unprecedented quality and diversity. Existing
diffusion models mainly try to reconstruct input image from a corrupted one
with a pixel-wise or feature-wise constraint along spatial axes. However, such
point-based reconstruction may fail to make each predicted pixel/feature fully
preserve its neighborhood context, impairing diffusion-based image synthesis.
As a powerful source of automatic supervisory signal, context has been well
studied for learning representations. Inspired by this, we for the first time
propose ConPreDiff to improve diffusion-based image synthesis with context
prediction. We explicitly reinforce each point to predict its neighborhood
context (i.e., multi-stride features/tokens/pixels) with a context decoder at
the end of diffusion denoising blocks in training stage, and remove the decoder
for inference. In this way, each point can better reconstruct itself by
preserving its semantic connections with neighborhood context. This new
paradigm of ConPreDiff can generalize to arbitrary discrete and continuous
diffusion backbones without introducing extra parameters in sampling procedure.
Extensive experiments are conducted on unconditional image generation,
text-to-image generation and image inpainting tasks. Our ConPreDiff
consistently outperforms previous methods and achieves a new SOTA text-to-image
generation results on MS-COCO, with a zero-shot FID score of 6.21.
ConPreDiff enhances diffusion models by incorporating context prediction to improve image synthesis quality across various tasks.
diffusion models context prediction ConPreDiff diffusion denoising blocks zero-shot FID score unconditional image generation text-to-image generation image inpainting