title: Improving Diffusion-Based Image Synthesis with Context Prediction
summary: Diffusion models are a new class of generative models, and have dramatically
  promoted image generation with unprecedented quality and diversity. Existing
  diffusion models mainly try to reconstruct input image from a corrupted one
  with a pixel-wise or feature-wise constraint along spatial axes. However, such
  point-based reconstruction may fail to make each predicted pixel/feature fully
  preserve its neighborhood context, impairing diffusion-based image synthesis.
  As a powerful source of automatic supervisory signal, context has been well
  studied for learning representations. Inspired by this, we for the first time
  propose ConPreDiff to improve diffusion-based image synthesis with context
  prediction. We explicitly reinforce each point to predict its neighborhood
  context (i.e., multi-stride features/tokens/pixels) with a context decoder at
  the end of diffusion denoising blocks in training stage, and remove the decoder
  for inference. In this way, each point can better reconstruct itself by
  preserving its semantic connections with neighborhood context. This new
  paradigm of ConPreDiff can generalize to arbitrary discrete and continuous
  diffusion backbones without introducing extra parameters in sampling procedure.
  Extensive experiments are conducted on unconditional image generation,
  text-to-image generation and image inpainting tasks. Our ConPreDiff
  consistently outperforms previous methods and achieves a new SOTA text-to-image
  generation results on MS-COCO, with a zero-shot FID score of 6.21.
paper_authors._id: 659798bf6a30bf8860e96a41
paper_authors.name: Ling Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T13:49:09.721Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 659798bf6a30bf8860e96a42
paper_authors.name: Jingwei Liu
paper_authors._id: 659798bf6a30bf8860e96a43
paper_authors.name: Shenda Hong
paper_authors._id: 659798bf6a30bf8860e96a44
paper_authors.name: Zhilong Zhang
paper_authors._id: 659798bf6a30bf8860e96a45
paper_authors.name: Zhilin Huang
paper_authors._id: 659798bf6a30bf8860e96a46
paper_authors.name: Zheming Cai
paper_authors._id: 659798bf6a30bf8860e96a47
paper_authors.name: Wentao Zhang
paper_authors._id: 659798bf6a30bf8860e96a48
paper_authors.name: Bin Cui
paper_title: Improving Diffusion-Based Image Synthesis with Context Prediction
paper_summary: Diffusion models are a new class of generative models, and have dramatically
  promoted image generation with unprecedented quality and diversity. Existing
  diffusion models mainly try to reconstruct input image from a corrupted one
  with a pixel-wise or feature-wise constraint along spatial axes. However, such
  point-based reconstruction may fail to make each predicted pixel/feature fully
  preserve its neighborhood context, impairing diffusion-based image synthesis.
  As a powerful source of automatic supervisory signal, context has been well
  studied for learning representations. Inspired by this, we for the first time
  propose ConPreDiff to improve diffusion-based image synthesis with context
  prediction. We explicitly reinforce each point to predict its neighborhood
  context (i.e., multi-stride features/tokens/pixels) with a context decoder at
  the end of diffusion denoising blocks in training stage, and remove the decoder
  for inference. In this way, each point can better reconstruct itself by
  preserving its semantic connections with neighborhood context. This new
  paradigm of ConPreDiff can generalize to arbitrary discrete and continuous
  diffusion backbones without introducing extra parameters in sampling procedure.
  Extensive experiments are conducted on unconditional image generation,
  text-to-image generation and image inpainting tasks. Our ConPreDiff
  consistently outperforms previous methods and achieves a new SOTA text-to-image
  generation results on MS-COCO, with a zero-shot FID score of 6.21.
paper_ai_summary: ConPreDiff enhances diffusion models by incorporating context prediction to improve image synthesis quality across various tasks.
paper_ai_keywords: diffusion models context prediction ConPreDiff diffusion denoising blocks zero-shot FID score unconditional image generation text-to-image generation image inpainting