title: Real-World Image Variation by Aligning Diffusion Inversion Chain
summary: Recent diffusion model advancements have enabled high-fidelity images to be
  generated using text prompts. However, a domain gap exists between generated
  images and real-world images, which poses a challenge in generating
  high-quality variations of real-world images. Our investigation uncovers that
  this domain gap originates from a latents' distribution gap in different
  diffusion processes. To address this issue, we propose a novel inference
  pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes
  diffusion models to generate image variations from a single image exemplar. Our
  pipeline enhances the generation quality of image variations by aligning the
  image generation process to the source image's inversion chain. Specifically,
  we demonstrate that step-wise latent distribution alignment is essential for
  generating high-quality variations. To attain this, we design a cross-image
  self-attention injection for feature interaction and a step-wise distribution
  normalization to align the latent features. Incorporating these alignment
  processes into a diffusion model allows RIVAL to generate high-quality image
  variations without further parameter optimization. Our experimental results
  demonstrate that our proposed approach outperforms existing methods with
  respect to semantic-condition similarity and perceptual quality. Furthermore,
  this generalized inference pipeline can be easily applied to other
  diffusion-based generation tasks, such as image-conditioned text-to-image
  generation and example-based image inpainting.
paper_authors._id: 6476af9b384249d57d374637
paper_authors.name: Yuechen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:01:08.002Z
paper_authors._id: 6418554a0956be7233a1023e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
paper_authors.fullname: zhang yuechen
paper_authors.type: user
paper_authors.user: julianjuaner
paper_authors._id: 6476af9b384249d57d374638
paper_authors.name: Jinbo Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:01:10.911Z
paper_authors._id: 64770e86d7cf39f2e937ae9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
paper_authors.fullname: Jinbo Xing
paper_authors.type: user
paper_authors.user: Doubiiu
paper_authors._id: 6476af9b384249d57d374639
paper_authors.name: Eric Lo
paper_authors._id: 6476af9b384249d57d37463a
paper_authors.name: Jiaya Jia
paper_title: Real-World Image Variation by Aligning Diffusion Inversion Chain
paper_summary: Recent diffusion model advancements have enabled high-fidelity images to be
  generated using text prompts. However, a domain gap exists between generated
  images and real-world images, which poses a challenge in generating
  high-quality variations of real-world images. Our investigation uncovers that
  this domain gap originates from a latents' distribution gap in different
  diffusion processes. To address this issue, we propose a novel inference
  pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes
  diffusion models to generate image variations from a single image exemplar. Our
  pipeline enhances the generation quality of image variations by aligning the
  image generation process to the source image's inversion chain. Specifically,
  we demonstrate that step-wise latent distribution alignment is essential for
  generating high-quality variations. To attain this, we design a cross-image
  self-attention injection for feature interaction and a step-wise distribution
  normalization to align the latent features. Incorporating these alignment
  processes into a diffusion model allows RIVAL to generate high-quality image
  variations without further parameter optimization. Our experimental results
  demonstrate that our proposed approach outperforms existing methods with
  respect to semantic-condition similarity and perceptual quality. Furthermore,
  this generalized inference pipeline can be easily applied to other
  diffusion-based generation tasks, such as image-conditioned text-to-image
  generation and example-based image inpainting.
paper_ai_summary: RIVAL uses diffusion models and step-wise latent alignment to generate high-quality real-world image variations from a single exemplar, surpassing existing methods.
paper_ai_keywords: diffusion models latent distribution gap image generation process source image's inversion chain step-wise latent distribution alignment cross-image self-attention feature interaction step-wise distribution normalization semantic-condition similarity perceptual quality image-conditioned text-to-image generation example-based image inpainting