title: Understanding Hallucinations in Diffusion Models through Mode
    Interpolation
summary: Colloquially speaking, image generation models based upon diffusion processes
  are frequently said to exhibit "hallucinations," samples that could never occur
  in the training data. But where do such hallucinations come from? In this
  paper, we study a particular failure mode in diffusion models, which we term
  mode interpolation. Specifically, we find that diffusion models smoothly
  "interpolate" between nearby data modes in the training set, to generate
  samples that are completely outside the support of the original training
  distribution; this phenomenon leads diffusion models to generate artifacts that
  never existed in real data (i.e., hallucinations). We systematically study the
  reasons for, and the manifestation of this phenomenon. Through experiments on
  1D and 2D Gaussians, we show how a discontinuous loss landscape in the
  diffusion model's decoder leads to a region where any smooth approximation will
  cause such hallucinations. Through experiments on artificial datasets with
  various shapes, we show how hallucination leads to the generation of
  combinations of shapes that never existed. Finally, we show that diffusion
  models in fact know when they go out of support and hallucinate. This is
  captured by the high variance in the trajectory of the generated sample towards
  the final few backward sampling process. Using a simple metric to capture this
  variance, we can remove over 95% of hallucinations at generation time while
  retaining 96% of in-support samples. We conclude our exploration by showing the
  implications of such hallucination (and its removal) on the collapse (and
  stabilization) of recursive training on synthetic data with experiments on
  MNIST and 2D Gaussians dataset. We release our code at
  https://github.com/locuslab/diffusion-model-hallucination.
paper_authors._id: 666cc6a4c0f3d5afc216a089
paper_authors.name: Sumukh K Aithal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:18:51.996Z
paper_authors._id: 6660c0b74228284ed72fc818
paper_authors.avatarUrl: /avatars/befaf728c4e517c88adc30536a29d90d.svg
paper_authors.fullname: Sumukh Aithal
paper_authors.type: user
paper_authors.user: sumukhaithal6
paper_authors._id: 666cc6a4c0f3d5afc216a08a
paper_authors.name: Pratyush Maini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:18:57.997Z
paper_authors._id: 632c826d1d303f5f9acf5917
paper_authors.avatarUrl: /avatars/defca406f2c4f027a7b1be9a5ebbbd01.svg
paper_authors.fullname: Pratyush Maini
paper_authors.type: user
paper_authors.user: pratyushmaini
paper_authors._id: 666cc6a4c0f3d5afc216a08b
paper_authors.name: Zachary C. Lipton
paper_authors._id: 666cc6a4c0f3d5afc216a08c
paper_authors.name: J. Zico Kolter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:19:13.886Z
paper_authors._id: 6561eb612cd8dac568605da3
paper_authors.avatarUrl: /avatars/6d145d7e225b16bde64680ba8c05f856.svg
paper_authors.fullname: Zico Kolter
paper_authors.type: user
paper_authors.user: zkolter
paper_title: Understanding Hallucinations in Diffusion Models through Mode
    Interpolation
paper_summary: Colloquially speaking, image generation models based upon diffusion processes
  are frequently said to exhibit "hallucinations," samples that could never occur
  in the training data. But where do such hallucinations come from? In this
  paper, we study a particular failure mode in diffusion models, which we term
  mode interpolation. Specifically, we find that diffusion models smoothly
  "interpolate" between nearby data modes in the training set, to generate
  samples that are completely outside the support of the original training
  distribution; this phenomenon leads diffusion models to generate artifacts that
  never existed in real data (i.e., hallucinations). We systematically study the
  reasons for, and the manifestation of this phenomenon. Through experiments on
  1D and 2D Gaussians, we show how a discontinuous loss landscape in the
  diffusion model's decoder leads to a region where any smooth approximation will
  cause such hallucinations. Through experiments on artificial datasets with
  various shapes, we show how hallucination leads to the generation of
  combinations of shapes that never existed. Finally, we show that diffusion
  models in fact know when they go out of support and hallucinate. This is
  captured by the high variance in the trajectory of the generated sample towards
  the final few backward sampling process. Using a simple metric to capture this
  variance, we can remove over 95% of hallucinations at generation time while
  retaining 96% of in-support samples. We conclude our exploration by showing the
  implications of such hallucination (and its removal) on the collapse (and
  stabilization) of recursive training on synthetic data with experiments on
  MNIST and 2D Gaussians dataset. We release our code at
  https://github.com/locuslab/diffusion-model-hallucination.
paper_ai_summary: Diffusion models generate unrealistic samples, termed hallucinations, through mode interpolation due to a discontinuous loss landscape in the decoder, leading to high variance in backward sampling; this can be mitigated with a simple variance metric.
paper_ai_keywords: diffusion models mode interpolation data modes training distribution hallucinations loss landscape generated sample variance recursive training