title: DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
summary: Diffusion models (DMs) have revolutionized generative learning. They utilize
  a diffusion process to encode data into a simple Gaussian distribution.
  However, encoding a complex, potentially multimodal data distribution into a
  single continuous Gaussian distribution arguably represents an unnecessarily
  challenging learning problem. We propose Discrete-Continuous Latent Variable
  Diffusion Models (DisCo-Diff) to simplify this task by introducing
  complementary discrete latent variables. We augment DMs with learnable discrete
  latents, inferred with an encoder, and train DM and encoder end-to-end.
  DisCo-Diff does not rely on pre-trained networks, making the framework
  universally applicable. The discrete latents significantly simplify learning
  the DM's complex noise-to-data mapping by reducing the curvature of the DM's
  generative ODE. An additional autoregressive transformer models the
  distribution of the discrete latents, a simple step because DisCo-Diff requires
  only few discrete variables with small codebooks. We validate DisCo-Diff on toy
  data, several image synthesis tasks as well as molecular docking, and find that
  introducing discrete latents consistently improves model performance. For
  example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned
  ImageNet-64/128 datasets with ODE sampler.
paper_authors._id: 6686497f1119f961e54211e9
paper_authors.name: Yilun Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:35:48.060Z
paper_authors._id: 649c37de5ffe05267a105fe8
paper_authors.avatarUrl: /avatars/4f262272e6222f879c6c0fedfa2e5861.svg
paper_authors.fullname: Yilun Xu
paper_authors.type: user
paper_authors.user: AaronXyl
paper_authors._id: 6686497f1119f961e54211ea
paper_authors.name: Gabriele Corso
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:46:42.221Z
paper_authors._id: 63445a3332ccc5ca9938acd4
paper_authors.avatarUrl: /avatars/4a1f6dd371094ada3addc10746a91616.svg
paper_authors.fullname: Gabriele Corso
paper_authors.type: user
paper_authors.user: gcorso
paper_authors._id: 6686497f1119f961e54211eb
paper_authors.name: Tommi Jaakkola
paper_authors._id: 6686497f1119f961e54211ec
paper_authors.name: Arash Vahdat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:47:07.441Z
paper_authors._id: 62f6956cd3bdacb7eec02920
paper_authors.avatarUrl: /avatars/b22db0823311f866c00db2efc4b9f814.svg
paper_authors.fullname: Arash Vahdat
paper_authors.type: user
paper_authors.user: avahdat
paper_authors._id: 6686497f1119f961e54211ed
paper_authors.name: Karsten Kreis
paper_title: DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
paper_summary: Diffusion models (DMs) have revolutionized generative learning. They utilize
  a diffusion process to encode data into a simple Gaussian distribution.
  However, encoding a complex, potentially multimodal data distribution into a
  single continuous Gaussian distribution arguably represents an unnecessarily
  challenging learning problem. We propose Discrete-Continuous Latent Variable
  Diffusion Models (DisCo-Diff) to simplify this task by introducing
  complementary discrete latent variables. We augment DMs with learnable discrete
  latents, inferred with an encoder, and train DM and encoder end-to-end.
  DisCo-Diff does not rely on pre-trained networks, making the framework
  universally applicable. The discrete latents significantly simplify learning
  the DM's complex noise-to-data mapping by reducing the curvature of the DM's
  generative ODE. An additional autoregressive transformer models the
  distribution of the discrete latents, a simple step because DisCo-Diff requires
  only few discrete variables with small codebooks. We validate DisCo-Diff on toy
  data, several image synthesis tasks as well as molecular docking, and find that
  introducing discrete latents consistently improves model performance. For
  example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned
  ImageNet-64/128 datasets with ODE sampler.
paper_ai_summary: Discrete-Continuous Latent Variable Diffusion Models simplify generative learning by introducing discrete latent variables, enhancing performance across various tasks.
paper_ai_keywords: diffusion models Discrete-Continuous Latent Variable Diffusion Models DisCo-Diff discrete latent variables learnable discrete latents encoder end-to-end training ODE autoregressive transformer FID scores ImageNet-64/128 molecular docking