title: Matryoshka Diffusion Models
summary: Diffusion models are the de facto approach for generating high-quality images
  and videos, but learning high-dimensional models remains a formidable task due
  to computational and optimization challenges. Existing methods often resort to
  training cascaded models in pixel space or using a downsampled latent space of
  a separately trained auto-encoder. In this paper, we introduce Matryoshka
  Diffusion Models(MDM), an end-to-end framework for high-resolution image and
  video synthesis. We propose a diffusion process that denoises inputs at
  multiple resolutions jointly and uses a NestedUNet architecture where features
  and parameters for small-scale inputs are nested within those of large scales.
  In addition, MDM enables a progressive training schedule from lower to higher
  resolutions, which leads to significant improvements in optimization for
  high-resolution generation. We demonstrate the effectiveness of our approach on
  various benchmarks, including class-conditioned image generation,
  high-resolution text-to-image, and text-to-video applications. Remarkably, we
  can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
  demonstrating strong zero-shot generalization using the CC12M dataset, which
  contains only 12 million images.
paper_authors._id: 6537db8f3af2f64a0d5df7e2
paper_authors.name: Jiatao Gu
paper_authors._id: 6537db8f3af2f64a0d5df7e3
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:25:23.199Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 6537db8f3af2f64a0d5df7e4
paper_authors.name: Yizhe Zhang
paper_authors._id: 6537db8f3af2f64a0d5df7e5
paper_authors.name: Josh Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T16:30:56.489Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_authors._id: 6537db8f3af2f64a0d5df7e6
paper_authors.name: Navdeep Jaitly
paper_title: Matryoshka Diffusion Models
paper_summary: Diffusion models are the de facto approach for generating high-quality images
  and videos, but learning high-dimensional models remains a formidable task due
  to computational and optimization challenges. Existing methods often resort to
  training cascaded models in pixel space or using a downsampled latent space of
  a separately trained auto-encoder. In this paper, we introduce Matryoshka
  Diffusion Models(MDM), an end-to-end framework for high-resolution image and
  video synthesis. We propose a diffusion process that denoises inputs at
  multiple resolutions jointly and uses a NestedUNet architecture where features
  and parameters for small-scale inputs are nested within those of large scales.
  In addition, MDM enables a progressive training schedule from lower to higher
  resolutions, which leads to significant improvements in optimization for
  high-resolution generation. We demonstrate the effectiveness of our approach on
  various benchmarks, including class-conditioned image generation,
  high-resolution text-to-image, and text-to-video applications. Remarkably, we
  can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
  demonstrating strong zero-shot generalization using the CC12M dataset, which
  contains only 12 million images.
paper_ai_keywords: diffusion models high-quality images videos learning high-dimensional models cascaded models pixel space downsampled latent space separately trained auto-encoder Matryoshka Diffusion Models (MDM) end-to-end framework high-resolution image and video synthesis diffusion process denoising inputs multiple resolutions NestedUNet architecture features parameters small-scale inputs large scales progressive training schedule optimization zero-shot generalization CC12M dataset text-to-image text-to-video