Matryoshka Diffusion Models
Diffusion models are the de facto approach for generating high-quality images
and videos, but learning high-dimensional models remains a formidable task due
to computational and optimization challenges. Existing methods often resort to
training cascaded models in pixel space or using a downsampled latent space of
a separately trained auto-encoder. In this paper, we introduce Matryoshka
Diffusion Models(MDM), an end-to-end framework for high-resolution image and
video synthesis. We propose a diffusion process that denoises inputs at
multiple resolutions jointly and uses a NestedUNet architecture where features
and parameters for small-scale inputs are nested within those of large scales.
In addition, MDM enables a progressive training schedule from lower to higher
resolutions, which leads to significant improvements in optimization for
high-resolution generation. We demonstrate the effectiveness of our approach on
various benchmarks, including class-conditioned image generation,
high-resolution text-to-image, and text-to-video applications. Remarkably, we
can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
demonstrating strong zero-shot generalization using the CC12M dataset, which
contains only 12 million images.
6537db8f3af2f64a0d5df7e2
Jiatao Gu
6537db8f3af2f64a0d5df7e3
Shuangfei Zhai
claimed_verified
2024-12-13T15:25:23.199Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
6537db8f3af2f64a0d5df7e4
Yizhe Zhang
6537db8f3af2f64a0d5df7e5
Josh Susskind
claimed_verified
2024-11-22T16:30:56.489Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
6537db8f3af2f64a0d5df7e6
Navdeep Jaitly
Matryoshka Diffusion Models
Diffusion models are the de facto approach for generating high-quality images
and videos, but learning high-dimensional models remains a formidable task due
to computational and optimization challenges. Existing methods often resort to
training cascaded models in pixel space or using a downsampled latent space of
a separately trained auto-encoder. In this paper, we introduce Matryoshka
Diffusion Models(MDM), an end-to-end framework for high-resolution image and
video synthesis. We propose a diffusion process that denoises inputs at
multiple resolutions jointly and uses a NestedUNet architecture where features
and parameters for small-scale inputs are nested within those of large scales.
In addition, MDM enables a progressive training schedule from lower to higher
resolutions, which leads to significant improvements in optimization for
high-resolution generation. We demonstrate the effectiveness of our approach on
various benchmarks, including class-conditioned image generation,
high-resolution text-to-image, and text-to-video applications. Remarkably, we
can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
demonstrating strong zero-shot generalization using the CC12M dataset, which
contains only 12 million images.
Matryoshka Diffusion Models use a NestedUNet architecture for joint denoising at multiple resolutions, enabling efficient high-resolution image and video synthesis.
diffusion models high-resolution image and video synthesis NestedUNet end-to-end framework progressive training schedule class-conditioned image generation high-resolution text-to-image text-to-video CC12M dataset