Kaleido Diffusion: Improving Conditional Diffusion Models with
  Autoregressive Latent Modeling
Diffusion models have emerged as a powerful tool for generating high-quality
images from textual descriptions. Despite their successes, these models often
exhibit limited diversity in the sampled images, particularly when sampling
with a high classifier-free guidance weight. To address this issue, we present
Kaleido, a novel approach that enhances the diversity of samples by
incorporating autoregressive latent priors. Kaleido integrates an
autoregressive language model that encodes the original caption and generates
latent variables, serving as abstract and intermediary representations for
guiding and facilitating the image generation process. In this paper, we
explore a variety of discrete latent representations, including textual
descriptions, detection bounding boxes, object blobs, and visual tokens. These
representations diversify and enrich the input conditions to the diffusion
models, enabling more diverse outputs. Our experimental results demonstrate
that Kaleido effectively broadens the diversity of the generated image samples
from a given textual description while maintaining high image quality.
Furthermore, we show that Kaleido adheres closely to the guidance provided by
the generated latent variables, demonstrating its capability to effectively
control and direct the image generation process.
665d398aa1ff38df97575d81
Jiatao Gu
665d398aa1ff38df97575d82
Ying Shen
extracted_pending
2024-08-03T18:03:03.793Z
631799d5b58b018463107444
/avatars/b9a8ce5460d9d8ae98b9f471d2011075.svg
Ying Shen
user
yingss
665d398aa1ff38df97575d83
Shuangfei Zhai
claimed_verified
2024-12-13T15:23:53.167Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
665d398aa1ff38df97575d84
Yizhe Zhang
admin_assigned
2024-06-03T08:39:08.020Z
5e64858c87403103f9f1055d
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
Yizhe Zhang
user
yizhezhang
665d398aa1ff38df97575d85
Navdeep Jaitly
admin_assigned
2024-06-03T08:39:14.484Z
631119df88942700629f4aab
/avatars/251c20a8fd48831a8c0cc0d198a42455.svg
Navdeep Jaitly
user
ndjaitly
665d398aa1ff38df97575d86
Joshua M. Susskind
admin_assigned
2024-06-03T08:39:21.104Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
Kaleido Diffusion: Improving Conditional Diffusion Models with
  Autoregressive Latent Modeling
Diffusion models have emerged as a powerful tool for generating high-quality
images from textual descriptions. Despite their successes, these models often
exhibit limited diversity in the sampled images, particularly when sampling
with a high classifier-free guidance weight. To address this issue, we present
Kaleido, a novel approach that enhances the diversity of samples by
incorporating autoregressive latent priors. Kaleido integrates an
autoregressive language model that encodes the original caption and generates
latent variables, serving as abstract and intermediary representations for
guiding and facilitating the image generation process. In this paper, we
explore a variety of discrete latent representations, including textual
descriptions, detection bounding boxes, object blobs, and visual tokens. These
representations diversify and enrich the input conditions to the diffusion
models, enabling more diverse outputs. Our experimental results demonstrate
that Kaleido effectively broadens the diversity of the generated image samples
from a given textual description while maintaining high image quality.
Furthermore, we show that Kaleido adheres closely to the guidance provided by
the generated latent variables, demonstrating its capability to effectively
control and direct the image generation process.
Kaleido enhances diffusion model diversity by integrating autoregressive latent priors, generating varied and high-quality images from textual descriptions.
diffusion models autoregressive latent priors autoregressive language model latent variables discrete latent representations textual descriptions detection bounding boxes object blobs visual tokens guidance weight classifier-free guidance weight