title: Kaleido Diffusion: Improving Conditional Diffusion Models with
    Autoregressive Latent Modeling
summary: Diffusion models have emerged as a powerful tool for generating high-quality
  images from textual descriptions. Despite their successes, these models often
  exhibit limited diversity in the sampled images, particularly when sampling
  with a high classifier-free guidance weight. To address this issue, we present
  Kaleido, a novel approach that enhances the diversity of samples by
  incorporating autoregressive latent priors. Kaleido integrates an
  autoregressive language model that encodes the original caption and generates
  latent variables, serving as abstract and intermediary representations for
  guiding and facilitating the image generation process. In this paper, we
  explore a variety of discrete latent representations, including textual
  descriptions, detection bounding boxes, object blobs, and visual tokens. These
  representations diversify and enrich the input conditions to the diffusion
  models, enabling more diverse outputs. Our experimental results demonstrate
  that Kaleido effectively broadens the diversity of the generated image samples
  from a given textual description while maintaining high image quality.
  Furthermore, we show that Kaleido adheres closely to the guidance provided by
  the generated latent variables, demonstrating its capability to effectively
  control and direct the image generation process.
paper_authors._id: 665d398aa1ff38df97575d81
paper_authors.name: Jiatao Gu
paper_authors._id: 665d398aa1ff38df97575d82
paper_authors.name: Ying Shen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-03T18:03:03.793Z
paper_authors._id: 631799d5b58b018463107444
paper_authors.avatarUrl: /avatars/b9a8ce5460d9d8ae98b9f471d2011075.svg
paper_authors.fullname: Ying Shen
paper_authors.type: user
paper_authors.user: yingss
paper_authors._id: 665d398aa1ff38df97575d83
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:53.167Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 665d398aa1ff38df97575d84
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:39:08.020Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_authors._id: 665d398aa1ff38df97575d85
paper_authors.name: Navdeep Jaitly
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:39:14.484Z
paper_authors._id: 631119df88942700629f4aab
paper_authors.avatarUrl: /avatars/251c20a8fd48831a8c0cc0d198a42455.svg
paper_authors.fullname: Navdeep Jaitly
paper_authors.type: user
paper_authors.user: ndjaitly
paper_authors._id: 665d398aa1ff38df97575d86
paper_authors.name: Joshua M. Susskind
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:39:21.104Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_title: Kaleido Diffusion: Improving Conditional Diffusion Models with
    Autoregressive Latent Modeling
paper_summary: Diffusion models have emerged as a powerful tool for generating high-quality
  images from textual descriptions. Despite their successes, these models often
  exhibit limited diversity in the sampled images, particularly when sampling
  with a high classifier-free guidance weight. To address this issue, we present
  Kaleido, a novel approach that enhances the diversity of samples by
  incorporating autoregressive latent priors. Kaleido integrates an
  autoregressive language model that encodes the original caption and generates
  latent variables, serving as abstract and intermediary representations for
  guiding and facilitating the image generation process. In this paper, we
  explore a variety of discrete latent representations, including textual
  descriptions, detection bounding boxes, object blobs, and visual tokens. These
  representations diversify and enrich the input conditions to the diffusion
  models, enabling more diverse outputs. Our experimental results demonstrate
  that Kaleido effectively broadens the diversity of the generated image samples
  from a given textual description while maintaining high image quality.
  Furthermore, we show that Kaleido adheres closely to the guidance provided by
  the generated latent variables, demonstrating its capability to effectively
  control and direct the image generation process.
paper_ai_keywords: diffusion models classifier-free guidance autoregressive latent priors autoregressive language model latent variables discrete latent representations textual descriptions detection bounding boxes object blobs visual tokens image quality