title: Unified Multimodal Discrete Diffusion
summary: Multimodal generative models that can understand and generate across multiple
  modalities are dominated by autoregressive (AR) approaches, which process
  tokens sequentially from left to right, or top to bottom. These models jointly
  handle images, text, video, and audio for various tasks such as image
  captioning, question answering, and image generation. In this work, we explore
  discrete diffusion models as a unified generative formulation in the joint text
  and image domain, building upon their recent success in text generation.
  Discrete diffusion models offer several advantages over AR models, including
  improved control over quality versus diversity of generated samples, the
  ability to perform joint multimodal inpainting (across both text and image
  domains), and greater controllability in generation through guidance.
  Leveraging these benefits, we present the first Unified Multimodal Discrete
  Diffusion (UniDisc) model which is capable of jointly understanding and
  generating text and images for a variety of downstream tasks. We compare
  UniDisc to multimodal AR models, performing a scaling analysis and
  demonstrating that UniDisc outperforms them in terms of both performance and
  inference-time compute, enhanced controllability, editability, inpainting, and
  flexible trade-off between inference time and generation quality. Code and
  additional visualizations are available at https://unidisc.github.io.
paper_authors._id: 67e5ff24cce3913200f29387
paper_authors.name: Alexander Swerdlow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:18:36.245Z
paper_authors._id: 62f6bd1dd278a8f3e7867392
paper_authors.avatarUrl: /avatars/7e5b6014d99909958eb0f95c486b2226.svg
paper_authors.fullname: Alexander Swerdlow
paper_authors.type: user
paper_authors.user: aswerdlow
paper_authors._id: 67e5ff24cce3913200f29388
paper_authors.name: Mihir Prabhudesai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:18:30.166Z
paper_authors._id: 6310ff7dd43c55e811f8772f
paper_authors.avatarUrl: /avatars/0dcab81771329d13cc62b08e55a71c9a.svg
paper_authors.fullname: Mihir Prabhudesai
paper_authors.type: user
paper_authors.user: mihirpd
paper_authors._id: 67e5ff24cce3913200f29389
paper_authors.name: Siddharth Gandhi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:18:42.006Z
paper_authors._id: 67ad6a0924ad6fd76672ff2f
paper_authors.avatarUrl: /avatars/141280d9d1b97740f0e8acf9b681411d.svg
paper_authors.fullname: Siddharth Gandhi
paper_authors.type: user
paper_authors.user: Sid1275
paper_authors._id: 67e5ff24cce3913200f2938a
paper_authors.name: Deepak Pathak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:18:48.839Z
paper_authors._id: 653692e8ac570e90963cf2c5
paper_authors.avatarUrl: /avatars/e391ca21c2292e916ab0ab00f8ee2ba6.svg
paper_authors.fullname: Deepak pathak
paper_authors.type: user
paper_authors.user: Deepak765
paper_authors._id: 67e5ff24cce3913200f2938b
paper_authors.name: Katerina Fragkiadaki
paper_title: Unified Multimodal Discrete Diffusion
paper_summary: Multimodal generative models that can understand and generate across multiple
  modalities are dominated by autoregressive (AR) approaches, which process
  tokens sequentially from left to right, or top to bottom. These models jointly
  handle images, text, video, and audio for various tasks such as image
  captioning, question answering, and image generation. In this work, we explore
  discrete diffusion models as a unified generative formulation in the joint text
  and image domain, building upon their recent success in text generation.
  Discrete diffusion models offer several advantages over AR models, including
  improved control over quality versus diversity of generated samples, the
  ability to perform joint multimodal inpainting (across both text and image
  domains), and greater controllability in generation through guidance.
  Leveraging these benefits, we present the first Unified Multimodal Discrete
  Diffusion (UniDisc) model which is capable of jointly understanding and
  generating text and images for a variety of downstream tasks. We compare
  UniDisc to multimodal AR models, performing a scaling analysis and
  demonstrating that UniDisc outperforms them in terms of both performance and
  inference-time compute, enhanced controllability, editability, inpainting, and
  flexible trade-off between inference time and generation quality. Code and
  additional visualizations are available at https://unidisc.github.io.
paper_ai_summary: A Unified Multimodal Discrete Diffusion model outperforms autoregressive models in tasks like image captioning, question answering, and image generation, offering better control, editability, and performance.
paper_ai_keywords: discrete diffusion models autoregressive models multimodal generative models joint text and image domain multimodal inpainting controllability inference-time compute generation quality