Unified Multimodal Discrete Diffusion
Multimodal generative models that can understand and generate across multiple
modalities are dominated by autoregressive (AR) approaches, which process
tokens sequentially from left to right, or top to bottom. These models jointly
handle images, text, video, and audio for various tasks such as image
captioning, question answering, and image generation. In this work, we explore
discrete diffusion models as a unified generative formulation in the joint text
and image domain, building upon their recent success in text generation.
Discrete diffusion models offer several advantages over AR models, including
improved control over quality versus diversity of generated samples, the
ability to perform joint multimodal inpainting (across both text and image
domains), and greater controllability in generation through guidance.
Leveraging these benefits, we present the first Unified Multimodal Discrete
Diffusion (UniDisc) model which is capable of jointly understanding and
generating text and images for a variety of downstream tasks. We compare
UniDisc to multimodal AR models, performing a scaling analysis and
demonstrating that UniDisc outperforms them in terms of both performance and
inference-time compute, enhanced controllability, editability, inpainting, and
flexible trade-off between inference time and generation quality. Code and
additional visualizations are available at https://unidisc.github.io.
67e5ff24cce3913200f29387
Alexander Swerdlow
admin_assigned
2025-03-28T09:18:36.245Z
62f6bd1dd278a8f3e7867392
/avatars/7e5b6014d99909958eb0f95c486b2226.svg
Alexander Swerdlow
user
aswerdlow
67e5ff24cce3913200f29388
Mihir Prabhudesai
admin_assigned
2025-03-28T09:18:30.166Z
6310ff7dd43c55e811f8772f
/avatars/0dcab81771329d13cc62b08e55a71c9a.svg
Mihir Prabhudesai
user
mihirpd
67e5ff24cce3913200f29389
Siddharth Gandhi
admin_assigned
2025-03-28T09:18:42.006Z
67ad6a0924ad6fd76672ff2f
/avatars/141280d9d1b97740f0e8acf9b681411d.svg
Siddharth Gandhi
user
Sid1275
67e5ff24cce3913200f2938a
Deepak Pathak
admin_assigned
2025-03-28T09:18:48.839Z
653692e8ac570e90963cf2c5
/avatars/e391ca21c2292e916ab0ab00f8ee2ba6.svg
Deepak pathak
user
Deepak765
67e5ff24cce3913200f2938b
Katerina Fragkiadaki
Unified Multimodal Discrete Diffusion
Multimodal generative models that can understand and generate across multiple
modalities are dominated by autoregressive (AR) approaches, which process
tokens sequentially from left to right, or top to bottom. These models jointly
handle images, text, video, and audio for various tasks such as image
captioning, question answering, and image generation. In this work, we explore
discrete diffusion models as a unified generative formulation in the joint text
and image domain, building upon their recent success in text generation.
Discrete diffusion models offer several advantages over AR models, including
improved control over quality versus diversity of generated samples, the
ability to perform joint multimodal inpainting (across both text and image
domains), and greater controllability in generation through guidance.
Leveraging these benefits, we present the first Unified Multimodal Discrete
Diffusion (UniDisc) model which is capable of jointly understanding and
generating text and images for a variety of downstream tasks. We compare
UniDisc to multimodal AR models, performing a scaling analysis and
demonstrating that UniDisc outperforms them in terms of both performance and
inference-time compute, enhanced controllability, editability, inpainting, and
flexible trade-off between inference time and generation quality. Code and
additional visualizations are available at https://unidisc.github.io.
A Unified Multimodal Discrete Diffusion model outperforms autoregressive models in tasks like image captioning, question answering, and image generation, offering better control, editability, and performance.
discrete diffusion models autoregressive models multimodal generative models joint text and image domain multimodal inpainting controllability inference-time compute generation quality