LaViDa: A Large Diffusion Language Model for Multimodal Understanding
Modern Vision-Language Models (VLMs) can solve a wide range of tasks
requiring visual reasoning. In real-world scenarios, desirable properties for
VLMs include fast inference and controllable generation (e.g., constraining
outputs to adhere to a desired format). However, existing autoregressive (AR)
VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs)
offer a promising alternative, enabling parallel decoding for faster inference
and bidirectional context for controllable generation through text-infilling.
While effective in language-only settings, DMs' potential for multimodal tasks
is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build
LaViDa by equipping DMs with a vision encoder and jointly fine-tune the
combined parts for multimodal instruction following. To address challenges
encountered, LaViDa incorporates novel techniques such as complementary masking
for effective training, prefix KV cache for efficient inference, and timestep
shifting for high-quality sampling. Experiments show that LaViDa achieves
competitive or superior performance to AR VLMs on multi-modal benchmarks such
as MMMU, while offering unique advantages of DMs, including flexible
speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO
captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x
speedup. On bidirectional tasks, it achieves +59% improvement on Constrained
Poem Completion. These results demonstrate LaViDa as a strong alternative to AR
VLMs. Code and models will be released in the camera-ready version.
682fd5758d2fd6fc7cd5c9f7
Shufan Li
claimed_verified
2025-05-26T08:16:54.044Z
6310531914aa81e1044363ed
/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
Shufan Li
user
jacklishufan
682fd5758d2fd6fc7cd5c9f8
Konstantinos Kallidromitis
682fd5758d2fd6fc7cd5c9f9
Hritik Bansal
682fd5758d2fd6fc7cd5c9fa
Akash Gokul
682fd5758d2fd6fc7cd5c9fb
Yusuke Kato
682fd5758d2fd6fc7cd5c9fc
Kazuki Kozuka
682fd5758d2fd6fc7cd5c9fd
Jason Kuen
682fd5758d2fd6fc7cd5c9fe
Zhe Lin
682fd5758d2fd6fc7cd5c9ff
Kai-Wei Chang
682fd5758d2fd6fc7cd5ca00
Aditya Grover
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
Modern Vision-Language Models (VLMs) can solve a wide range of tasks
requiring visual reasoning. In real-world scenarios, desirable properties for
VLMs include fast inference and controllable generation (e.g., constraining
outputs to adhere to a desired format). However, existing autoregressive (AR)
VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs)
offer a promising alternative, enabling parallel decoding for faster inference
and bidirectional context for controllable generation through text-infilling.
While effective in language-only settings, DMs' potential for multimodal tasks
is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build
LaViDa by equipping DMs with a vision encoder and jointly fine-tune the
combined parts for multimodal instruction following. To address challenges
encountered, LaViDa incorporates novel techniques such as complementary masking
for effective training, prefix KV cache for efficient inference, and timestep
shifting for high-quality sampling. Experiments show that LaViDa achieves
competitive or superior performance to AR VLMs on multi-modal benchmarks such
as MMMU, while offering unique advantages of DMs, including flexible
speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO
captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x
speedup. On bidirectional tasks, it achieves +59% improvement on Constrained
Poem Completion. These results demonstrate LaViDa as a strong alternative to AR
VLMs. Code and models will be released in the camera-ready version.
LaViDa, a family of vision-language models built on discrete diffusion models, offers competitive performance on multimodal benchmarks with advantages in speed, controllability, and bidirectional reasoning.
autoregressive (AR) VLMs discrete diffusion models (DMs) parallel decoding bidirectional context text-infilling multimodal instruction following complementary masking prefix KV cache timestep shifting MMMU COCO captioning Constrained Poem Completion Open-LLaVa-Next-8B CIDEr