title: LaViDa: A Large Diffusion Language Model for Multimodal Understanding
summary: Modern Vision-Language Models (VLMs) can solve a wide range of tasks
  requiring visual reasoning. In real-world scenarios, desirable properties for
  VLMs include fast inference and controllable generation (e.g., constraining
  outputs to adhere to a desired format). However, existing autoregressive (AR)
  VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs)
  offer a promising alternative, enabling parallel decoding for faster inference
  and bidirectional context for controllable generation through text-infilling.
  While effective in language-only settings, DMs' potential for multimodal tasks
  is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build
  LaViDa by equipping DMs with a vision encoder and jointly fine-tune the
  combined parts for multimodal instruction following. To address challenges
  encountered, LaViDa incorporates novel techniques such as complementary masking
  for effective training, prefix KV cache for efficient inference, and timestep
  shifting for high-quality sampling. Experiments show that LaViDa achieves
  competitive or superior performance to AR VLMs on multi-modal benchmarks such
  as MMMU, while offering unique advantages of DMs, including flexible
  speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO
  captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x
  speedup. On bidirectional tasks, it achieves +59% improvement on Constrained
  Poem Completion. These results demonstrate LaViDa as a strong alternative to AR
  VLMs. Code and models will be released in the camera-ready version.
paper_authors._id: 682fd5758d2fd6fc7cd5c9f7
paper_authors.name: Shufan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:54.044Z
paper_authors._id: 6310531914aa81e1044363ed
paper_authors.avatarUrl: /avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
paper_authors.fullname: Shufan Li
paper_authors.type: user
paper_authors.user: jacklishufan
paper_authors._id: 682fd5758d2fd6fc7cd5c9f8
paper_authors.name: Konstantinos Kallidromitis
paper_authors._id: 682fd5758d2fd6fc7cd5c9f9
paper_authors.name: Hritik Bansal
paper_authors._id: 682fd5758d2fd6fc7cd5c9fa
paper_authors.name: Akash Gokul
paper_authors._id: 682fd5758d2fd6fc7cd5c9fb
paper_authors.name: Yusuke Kato
paper_authors._id: 682fd5758d2fd6fc7cd5c9fc
paper_authors.name: Kazuki Kozuka
paper_authors._id: 682fd5758d2fd6fc7cd5c9fd
paper_authors.name: Jason Kuen
paper_authors._id: 682fd5758d2fd6fc7cd5c9fe
paper_authors.name: Zhe Lin
paper_authors._id: 682fd5758d2fd6fc7cd5c9ff
paper_authors.name: Kai-Wei Chang
paper_authors._id: 682fd5758d2fd6fc7cd5ca00
paper_authors.name: Aditya Grover
paper_title: LaViDa: A Large Diffusion Language Model for Multimodal Understanding
paper_summary: Modern Vision-Language Models (VLMs) can solve a wide range of tasks
  requiring visual reasoning. In real-world scenarios, desirable properties for
  VLMs include fast inference and controllable generation (e.g., constraining
  outputs to adhere to a desired format). However, existing autoregressive (AR)
  VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs)
  offer a promising alternative, enabling parallel decoding for faster inference
  and bidirectional context for controllable generation through text-infilling.
  While effective in language-only settings, DMs' potential for multimodal tasks
  is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build
  LaViDa by equipping DMs with a vision encoder and jointly fine-tune the
  combined parts for multimodal instruction following. To address challenges
  encountered, LaViDa incorporates novel techniques such as complementary masking
  for effective training, prefix KV cache for efficient inference, and timestep
  shifting for high-quality sampling. Experiments show that LaViDa achieves
  competitive or superior performance to AR VLMs on multi-modal benchmarks such
  as MMMU, while offering unique advantages of DMs, including flexible
  speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO
  captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x
  speedup. On bidirectional tasks, it achieves +59% improvement on Constrained
  Poem Completion. These results demonstrate LaViDa as a strong alternative to AR
  VLMs. Code and models will be released in the camera-ready version.
paper_ai_summary: LaViDa, a family of vision-language models built on discrete diffusion models, offers competitive performance on multimodal benchmarks with advantages in speed, controllability, and bidirectional reasoning.
paper_ai_keywords: autoregressive (AR) VLMs discrete diffusion models (DMs) parallel decoding bidirectional context text-infilling multimodal instruction following complementary masking prefix KV cache timestep shifting MMMU COCO captioning Constrained Poem Completion Open-LLaVa-Next-8B CIDEr