title: LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
summary: In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large
  Language Model (MLLM) that integrates visual instruction tuning with masked
  diffusion models, representing a departure from the autoregressive paradigms
  dominant in current multimodal approaches. Built upon LLaDA, a representative
  large language diffusion model, LLaDA-V incorporates a vision encoder and MLP
  connector that projects visual features into the language embedding space,
  enabling effective multimodal alignment. Our empirical investigation reveals
  several intriguing results: First, LLaDA-V demonstrates promising multimodal
  performance despite its language model being weaker on purely textual tasks
  than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same
  instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal
  tasks with better data scalability. It also narrows the performance gap to
  Qwen2-VL, suggesting the effectiveness of its architecture for multimodal
  tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal
  understanding compared to existing hybrid autoregressive-diffusion and purely
  diffusion-based MLLMs. Our findings suggest that large language diffusion
  models show promise in multimodal contexts and warrant further investigation in
  future research. Project page and codes:
  https://ml-gsai.github.io/LLaDA-V-demo/.
paper_authors._id: 682fe37bb998c9f79463b563
paper_authors.name: Zebin You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:36:23.049Z
paper_authors._id: 624f909eac5dd186b01ac3f5
paper_authors.avatarUrl: /avatars/0aafdb1cbb492fda52a0303031cc6c14.svg
paper_authors.fullname: Zebin You
paper_authors.type: user
paper_authors.user: yyyou
paper_authors._id: 682fe37bb998c9f79463b564
paper_authors.name: Shen Nie
paper_authors._id: 682fe37bb998c9f79463b565
paper_authors.name: Xiaolu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:36:49.675Z
paper_authors._id: 67513d6d3b8586521cda5d76
paper_authors.avatarUrl: /avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
paper_authors.fullname: Xiaolu  Zhang
paper_authors.type: user
paper_authors.user: xiaolu0714
paper_authors._id: 682fe37bb998c9f79463b566
paper_authors.name: Jun Hu
paper_authors._id: 682fe37bb998c9f79463b567
paper_authors.name: Jun Zhou
paper_authors._id: 682fe37bb998c9f79463b568
paper_authors.name: Zhiwu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:37:09.401Z
paper_authors._id: 6351e3035b904878f1d36719
paper_authors.avatarUrl: /avatars/a7b3a5faacaa483fab1a5b1b95bcc0a4.svg
paper_authors.fullname: Zhiwu Lu
paper_authors.type: user
paper_authors.user: zewlu
paper_authors._id: 682fe37bb998c9f79463b569
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:37:03.209Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_authors._id: 682fe37bb998c9f79463b56a
paper_authors.name: Chongxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:36:56.958Z
paper_authors._id: 64c07b488e2612254361153b
paper_authors.avatarUrl: /avatars/ade0f783cc4c2d3e73f402637f595471.svg
paper_authors.fullname: chongxuan li
paper_authors.type: user
paper_authors.user: zhenxuan00
paper_title: LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
paper_summary: In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large
  Language Model (MLLM) that integrates visual instruction tuning with masked
  diffusion models, representing a departure from the autoregressive paradigms
  dominant in current multimodal approaches. Built upon LLaDA, a representative
  large language diffusion model, LLaDA-V incorporates a vision encoder and MLP
  connector that projects visual features into the language embedding space,
  enabling effective multimodal alignment. Our empirical investigation reveals
  several intriguing results: First, LLaDA-V demonstrates promising multimodal
  performance despite its language model being weaker on purely textual tasks
  than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same
  instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal
  tasks with better data scalability. It also narrows the performance gap to
  Qwen2-VL, suggesting the effectiveness of its architecture for multimodal
  tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal
  understanding compared to existing hybrid autoregressive-diffusion and purely
  diffusion-based MLLMs. Our findings suggest that large language diffusion
  models show promise in multimodal contexts and warrant further investigation in
  future research. Project page and codes:
  https://ml-gsai.github.io/LLaDA-V-demo/.
paper_ai_summary: A diffusion-based Multimodal Large Language Model (LLaDA-V) with integrated visual instruction tuning performs competitively on multimodal tasks and outperforms existing models in multimodal understanding.
paper_ai_keywords: diffusion-based Multimodal Large Language Model (MLLM) visual instruction tuning masked diffusion models autoregressive paradigms vision encoder MLP connector language embedding space multimodal performance LLaDA LLaMA3-8B Qwen2-7B LLaMA3-V Qwen2-VL multimodal understanding hybrid autoregressive-diffusion