title: Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel
    Decoding
summary: In this work, we propose Dimple, the first Discrete Diffusion Multimodal
  Large Language Model (DMLLM). We observe that training with a purely discrete
  diffusion approach leads to significant training instability, suboptimal
  performance, and severe length bias issues. To address these challenges, we
  design a novel training paradigm that combines an initial autoregressive phase
  with a subsequent diffusion phase. This approach yields the Dimple-7B model,
  trained on the same dataset and using a similar training pipeline as
  LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%,
  demonstrating that DMLLM can achieve performance comparable to that of
  autoregressive models. To improve inference efficiency, we propose a decoding
  strategy termed confident decoding, which dynamically adjusts the number of
  tokens generated at each step, significantly reducing the number of generation
  iterations. In autoregressive models, the number of forward iterations during
  generation equals the response length. With confident decoding, however, the
  number of iterations needed by Dimple is even only text{response
  length}{3}. We also re-implement the prefilling technique in autoregressive
  models and demonstrate that it does not significantly impact performance on
  most benchmark evaluations, while offering a speedup of 1.5x to 7x.
  Additionally, we explore Dimple's capability to precisely control its response
  using structure priors. These priors enable structured responses in a manner
  distinct from instruction-based or chain-of-thought prompting, and allow
  fine-grained control over response format and length, which is difficult to
  achieve in autoregressive models. Overall, this work validates the feasibility
  and advantages of DMLLM and enhances its inference efficiency and
  controllability. Code and models are available at
  https://github.com/yu-rp/Dimple.
paper_authors._id: 682fdd034640a9db4d1cc04d
paper_authors.name: Runpeng Yu
paper_authors._id: 682fdd034640a9db4d1cc04e
paper_authors.name: Xinyin Ma
paper_authors._id: 682fdd034640a9db4d1cc04f
paper_authors.name: Xinchao Wang
paper_title: Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel
    Decoding
paper_summary: In this work, we propose Dimple, the first Discrete Diffusion Multimodal
  Large Language Model (DMLLM). We observe that training with a purely discrete
  diffusion approach leads to significant training instability, suboptimal
  performance, and severe length bias issues. To address these challenges, we
  design a novel training paradigm that combines an initial autoregressive phase
  with a subsequent diffusion phase. This approach yields the Dimple-7B model,
  trained on the same dataset and using a similar training pipeline as
  LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%,
  demonstrating that DMLLM can achieve performance comparable to that of
  autoregressive models. To improve inference efficiency, we propose a decoding
  strategy termed confident decoding, which dynamically adjusts the number of
  tokens generated at each step, significantly reducing the number of generation
  iterations. In autoregressive models, the number of forward iterations during
  generation equals the response length. With confident decoding, however, the
  number of iterations needed by Dimple is even only text{response
  length}{3}. We also re-implement the prefilling technique in autoregressive
  models and demonstrate that it does not significantly impact performance on
  most benchmark evaluations, while offering a speedup of 1.5x to 7x.
  Additionally, we explore Dimple's capability to precisely control its response
  using structure priors. These priors enable structured responses in a manner
  distinct from instruction-based or chain-of-thought prompting, and allow
  fine-grained control over response format and length, which is difficult to
  achieve in autoregressive models. Overall, this work validates the feasibility
  and advantages of DMLLM and enhances its inference efficiency and
  controllability. Code and models are available at
  https://github.com/yu-rp/Dimple.
paper_ai_summary: Dimple, a Discrete Diffusion Multimodal Large Language Model, achieves performance comparable to autoregressive models through a hybrid training approach and enhances inference efficiency with confident decoding and structure priors.
paper_ai_keywords: Discrete Diffusion Multimodal Large Language Model DMLLM autoregressive phase diffusion phase confident decoding prefilling technique structure priors