Beyond Fixed: Variable-Length Denoising for Diffusion Large Language
  Models
Diffusion Large Language Models (DLLMs) are emerging as a powerful
alternative to the dominant Autoregressive Large Language Models, offering
efficient parallel generation and capable global context modeling. However, the
practical application of DLLMs is hindered by a critical architectural
constraint: the need for a statically predefined generation length. This static
length allocation leads to a problematic trade-off: insufficient lengths
cripple performance on complex tasks, while excessive lengths incur significant
computational overhead and sometimes result in performance degradation. While
the inference framework is rigid, we observe that the model itself possesses
internal signals that correlate with the optimal response length for a given
task. To bridge this gap, we leverage these latent signals and introduce
DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive
Length Expansion for Diffusion Large Language Models. DAEDAL operates in two
phases: 1) Before the denoising process, DAEDAL starts from a short initial
length and iteratively expands it to a coarse task-appropriate length, guided
by a sequence completion metric. 2) During the denoising process, DAEDAL
dynamically intervenes by pinpointing and expanding insufficient generation
regions through mask token insertion, ensuring the final output is fully
developed. Extensive experiments on DLLMs demonstrate that DAEDAL achieves
performance comparable, and in some cases superior, to meticulously tuned
fixed-length baselines, while simultaneously enhancing computational efficiency
by achieving a higher effective token ratio. By resolving the static length
constraint, DAEDAL unlocks new potential for DLLMs, bridging a critical gap
with their Autoregressive counterparts and paving the way for more efficient
and capable generation.
689020e10a411b3b8d28d67f
Jinsong Li
689020e10a411b3b8d28d680
Xiaoyi Dong
689020e10a411b3b8d28d681
Yuhang Zang
689020e10a411b3b8d28d682
Yuhang Cao
689020e10a411b3b8d28d683
Jiaqi Wang
689020e10a411b3b8d28d684
Dahua Lin
Beyond Fixed: Variable-Length Denoising for Diffusion Large Language
  Models
Diffusion Large Language Models (DLLMs) are emerging as a powerful
alternative to the dominant Autoregressive Large Language Models, offering
efficient parallel generation and capable global context modeling. However, the
practical application of DLLMs is hindered by a critical architectural
constraint: the need for a statically predefined generation length. This static
length allocation leads to a problematic trade-off: insufficient lengths
cripple performance on complex tasks, while excessive lengths incur significant
computational overhead and sometimes result in performance degradation. While
the inference framework is rigid, we observe that the model itself possesses
internal signals that correlate with the optimal response length for a given
task. To bridge this gap, we leverage these latent signals and introduce
DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive
Length Expansion for Diffusion Large Language Models. DAEDAL operates in two
phases: 1) Before the denoising process, DAEDAL starts from a short initial
length and iteratively expands it to a coarse task-appropriate length, guided
by a sequence completion metric. 2) During the denoising process, DAEDAL
dynamically intervenes by pinpointing and expanding insufficient generation
regions through mask token insertion, ensuring the final output is fully
developed. Extensive experiments on DLLMs demonstrate that DAEDAL achieves
performance comparable, and in some cases superior, to meticulously tuned
fixed-length baselines, while simultaneously enhancing computational efficiency
by achieving a higher effective token ratio. By resolving the static length
constraint, DAEDAL unlocks new potential for DLLMs, bridging a critical gap
with their Autoregressive counterparts and paving the way for more efficient
and capable generation.
DAEDAL, a novel training-free denoising strategy, enables dynamic length adaptation in Diffusion Large Language Models, improving performance and computational efficiency.
Diffusion Large Language Models DLLMs Autoregressive Large Language Models denoising strategy Dynamic Adaptive Length Expansion sequence completion metric mask token insertion effective token ratio