Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and
parallel decoding but suffer from prohibitive quadratic computational
complexity and memory overhead during inference. Current caching techniques
accelerate decoding by storing full-layer states, yet impose substantial memory
usage that limit long-context applications. Our analysis of attention patterns
in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining
salient across decoding steps and low-relevance tokens staying unimportant,
motivating selective cache eviction. We propose Sparse-dLLM, the first
training-free framework integrating dynamic cache eviction with sparse
attention via delayed bidirectional sparse caching. By leveraging the stability
of token saliency over steps, it retains critical tokens and dynamically evicts
unimportant prefix/suffix entries using an attention-guided strategy. Extensive
experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to
10times higher throughput than vanilla dLLMs, with comparable performance
and similar peak memory costs, outperforming previous methods in efficiency and
effectiveness.
689229189dcf0e7ac9a66edf
Yuerong Song
689229189dcf0e7ac9a66ee0
Xiaoran Liu
689229189dcf0e7ac9a66ee1
Ruixiao Li
689229189dcf0e7ac9a66ee2
Zhigeng Liu
689229189dcf0e7ac9a66ee3
Zengfeng Huang
689229189dcf0e7ac9a66ee4
Qipeng Guo
689229189dcf0e7ac9a66ee5
Ziwei He
689229189dcf0e7ac9a66ee6
Xipeng Qiu
Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and
parallel decoding but suffer from prohibitive quadratic computational
complexity and memory overhead during inference. Current caching techniques
accelerate decoding by storing full-layer states, yet impose substantial memory
usage that limit long-context applications. Our analysis of attention patterns
in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining
salient across decoding steps and low-relevance tokens staying unimportant,
motivating selective cache eviction. We propose Sparse-dLLM, the first
training-free framework integrating dynamic cache eviction with sparse
attention via delayed bidirectional sparse caching. By leveraging the stability
of token saliency over steps, it retains critical tokens and dynamically evicts
unimportant prefix/suffix entries using an attention-guided strategy. Extensive
experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to
10times higher throughput than vanilla dLLMs, with comparable performance
and similar peak memory costs, outperforming previous methods in efficiency and
effectiveness.
Sparse-dLLM improves the efficiency of diffusion large language models by implementing dynamic cache eviction and sparse attention, enhancing throughput without compromising performance.
diffusion large language models dLLMs computational complexity memory overhead caching techniques attention patterns cross-layer sparsity pivotal tokens low-relevance tokens selective cache eviction sparse attention delayed bidirectional sparse caching token saliency throughput peak memory costs