PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference
  Time by Leveraging Sparsity
Diffusion models have shown impressive results in generating high-quality
conditional samples using guidance techniques such as Classifier-Free Guidance
(CFG). However, existing methods often require additional training or neural
function evaluations (NFEs), making them incompatible with guidance-distilled
models. Also, they rely on heuristic approaches that need identifying target
layers. In this work, we propose a novel and efficient method, termed PLADIS,
which boosts pre-trained models (U-Net/Transformer) by leveraging sparse
attention. Specifically, we extrapolate query-key correlations using softmax
and its sparse counterpart in the cross-attention layer during inference,
without requiring extra training or NFEs. By leveraging the noise robustness of
sparse attention, our PLADIS unleashes the latent potential of text-to-image
diffusion models, enabling them to excel in areas where they once struggled
with newfound effectiveness. It integrates seamlessly with guidance techniques,
including guidance-distilled models. Extensive experiments show notable
improvements in text alignment and human preference, offering a highly
efficient and universally applicable solution.
67d2ca0767366130cccad93d
Kwanyoung Kim
claimed_verified
2025-03-14T08:58:03.528Z
63973ee44e7b4959dc98028f
/avatars/2e166fee60844729479bfa4291796c8a.svg
Kwanyoung
user
kwanyoung
67d2ca0767366130cccad93e
Byeongsu Sim
admin_assigned
2025-03-17T08:47:11.835Z
668377232d89090894bea7b4
/avatars/1a74a08d645a352db4a460036b9fb6db.svg
byeongsu sim
user
byeongsus
PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference
  Time by Leveraging Sparsity
Diffusion models have shown impressive results in generating high-quality
conditional samples using guidance techniques such as Classifier-Free Guidance
(CFG). However, existing methods often require additional training or neural
function evaluations (NFEs), making them incompatible with guidance-distilled
models. Also, they rely on heuristic approaches that need identifying target
layers. In this work, we propose a novel and efficient method, termed PLADIS,
which boosts pre-trained models (U-Net/Transformer) by leveraging sparse
attention. Specifically, we extrapolate query-key correlations using softmax
and its sparse counterpart in the cross-attention layer during inference,
without requiring extra training or NFEs. By leveraging the noise robustness of
sparse attention, our PLADIS unleashes the latent potential of text-to-image
diffusion models, enabling them to excel in areas where they once struggled
with newfound effectiveness. It integrates seamlessly with guidance techniques,
including guidance-distilled models. Extensive experiments show notable
improvements in text alignment and human preference, offering a highly
efficient and universally applicable solution.
PLADIS leverages sparse attention in cross-attention layers to enhance pre-trained text-to-image diffusion models, improving text alignment and human preference without additional training.
diffusion models Classifier-Free Guidance (CFG) guidance-distilled models neural function evaluations (NFEs) PLADIS U-Net Transformer sparse attention softmax cross-attention layer text-to-image