title: Re-ttention: Ultra Sparse Visual Generation via Attention Statistical
    Reshape
summary: Diffusion Transformers (DiT) have become the de-facto model for generating
  high-quality visual content like videos and images. A huge bottleneck is the
  attention mechanism where complexity scales quadratically with resolution and
  video length. One logical way to lessen this burden is sparse attention, where
  only a subset of tokens or patches are included in the calculation. However,
  existing techniques fail to preserve visual quality at extremely high sparsity
  levels and might even incur non-negligible compute overheads. % To address this
  concern, we propose Re-ttention, which implements very high sparse attention
  for visual generation models by leveraging the temporal redundancy of Diffusion
  Models to overcome the probabilistic normalization shift within the attention
  mechanism. Specifically, Re-ttention reshapes attention scores based on the
  prior softmax distribution history in order to preserve the visual quality of
  the full quadratic attention at very high sparsity levels. % Experimental
  results on T2V/T2I models such as CogVideoX and the PixArt DiTs demonstrate
  that Re-ttention requires as few as 3.1\% of the tokens during inference,
  outperforming contemporary methods like FastDiTAttn, Sparse VideoGen and
  MInference. Further, we measure latency to show that our method can attain over
  45\% end-to-end % and over 92\% self-attention latency reduction on an H100 GPU
  at negligible overhead cost.
    Code available online here:
  https://github.com/cccrrrccc/Re-ttention{https://github.com/cccrrrccc/Re-ttention}
paper_authors._id: 68393b9ef0458c8bcb652d78
paper_authors.name: Ruichen Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-30T17:29:37.663Z
paper_authors._id: 6814e98abbe5b8a5d92fc335
paper_authors.avatarUrl: /avatars/83c1963280bc94fdc7a2b21f6e0f1c59.svg
paper_authors.fullname: Ruichen Chen
paper_authors.type: user
paper_authors.user: crc5577
paper_authors._id: 68393b9ef0458c8bcb652d79
paper_authors.name: Keith G. Mills
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-30T06:54:23.854Z
paper_authors._id: 661c391720b47b0daddfcc5a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RwvDJRtlEEPch27xId9Cb.png
paper_authors.fullname: Keith G. Mills
paper_authors.type: user
paper_authors.user: kgmills
paper_authors._id: 68393b9ef0458c8bcb652d7a
paper_authors.name: Liyao Jiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-30T05:01:59.348Z
paper_authors._id: 64ac49ccb7d86b40fd60a8dd
paper_authors.avatarUrl: /avatars/e9f5482cffdd1d5917523a496a3805f0.svg
paper_authors.fullname: Liyao Jiang
paper_authors.type: user
paper_authors.user: LiyaoJiang
paper_authors._id: 68393b9ef0458c8bcb652d7b
paper_authors.name: Chao Gao
paper_authors._id: 68393b9ef0458c8bcb652d7c
paper_authors.name: Di Niu
paper_title: Re-ttention: Ultra Sparse Visual Generation via Attention Statistical
    Reshape
paper_summary: Diffusion Transformers (DiT) have become the de-facto model for generating
  high-quality visual content like videos and images. A huge bottleneck is the
  attention mechanism where complexity scales quadratically with resolution and
  video length. One logical way to lessen this burden is sparse attention, where
  only a subset of tokens or patches are included in the calculation. However,
  existing techniques fail to preserve visual quality at extremely high sparsity
  levels and might even incur non-negligible compute overheads. % To address this
  concern, we propose Re-ttention, which implements very high sparse attention
  for visual generation models by leveraging the temporal redundancy of Diffusion
  Models to overcome the probabilistic normalization shift within the attention
  mechanism. Specifically, Re-ttention reshapes attention scores based on the
  prior softmax distribution history in order to preserve the visual quality of
  the full quadratic attention at very high sparsity levels. % Experimental
  results on T2V/T2I models such as CogVideoX and the PixArt DiTs demonstrate
  that Re-ttention requires as few as 3.1\% of the tokens during inference,
  outperforming contemporary methods like FastDiTAttn, Sparse VideoGen and
  MInference. Further, we measure latency to show that our method can attain over
  45\% end-to-end % and over 92\% self-attention latency reduction on an H100 GPU
  at negligible overhead cost.
    Code available online here:
  https://github.com/cccrrrccc/Re-ttention{https://github.com/cccrrrccc/Re-ttention}
paper_ai_summary: Re-ttention uses temporal redundancy in diffusion models to enable high sparse attention in visual generation, maintaining quality with minimal computational overhead.
paper_ai_keywords: Diffusion Transformers sparse attention visual generation models probabilistic normalization shift Re-ttention attention scores softmax distribution latency reduction H100 GPU