title: DiTFastAttn: Attention Compression for Diffusion Transformer Models
summary: Diffusion Transformers (DiT) excel at image and video generation but face
  computational challenges due to self-attention's quadratic complexity. We
  propose DiTFastAttn, a novel post-training compression method to alleviate
  DiT's computational bottleneck. We identify three key redundancies in the
  attention computation during DiT inference: 1. spatial redundancy, where many
  attention heads focus on local information; 2. temporal redundancy, with high
  similarity between neighboring steps' attention outputs; 3. conditional
  redundancy, where conditional and unconditional inferences exhibit significant
  similarity. To tackle these redundancies, we propose three techniques: 1.
  Window Attention with Residual Caching to reduce spatial redundancy; 2.
  Temporal Similarity Reduction to exploit the similarity between steps; 3.
  Conditional Redundancy Elimination to skip redundant computations during
  conditional generation. To demonstrate the effectiveness of DiTFastAttn, we
  apply it to DiT, PixArt-Sigma for image generation tasks, and OpenSora for
  video generation tasks. Evaluation results show that for image generation, our
  method reduces up to 88\% of the FLOPs and achieves up to 1.6x speedup at high
  resolution generation.
paper_authors._id: 666ba072fd6d120d6be23800
paper_authors.name: Zhihang Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:23:10.971Z
paper_authors._id: 63f489c3cfd7ba6e26f3f929
paper_authors.avatarUrl: /avatars/bcef9e49974debbcef66390dc096af60.svg
paper_authors.fullname: Zhihang Yuan
paper_authors.type: user
paper_authors.user: hahnyuan
paper_authors._id: 666ba072fd6d120d6be23801
paper_authors.name: Pu Lu
paper_authors._id: 666ba072fd6d120d6be23802
paper_authors.name: Hanling Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:02:13.406Z
paper_authors._id: 6599c0078982abaa5cde36c8
paper_authors.avatarUrl: /avatars/a94202b7f735eb43baa25e5ac0234be1.svg
paper_authors.fullname: Hanling Zhang
paper_authors.type: user
paper_authors.user: p2o6e100
paper_authors._id: 666ba072fd6d120d6be23803
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T11:47:16.980Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 666ba072fd6d120d6be23804
paper_authors.name: Linfeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:23:32.744Z
paper_authors._id: 642ec9831d1737803dc1c30a
paper_authors.avatarUrl: /avatars/c9ded838bad09004c15a27200e66a108.svg
paper_authors.fullname: linfeng zhang
paper_authors.type: user
paper_authors.user: linfengZ
paper_authors._id: 666ba072fd6d120d6be23805
paper_authors.name: Tianchen Zhao
paper_authors._id: 666ba072fd6d120d6be23806
paper_authors.name: Shengen Yan
paper_authors._id: 666ba072fd6d120d6be23807
paper_authors.name: Guohao Dai
paper_authors._id: 666ba072fd6d120d6be23808
paper_authors.name: Yu Wang
paper_title: DiTFastAttn: Attention Compression for Diffusion Transformer Models
paper_summary: Diffusion Transformers (DiT) excel at image and video generation but face
  computational challenges due to self-attention's quadratic complexity. We
  propose DiTFastAttn, a novel post-training compression method to alleviate
  DiT's computational bottleneck. We identify three key redundancies in the
  attention computation during DiT inference: 1. spatial redundancy, where many
  attention heads focus on local information; 2. temporal redundancy, with high
  similarity between neighboring steps' attention outputs; 3. conditional
  redundancy, where conditional and unconditional inferences exhibit significant
  similarity. To tackle these redundancies, we propose three techniques: 1.
  Window Attention with Residual Caching to reduce spatial redundancy; 2.
  Temporal Similarity Reduction to exploit the similarity between steps; 3.
  Conditional Redundancy Elimination to skip redundant computations during
  conditional generation. To demonstrate the effectiveness of DiTFastAttn, we
  apply it to DiT, PixArt-Sigma for image generation tasks, and OpenSora for
  video generation tasks. Evaluation results show that for image generation, our
  method reduces up to 88\% of the FLOPs and achieves up to 1.6x speedup at high
  resolution generation.
paper_ai_summary: DiTFastAttn reduces computational complexity in Diffusion Transformers by addressing spatial, temporal, and conditional redundancies.
paper_ai_keywords: Diffusion Transformers DiT self-attention quadratic complexity post-training compression Window Attention with Residual Caching Temporal Similarity Reduction Conditional Redundancy Elimination FLOPs speedup high resolution generation PixArt-Sigma OpenSora