PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and
  Quantized Attention in Visual Generation Models
In visual generation, the quadratic complexity of attention mechanisms
results in high memory and computational costs, especially for longer token
sequences required in high-resolution image or multi-frame video generation. To
address this, prior research has explored techniques such as sparsification and
quantization. However, these techniques face significant challenges under low
density and reduced bitwidths. Through systematic analysis, we identify that
the core difficulty stems from the dispersed and irregular characteristics of
visual attention patterns. Therefore, instead of introducing specialized
sparsification and quantization design to accommodate such patterns, we propose
an alternative strategy: *reorganizing* the attention pattern to alleviate the
challenges. Inspired by the local aggregation nature of visual feature
extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)**
technique, which unifies the diverse attention patterns into a
hardware-friendly block-wise pattern. This unification substantially simplifies
and enhances both sparsification and quantization. We evaluate the
performance-efficiency trade-offs of various design choices and finalize a
methodology tailored for the unified pattern. Our approach, **PAROAttention**,
achieves video and image generation with lossless metrics, and nearly identical
results from full-precision (FP) baselines, while operating at notably lower
density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to
**2.7x** end-to-end latency speedup.
6858e225c0c8e29df8ea3d0f
Tianchen Zhao
claimed_verified
2025-06-23T08:14:13.425Z
6454568636821f6860fed410
https://cdn-avatars.huggingface.co/v1/production/uploads/6454568636821f6860fed410/VsKn0GawOBZo1hmqHnmV1.png
Tianchen Zhao
user
A-suozhang
6858e225c0c8e29df8ea3d10
Ke Hong
6858e225c0c8e29df8ea3d11
Xinhao Yang
6858e225c0c8e29df8ea3d12
Xuefeng Xiao
6858e225c0c8e29df8ea3d13
Huixia Li
6858e225c0c8e29df8ea3d14
Feng Ling
6858e225c0c8e29df8ea3d15
Ruiqi Xie
claimed_verified
2025-06-27T08:57:52.656Z
66f6442e04f2d5ae9789ab2b
/avatars/72beedaeda45311725aabe80a8ec91a5.svg
RIcky Xie
user
RickyHsieh
6858e225c0c8e29df8ea3d16
Siqi Chen
6858e225c0c8e29df8ea3d17
Hongyu Zhu
6858e225c0c8e29df8ea3d18
Yichong Zhang
6858e225c0c8e29df8ea3d19
Yu Wang
PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and
  Quantized Attention in Visual Generation Models
In visual generation, the quadratic complexity of attention mechanisms
results in high memory and computational costs, especially for longer token
sequences required in high-resolution image or multi-frame video generation. To
address this, prior research has explored techniques such as sparsification and
quantization. However, these techniques face significant challenges under low
density and reduced bitwidths. Through systematic analysis, we identify that
the core difficulty stems from the dispersed and irregular characteristics of
visual attention patterns. Therefore, instead of introducing specialized
sparsification and quantization design to accommodate such patterns, we propose
an alternative strategy: *reorganizing* the attention pattern to alleviate the
challenges. Inspired by the local aggregation nature of visual feature
extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)**
technique, which unifies the diverse attention patterns into a
hardware-friendly block-wise pattern. This unification substantially simplifies
and enhances both sparsification and quantization. We evaluate the
performance-efficiency trade-offs of various design choices and finalize a
methodology tailored for the unified pattern. Our approach, **PAROAttention**,
achieves video and image generation with lossless metrics, and nearly identical
results from full-precision (FP) baselines, while operating at notably lower
density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to
**2.7x** end-to-end latency speedup.
PAROAttention reorganizes visual attention patterns to enable efficient sparsification and quantization, reducing memory and computational costs with minimal impact on performance.
attention mechanisms sparsification quantization visual attention patterns Pattern-Aware token ReOrdering (PARO) local aggregation hardware-friendly block-wise pattern end-to-end latency speedup INT8/INT4 PAROAttention