title: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers
    Up
summary: Diffusion Transformers (DiT) have become a leading architecture in image
  generation. However, the quadratic complexity of attention mechanisms, which
  are responsible for modeling token-wise relationships, results in significant
  latency when generating high-resolution images. To address this issue, we aim
  at a linear attention mechanism in this paper that reduces the complexity of
  pre-trained DiTs to linear. We begin our exploration with a comprehensive
  summary of existing efficient attention mechanisms and identify four key
  factors crucial for successful linearization of pre-trained DiTs: locality,
  formulation consistency, high-rank attention maps, and feature integrity. Based
  on these insights, we introduce a convolution-like local attention strategy
  termed CLEAR, which limits feature interactions to a local window around each
  query token, and thus achieves linear complexity. Our experiments indicate
  that, by fine-tuning the attention layer on merely 10K self-generated samples
  for 10K iterations, we can effectively transfer knowledge from a pre-trained
  DiT to a student model with linear complexity, yielding results comparable to
  the teacher model. Simultaneously, it reduces attention computations by 99.5%
  and accelerates generation by 6.3 times for generating 8K-resolution images.
  Furthermore, we investigate favorable properties in the distilled attention
  layers, such as zero-shot generalization cross various models and plugins, and
  improved support for multi-GPU parallel inference. Models and codes are
  available here: https://github.com/Huage001/CLEAR.
paper_authors._id: 6768d7f797a8f966b3362aa6
paper_authors.name: Songhua Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T16:41:34.617Z
paper_authors._id: 64b929308b53fb5dbd059ce3
paper_authors.avatarUrl: /avatars/564c44cf45db794747a96f79f30ecd91.svg
paper_authors.fullname: Liu Songhua
paper_authors.type: user
paper_authors.user: Huage001
paper_authors._id: 6768d7f797a8f966b3362aa7
paper_authors.name: Zhenxiong Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:32:44.224Z
paper_authors._id: 674e743be91289226ef9e857
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/FnLwJTTeItGbAUg4IPr7l.jpeg
paper_authors.fullname: 唐振雄
paper_authors.type: user
paper_authors.user: ZhenxiongTang
paper_authors._id: 6768d7f797a8f966b3362aa8
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:33:21.470Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers
    Up
paper_summary: Diffusion Transformers (DiT) have become a leading architecture in image
  generation. However, the quadratic complexity of attention mechanisms, which
  are responsible for modeling token-wise relationships, results in significant
  latency when generating high-resolution images. To address this issue, we aim
  at a linear attention mechanism in this paper that reduces the complexity of
  pre-trained DiTs to linear. We begin our exploration with a comprehensive
  summary of existing efficient attention mechanisms and identify four key
  factors crucial for successful linearization of pre-trained DiTs: locality,
  formulation consistency, high-rank attention maps, and feature integrity. Based
  on these insights, we introduce a convolution-like local attention strategy
  termed CLEAR, which limits feature interactions to a local window around each
  query token, and thus achieves linear complexity. Our experiments indicate
  that, by fine-tuning the attention layer on merely 10K self-generated samples
  for 10K iterations, we can effectively transfer knowledge from a pre-trained
  DiT to a student model with linear complexity, yielding results comparable to
  the teacher model. Simultaneously, it reduces attention computations by 99.5%
  and accelerates generation by 6.3 times for generating 8K-resolution images.
  Furthermore, we investigate favorable properties in the distilled attention
  layers, such as zero-shot generalization cross various models and plugins, and
  improved support for multi-GPU parallel inference. Models and codes are
  available here: https://github.com/Huage001/CLEAR.
paper_ai_keywords: Diffusion Transformers (DiT) quadratic complexity attention mechanisms token-wise relationships latency linear attention mechanism pre-trained DiTs efficient attention mechanisms locality formulation consistency high-rank attention maps feature integrity convolution-like local attention strategy CLEAR fine-tuning self-generated samples linear complexity knowledge transfer student model zero-shot generalization multi-GPU parallel inference