title: Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens
    for Text-to-Any-Task
summary: The global self-attention mechanism in diffusion transformers involves
  redundant computation due to the sparse and redundant nature of visual
  information, and the attention map of tokens within a spatial window shows
  significant similarity. To address this redundancy, we propose the Proxy Token
  Diffusion Transformer (PT-DiT), which employs sparse representative token
  attention (where the number of representative tokens is much smaller than the
  total number of tokens) to model global visual information efficiently.
  Specifically, in each transformer block, we randomly sample one token from each
  spatial-temporal window to serve as a proxy token for that region. The global
  semantics are captured through the self-attention of these proxy tokens and
  then injected into all latent tokens via cross-attention. Simultaneously, we
  introduce window and shift window attention to address the limitations in
  detail modeling caused by the sparse attention mechanism. Building on the
  well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a
  variety of models for T2I, T2V, and T2MV tasks. Experimental results show that
  PT-DiT achieves competitive performance while reducing the computational
  complexity in both image and video generation tasks (e.g., a 48% reduction
  compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code
  is available at https://github.com/360CVGroup/Qihoo-T2X.
paper_authors._id: 66de667dfd7d68a2932da0c6
paper_authors.name: Jing Wang
paper_authors._id: 66de667dfd7d68a2932da0c7
paper_authors.name: Ao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T08:02:08.111Z
paper_authors._id: 65292034825f4bba46c78581
paper_authors.avatarUrl: /avatars/7f212daaa20ab0d7405e9d6351ec308c.svg
paper_authors.fullname: Ao Ma
paper_authors.type: user
paper_authors.user: AoMa
paper_authors._id: 66de667dfd7d68a2932da0c8
paper_authors.name: Jiasong Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T08:37:59.995Z
paper_authors._id: 65542ae4520d9bb9d9d21659
paper_authors.avatarUrl: /avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
paper_authors.fullname: js
paper_authors.type: user
paper_authors.user: feng8
paper_authors._id: 66de667dfd7d68a2932da0c9
paper_authors.name: Dawei Leng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-18T02:23:14.520Z
paper_authors._id: 649935abbe8fd92c27ab1ed8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
paper_authors.fullname: David Leon
paper_authors.type: user
paper_authors.user: DavidLeon
paper_authors._id: 66de667dfd7d68a2932da0ca
paper_authors.name: Yuhui Yin
paper_authors._id: 66de667dfd7d68a2932da0cb
paper_authors.name: Xiaodan Liang
paper_title: Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens
    for Text-to-Any-Task
paper_summary: The global self-attention mechanism in diffusion transformers involves
  redundant computation due to the sparse and redundant nature of visual
  information, and the attention map of tokens within a spatial window shows
  significant similarity. To address this redundancy, we propose the Proxy Token
  Diffusion Transformer (PT-DiT), which employs sparse representative token
  attention (where the number of representative tokens is much smaller than the
  total number of tokens) to model global visual information efficiently.
  Specifically, in each transformer block, we randomly sample one token from each
  spatial-temporal window to serve as a proxy token for that region. The global
  semantics are captured through the self-attention of these proxy tokens and
  then injected into all latent tokens via cross-attention. Simultaneously, we
  introduce window and shift window attention to address the limitations in
  detail modeling caused by the sparse attention mechanism. Building on the
  well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a
  variety of models for T2I, T2V, and T2MV tasks. Experimental results show that
  PT-DiT achieves competitive performance while reducing the computational
  complexity in both image and video generation tasks (e.g., a 48% reduction
  compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code
  is available at https://github.com/360CVGroup/Qihoo-T2X.
paper_ai_keywords: global self-attention mechanism diffusion transformers attention map tokens spatial window Proxy Token Diffusion Transformer (PT-DiT) sparse representative token attention proxy tokens latent tokens cross-attention window and shift window attention Qihoo-T2X family T2I T2V T2MV image generation video generation DiT Pixart-alpha