title: Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long
    Video Generation
summary: Sora has unveiled the immense potential of the Diffusion Transformer (DiT)
  architecture in single-scene video generation. However, the more challenging
  task of multi-scene video generation, which offers broader applications,
  remains relatively underexplored. To bridge this gap, we propose Mask^2DiT, a
  novel approach that establishes fine-grained, one-to-one alignment between
  video segments and their corresponding text annotations. Specifically, we
  introduce a symmetric binary mask at each attention layer within the DiT
  architecture, ensuring that each text annotation applies exclusively to its
  respective video segment while preserving temporal coherence across visual
  tokens. This attention mechanism enables precise segment-level
  textual-to-visual alignment, allowing the DiT architecture to effectively
  handle video generation tasks with a fixed number of scenes. To further equip
  the DiT architecture with the ability to generate additional scenes based on
  existing ones, we incorporate a segment-level conditional mask, which
  conditions each newly generated segment on the preceding video segments,
  thereby enabling auto-regressive scene extension. Both qualitative and
  quantitative experiments confirm that Mask^2DiT excels in maintaining visual
  consistency across segments while ensuring semantic alignment between each
  segment and its corresponding text description. Our project page is
  https://tianhao-qi.github.io/Mask2DiTProject.
paper_authors._id: 67e3e1fc50e42a17c472be23
paper_authors.name: Tianhao Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:43:57.893Z
paper_authors._id: 6371b9bb3d1bd47a4ec73ec5
paper_authors.avatarUrl: /avatars/ed92874a85efcd9ea487a5b59959eb46.svg
paper_authors.fullname: Tianhao Qi
paper_authors.type: user
paper_authors.user: qth
paper_authors._id: 67e3e1fc50e42a17c472be24
paper_authors.name: Jianlong Yuan
paper_authors._id: 67e3e1fc50e42a17c472be25
paper_authors.name: Wanquan Feng
paper_authors._id: 67e3e1fc50e42a17c472be26
paper_authors.name: Shancheng Fang
paper_authors._id: 67e3e1fc50e42a17c472be27
paper_authors.name: Jiawei Liu
paper_authors._id: 67e3e1fc50e42a17c472be28
paper_authors.name: SiYu Zhou
paper_authors._id: 67e3e1fc50e42a17c472be29
paper_authors.name: Qian He
paper_authors._id: 67e3e1fc50e42a17c472be2a
paper_authors.name: Hongtao Xie
paper_authors._id: 67e3e1fc50e42a17c472be2b
paper_authors.name: Yongdong Zhang
paper_title: Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long
    Video Generation
paper_summary: Sora has unveiled the immense potential of the Diffusion Transformer (DiT)
  architecture in single-scene video generation. However, the more challenging
  task of multi-scene video generation, which offers broader applications,
  remains relatively underexplored. To bridge this gap, we propose Mask^2DiT, a
  novel approach that establishes fine-grained, one-to-one alignment between
  video segments and their corresponding text annotations. Specifically, we
  introduce a symmetric binary mask at each attention layer within the DiT
  architecture, ensuring that each text annotation applies exclusively to its
  respective video segment while preserving temporal coherence across visual
  tokens. This attention mechanism enables precise segment-level
  textual-to-visual alignment, allowing the DiT architecture to effectively
  handle video generation tasks with a fixed number of scenes. To further equip
  the DiT architecture with the ability to generate additional scenes based on
  existing ones, we incorporate a segment-level conditional mask, which
  conditions each newly generated segment on the preceding video segments,
  thereby enabling auto-regressive scene extension. Both qualitative and
  quantitative experiments confirm that Mask^2DiT excels in maintaining visual
  consistency across segments while ensuring semantic alignment between each
  segment and its corresponding text description. Our project page is
  https://tianhao-qi.github.io/Mask2DiTProject.
paper_ai_summary: Mask$^2$DiT enhances Diffusion Transformer architecture for multi-scene video generation by ensuring precise alignment between text annotations and video segments.
paper_ai_keywords: Diffusion Transformer Mask$^2$DiT attention layer binary mask symmetric binary mask temporal coherence textual-to-visual alignment segment-level scene extension conditional mask auto-regressive scene extension visual consistency semantic alignment