title: FullDiT: Multi-Task Video Generative Foundation Model with Full
    Attention
summary: Current video generative foundation models primarily focus on text-to-video
  tasks, providing limited control for fine-grained video content creation.
  Although adapter-based approaches (e.g., ControlNet) enable additional controls
  with minimal fine-tuning, they encounter challenges when integrating multiple
  conditions, including: branch conflicts between independently trained adapters,
  parameter redundancy leading to increased computational cost, and suboptimal
  performance compared to full fine-tuning. To address these challenges, we
  introduce FullDiT, a unified foundation model for video generation that
  seamlessly integrates multiple conditions via unified full-attention
  mechanisms. By fusing multi-task conditions into a unified sequence
  representation and leveraging the long-context learning ability of full
  self-attention to capture condition dynamics, FullDiT reduces parameter
  overhead, avoids conditions conflict, and shows scalability and emergent
  ability. We further introduce FullBench for multi-task video generation
  evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
  results, highlighting the efficacy of full-attention in complex multi-task
  video generation.
paper_authors._id: 67e419a8e3471e8d9fa0b303
paper_authors.name: Xuan Ju
paper_authors._id: 67e419a8e3471e8d9fa0b304
paper_authors.name: Weicai Ye
paper_authors._id: 67e419a8e3471e8d9fa0b305
paper_authors.name: Quande Liu
paper_authors._id: 67e419a8e3471e8d9fa0b306
paper_authors.name: Qiulin Wang
paper_authors._id: 67e419a8e3471e8d9fa0b307
paper_authors.name: Xintao Wang
paper_authors._id: 67e419a8e3471e8d9fa0b308
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:15.350Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67e419a8e3471e8d9fa0b309
paper_authors.name: Di Zhang
paper_authors._id: 67e419a8e3471e8d9fa0b30a
paper_authors.name: Kun Gai
paper_authors._id: 67e419a8e3471e8d9fa0b30b
paper_authors.name: Qiang Xu
paper_title: FullDiT: Multi-Task Video Generative Foundation Model with Full
    Attention
paper_summary: Current video generative foundation models primarily focus on text-to-video
  tasks, providing limited control for fine-grained video content creation.
  Although adapter-based approaches (e.g., ControlNet) enable additional controls
  with minimal fine-tuning, they encounter challenges when integrating multiple
  conditions, including: branch conflicts between independently trained adapters,
  parameter redundancy leading to increased computational cost, and suboptimal
  performance compared to full fine-tuning. To address these challenges, we
  introduce FullDiT, a unified foundation model for video generation that
  seamlessly integrates multiple conditions via unified full-attention
  mechanisms. By fusing multi-task conditions into a unified sequence
  representation and leveraging the long-context learning ability of full
  self-attention to capture condition dynamics, FullDiT reduces parameter
  overhead, avoids conditions conflict, and shows scalability and emergent
  ability. We further introduce FullBench for multi-task video generation
  evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
  results, highlighting the efficacy of full-attention in complex multi-task
  video generation.
paper_ai_summary: FullDiT is a unified video generation model using full-attention mechanisms to effectively integrate multiple conditions, reducing parameter overhead and improving performance.
paper_ai_keywords: adapter-based approaches ControlNet branch conflicts parameter redundancy unified foundation model full-attention mechanisms sequence representation long-context learning full self-attention condition dynamics multi-task video generation FullBench state-of-the-art results