title: DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion
    Transformer for Tuning-Free Multi-Prompt Longer Video Generation
summary: Sora-like video generation models have achieved remarkable progress with a
  Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current
  video generation models predominantly focus on single-prompt, struggling to
  generate coherent scenes with multiple sequential prompts that better reflect
  real-world dynamic scenarios. While some pioneering works have explored
  multi-prompt video generation, they face significant challenges including
  strict training data requirements, weak prompt following, and unnatural
  transitions. To address these problems, we propose DiTCtrl, a training-free
  multi-prompt video generation method under MM-DiT architectures for the first
  time. Our key idea is to take the multi-prompt video generation task as
  temporal video editing with smooth transitions. To achieve this goal, we first
  analyze MM-DiT's attention mechanism, finding that the 3D full attention
  behaves similarly to that of the cross/self-attention blocks in the UNet-like
  diffusion models, enabling mask-guided precise semantic control across
  different prompts with attention sharing for multi-prompt video generation.
  Based on our careful design, the video generated by DiTCtrl achieves smooth
  transitions and consistent object motion given multiple sequential prompts
  without additional training. Besides, we also present MPVBench, a new benchmark
  specially designed for multi-prompt video generation to evaluate the
  performance of multi-prompt generation. Extensive experiments demonstrate that
  our method achieves state-of-the-art performance without additional training.
paper_authors._id: 676b9b876fb4876383b8591b
paper_authors.name: Minghong Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:32:49.066Z
paper_authors._id: 64f94370c3c12b377cc51086
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f94370c3c12b377cc51086/6CXcHhqAoykqXcShqM8Rd.jpeg
paper_authors.fullname: Minghong Cai
paper_authors.type: user
paper_authors.user: onevfall
paper_authors._id: 676b9b876fb4876383b8591c
paper_authors.name: Xiaodong Cun
paper_authors._id: 676b9b876fb4876383b8591d
paper_authors.name: Xiaoyu Li
paper_authors._id: 676b9b876fb4876383b8591e
paper_authors.name: Wenze Liu
paper_authors._id: 676b9b876fb4876383b8591f
paper_authors.name: Zhaoyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:12:28.534Z
paper_authors._id: 658409ceca19ccf6d9989add
paper_authors.avatarUrl: /avatars/3ac1dbd25e52435185babdeb3da28875.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZyZcuhk
paper_authors._id: 676b9b876fb4876383b85920
paper_authors.name: Yong Zhang
paper_authors._id: 676b9b876fb4876383b85921
paper_authors.name: Ying Shan
paper_authors._id: 676b9b876fb4876383b85922
paper_authors.name: Xiangyu Yue
paper_title: DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion
    Transformer for Tuning-Free Multi-Prompt Longer Video Generation
paper_summary: Sora-like video generation models have achieved remarkable progress with a
  Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current
  video generation models predominantly focus on single-prompt, struggling to
  generate coherent scenes with multiple sequential prompts that better reflect
  real-world dynamic scenarios. While some pioneering works have explored
  multi-prompt video generation, they face significant challenges including
  strict training data requirements, weak prompt following, and unnatural
  transitions. To address these problems, we propose DiTCtrl, a training-free
  multi-prompt video generation method under MM-DiT architectures for the first
  time. Our key idea is to take the multi-prompt video generation task as
  temporal video editing with smooth transitions. To achieve this goal, we first
  analyze MM-DiT's attention mechanism, finding that the 3D full attention
  behaves similarly to that of the cross/self-attention blocks in the UNet-like
  diffusion models, enabling mask-guided precise semantic control across
  different prompts with attention sharing for multi-prompt video generation.
  Based on our careful design, the video generated by DiTCtrl achieves smooth
  transitions and consistent object motion given multiple sequential prompts
  without additional training. Besides, we also present MPVBench, a new benchmark
  specially designed for multi-prompt video generation to evaluate the
  performance of multi-prompt generation. Extensive experiments demonstrate that
  our method achieves state-of-the-art performance without additional training.
paper_ai_keywords: Multi-Modal Diffusion Transformer MM-DiT video generation multi-prompt sequential prompts dynamic scenarios training data requirements prompt following unnatural transitions DiTCtrl temporal video editing attention mechanism 3D full attention cross/self-attention UNet-like diffusion models mask-guided precise semantic control attention sharing smooth transitions consistent object motion MPVBench benchmark