DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion
  Transformer for Tuning-Free Multi-Prompt Longer Video Generation
Sora-like video generation models have achieved remarkable progress with a
Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current
video generation models predominantly focus on single-prompt, struggling to
generate coherent scenes with multiple sequential prompts that better reflect
real-world dynamic scenarios. While some pioneering works have explored
multi-prompt video generation, they face significant challenges including
strict training data requirements, weak prompt following, and unnatural
transitions. To address these problems, we propose DiTCtrl, a training-free
multi-prompt video generation method under MM-DiT architectures for the first
time. Our key idea is to take the multi-prompt video generation task as
temporal video editing with smooth transitions. To achieve this goal, we first
analyze MM-DiT's attention mechanism, finding that the 3D full attention
behaves similarly to that of the cross/self-attention blocks in the UNet-like
diffusion models, enabling mask-guided precise semantic control across
different prompts with attention sharing for multi-prompt video generation.
Based on our careful design, the video generated by DiTCtrl achieves smooth
transitions and consistent object motion given multiple sequential prompts
without additional training. Besides, we also present MPVBench, a new benchmark
specially designed for multi-prompt video generation to evaluate the
performance of multi-prompt generation. Extensive experiments demonstrate that
our method achieves state-of-the-art performance without additional training.
676b9b876fb4876383b8591b
Minghong Cai
claimed_verified
2024-12-30T19:32:49.066Z
64f94370c3c12b377cc51086
https://cdn-avatars.huggingface.co/v1/production/uploads/64f94370c3c12b377cc51086/6CXcHhqAoykqXcShqM8Rd.jpeg
Minghong Cai
user
onevfall
676b9b876fb4876383b8591c
Xiaodong Cun
claimed_verified
2025-06-30T06:23:49.880Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
676b9b876fb4876383b8591d
Xiaoyu Li
676b9b876fb4876383b8591e
Wenze Liu
676b9b876fb4876383b8591f
Zhaoyang Zhang
claimed_verified
2025-03-18T08:12:28.534Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
676b9b876fb4876383b85920
Yong Zhang
676b9b876fb4876383b85921
Ying Shan
676b9b876fb4876383b85922
Xiangyu Yue
DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion
  Transformer for Tuning-Free Multi-Prompt Longer Video Generation
Sora-like video generation models have achieved remarkable progress with a
Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current
video generation models predominantly focus on single-prompt, struggling to
generate coherent scenes with multiple sequential prompts that better reflect
real-world dynamic scenarios. While some pioneering works have explored
multi-prompt video generation, they face significant challenges including
strict training data requirements, weak prompt following, and unnatural
transitions. To address these problems, we propose DiTCtrl, a training-free
multi-prompt video generation method under MM-DiT architectures for the first
time. Our key idea is to take the multi-prompt video generation task as
temporal video editing with smooth transitions. To achieve this goal, we first
analyze MM-DiT's attention mechanism, finding that the 3D full attention
behaves similarly to that of the cross/self-attention blocks in the UNet-like
diffusion models, enabling mask-guided precise semantic control across
different prompts with attention sharing for multi-prompt video generation.
Based on our careful design, the video generated by DiTCtrl achieves smooth
transitions and consistent object motion given multiple sequential prompts
without additional training. Besides, we also present MPVBench, a new benchmark
specially designed for multi-prompt video generation to evaluate the
performance of multi-prompt generation. Extensive experiments demonstrate that
our method achieves state-of-the-art performance without additional training.
DiTCtrl, a training-free method for multi-prompt video generation, enhances MM-DiT architectures by enabling smooth transitions and consistent motion using an analyzed attention mechanism.
Multi-Modal Diffusion Transformer MM-DiT DiTCtrl attention mechanism 3D full attention UNet-like diffusion models mask-guided semantic control multi-prompt video generation MPVBench state-of-the-art performance