title: VideoTetris: Towards Compositional Text-to-Video Generation
summary: Diffusion models have demonstrated great success in text-to-video (T2V)
  generation. However, existing methods may face challenges when handling complex
  (long) video generation scenarios that involve multiple objects or dynamic
  changes in object numbers. To address these limitations, we propose
  VideoTetris, a novel framework that enables compositional T2V generation.
  Specifically, we propose spatio-temporal compositional diffusion to precisely
  follow complex textual semantics by manipulating and composing the attention
  maps of denoising networks spatially and temporally. Moreover, we propose an
  enhanced video data preprocessing to enhance the training data regarding motion
  dynamics and prompt understanding, equipped with a new reference frame
  attention mechanism to improve the consistency of auto-regressive video
  generation. Extensive experiments demonstrate that our VideoTetris achieves
  impressive qualitative and quantitative results in compositional T2V
  generation. Code is available at: https://github.com/YangLing0818/VideoTetris
paper_authors._id: 6662687368009120aff96b94
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:23:30.477Z
paper_authors._id: 64e357dd825f4133e7427bf8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e357dd825f4133e7427bf8/HwaWhINrzkbXG6SHG2oyf.jpeg
paper_authors.fullname: tyfeld
paper_authors.type: user
paper_authors.user: tyfeld
paper_authors._id: 6662687368009120aff96b95
paper_authors.name: Ling Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:56.636Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 6662687368009120aff96b96
paper_authors.name: Haotian Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T10:17:30.822Z
paper_authors._id: 64ba53dcbc787364968a7649
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KgbDlawW8n2s_V3kzhJLW.png
paper_authors.fullname: Haotian Yang
paper_authors.type: user
paper_authors.user: haotianxoxo
paper_authors._id: 6662687368009120aff96b97
paper_authors.name: Yuan Gao
paper_authors._id: 6662687368009120aff96b98
paper_authors.name: Yufan Deng
paper_authors._id: 6662687368009120aff96b99
paper_authors.name: Jingmin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T10:03:28.754Z
paper_authors._id: 64218b51e61513ec1c63d6d4
paper_authors.avatarUrl: /avatars/2a14f872a391737b3d862788ab9f99df.svg
paper_authors.fullname: Jingmin Chen 
paper_authors.type: user
paper_authors.user: Jingmin
paper_authors._id: 6662687368009120aff96b9a
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T10:03:20.223Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 6662687368009120aff96b9b
paper_authors.name: Zhaochen Yu
paper_authors._id: 6662687368009120aff96b9c
paper_authors.name: Xin Tao
paper_authors._id: 6662687368009120aff96b9d
paper_authors.name: Pengfei Wan
paper_authors._id: 6662687368009120aff96b9e
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T09:46:13.604Z
paper_authors._id: 6306ec4f1e731505a77646f0
paper_authors.avatarUrl: /avatars/5d065dd89324072176cc29498cd98b03.svg
paper_authors.fullname: ZhangDi
paper_authors.type: user
paper_authors.user: ZhangDi
paper_authors._id: 6662687368009120aff96b9f
paper_authors.name: Bin Cui
paper_title: VideoTetris: Towards Compositional Text-to-Video Generation
paper_summary: Diffusion models have demonstrated great success in text-to-video (T2V)
  generation. However, existing methods may face challenges when handling complex
  (long) video generation scenarios that involve multiple objects or dynamic
  changes in object numbers. To address these limitations, we propose
  VideoTetris, a novel framework that enables compositional T2V generation.
  Specifically, we propose spatio-temporal compositional diffusion to precisely
  follow complex textual semantics by manipulating and composing the attention
  maps of denoising networks spatially and temporally. Moreover, we propose an
  enhanced video data preprocessing to enhance the training data regarding motion
  dynamics and prompt understanding, equipped with a new reference frame
  attention mechanism to improve the consistency of auto-regressive video
  generation. Extensive experiments demonstrate that our VideoTetris achieves
  impressive qualitative and quantitative results in compositional T2V
  generation. Code is available at: https://github.com/YangLing0818/VideoTetris
paper_ai_summary: VideoTetris framework uses compositional diffusion and enhanced preprocessing to generate complex text-to-video content with improved consistency and dynamic handling.
paper_ai_keywords: diffusion models text-to-video spatio-temporal compositional diffusion denoising networks attention maps video data preprocessing reference frame attention mechanism auto-regressive video generation