title: One-Minute Video Generation with Test-Time Training
summary: Transformers today still struggle to generate one-minute videos because
  self-attention layers are inefficient for long context. Alternatives such as
  Mamba layers struggle with complex multi-scene stories because their hidden
  states are less expressive. We experiment with Test-Time Training (TTT) layers,
  whose hidden states themselves can be neural networks, therefore more
  expressive. Adding TTT layers into a pre-trained Transformer enables it to
  generate one-minute videos from text storyboards. For proof of concept, we
  curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
  Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
  generate much more coherent videos that tell complex stories, leading by 34 Elo
  points in a human evaluation of 100 videos per method. Although promising,
  results still contain artifacts, likely due to the limited capability of the
  pre-trained 5B model. The efficiency of our implementation can also be
  improved. We have only experimented with one-minute videos due to resource
  constraints, but the approach can be extended to longer videos and more complex
  stories. Sample videos, code and annotations are available at:
  https://test-time-training.github.io/video-dit
paper_authors._id: 67f4a0ccfefcc542f83f84e7
paper_authors.name: Karan Dalal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:09:24.283Z
paper_authors._id: 646ebf2298e8f749fc60ce38
paper_authors.avatarUrl: /avatars/5a76c9343451c24e9697d3165cdc0af6.svg
paper_authors.fullname: Karan Dalal
paper_authors.type: user
paper_authors.user: karansdalal
paper_authors._id: 67f4a0ccfefcc542f83f84e8
paper_authors.name: Daniel Koceja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:09:31.602Z
paper_authors._id: 66b01b126038fe024ae979d3
paper_authors.avatarUrl: /avatars/75b561be803fa28cb78c6cff9eb5ac54.svg
paper_authors.fullname: Daniel Koceja
paper_authors.type: user
paper_authors.user: koceja
paper_authors._id: 67f4a0ccfefcc542f83f84e9
paper_authors.name: Gashon Hussein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:09:39.483Z
paper_authors._id: 638835fede1fa6adc5485a05
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669871090087-noauth.jpeg
paper_authors.fullname: Gashon Hussein
paper_authors.type: user
paper_authors.user: GashonHussein
paper_authors._id: 67f4a0ccfefcc542f83f84ea
paper_authors.name: Jiarui Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T20:06:33.532Z
paper_authors._id: 61f7671a481b345bd1059092
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643603726045-noauth.jpeg
paper_authors.fullname: Jiarui Xu
paper_authors.type: user
paper_authors.user: xvjiarui
paper_authors._id: 67f4a0ccfefcc542f83f84eb
paper_authors.name: Yue Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:51:29.995Z
paper_authors._id: 638fe91639f7e2a7f9d2a8c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
paper_authors.fullname: Yue Zhao
paper_authors.type: user
paper_authors.user: zhaoyue-zephyrus
paper_authors._id: 67f4a0ccfefcc542f83f84ec
paper_authors.name: Youjin Song
paper_authors._id: 67f4a0ccfefcc542f83f84ed
paper_authors.name: Shihao Han
paper_authors._id: 67f4a0ccfefcc542f83f84ee
paper_authors.name: Ka Chun Cheung
paper_authors._id: 67f4a0ccfefcc542f83f84ef
paper_authors.name: Jan Kautz
paper_authors._id: 67f4a0ccfefcc542f83f84f0
paper_authors.name: Carlos Guestrin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:10:26.609Z
paper_authors._id: 677c45511fbb93b90f1c6f3d
paper_authors.avatarUrl: /avatars/3a78fdd8d1debc8d267e80e4b7a6bf77.svg
paper_authors.fullname: Carlos Guestrin
paper_authors.type: user
paper_authors.user: guestrin
paper_authors._id: 67f4a0ccfefcc542f83f84f1
paper_authors.name: Tatsunori Hashimoto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:10:20.291Z
paper_authors._id: 67ecbb7eb57a7b083182ea3f
paper_authors.avatarUrl: /avatars/8fb800e0729771e61ff0d2f9a05eb5b9.svg
paper_authors.fullname: Tatsunori Hashimoto
paper_authors.type: user
paper_authors.user: hashtag56
paper_authors._id: 67f4a0ccfefcc542f83f84f2
paper_authors.name: Sanmi Koyejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:10:14.122Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_authors._id: 67f4a0ccfefcc542f83f84f3
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T09:10:07.480Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 67f4a0ccfefcc542f83f84f4
paper_authors.name: Yu Sun
paper_authors._id: 67f4a0ccfefcc542f83f84f5
paper_authors.name: Xiaolong Wang
paper_title: One-Minute Video Generation with Test-Time Training
paper_summary: Transformers today still struggle to generate one-minute videos because
  self-attention layers are inefficient for long context. Alternatives such as
  Mamba layers struggle with complex multi-scene stories because their hidden
  states are less expressive. We experiment with Test-Time Training (TTT) layers,
  whose hidden states themselves can be neural networks, therefore more
  expressive. Adding TTT layers into a pre-trained Transformer enables it to
  generate one-minute videos from text storyboards. For proof of concept, we
  curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
  Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
  generate much more coherent videos that tell complex stories, leading by 34 Elo
  points in a human evaluation of 100 videos per method. Although promising,
  results still contain artifacts, likely due to the limited capability of the
  pre-trained 5B model. The efficiency of our implementation can also be
  improved. We have only experimented with one-minute videos due to resource
  constraints, but the approach can be extended to longer videos and more complex
  stories. Sample videos, code and annotations are available at:
  https://test-time-training.github.io/video-dit
paper_ai_summary: Test-Time Training (TTT) layers enable pre-trained Transformers to generate coherent one-minute videos from text storyboards, outperforming alternatives like Mamba~2 and Gated DeltaNet.
paper_ai_keywords: self-attention layers Mamba layers Test-Time Training (TTT) layers pre-trained Transformer one-minute videos text storyboards Tom and Jerry cartoons Elo points sliding-window attention layers