SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video
  Generation
Human beings are endowed with a complementary learning system, which bridges
the slow learning of general world dynamics with fast storage of episodic
memory from a new experience. Previous video generation models, however,
primarily focus on slow learning by pre-training on vast amounts of data,
overlooking the fast learning phase crucial for episodic memory storage. This
oversight leads to inconsistencies across temporally distant frames when
generating longer videos, as these frames fall beyond the model's context
window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning
system for action-driven long video generation. Our approach incorporates a
masked conditional video diffusion model for the slow learning of world
dynamics, alongside an inference-time fast learning strategy based on a
temporal LoRA module. Specifically, the fast learning process updates its
temporal LoRA parameters based on local inputs and outputs, thereby efficiently
storing episodic memory in its parameters. We further propose a slow-fast
learning loop algorithm that seamlessly integrates the inner fast learning loop
into the outer slow learning loop, enabling the recall of prior multi-episode
experiences for context-aware skill learning. To facilitate the slow learning
of an approximate world model, we collect a large-scale dataset of 200k videos
with language action annotations, covering a wide range of scenarios. Extensive
experiments show that SlowFast-VGen outperforms baselines across various
metrics for action-driven video generation, achieving an FVD score of 514
compared to 782, and maintaining consistency in longer videos, with an average
of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm
significantly enhances performances on long-horizon planning tasks as well.
Project Website: https://slowfast-vgen.github.io
6723ce514a49116d8ec02f72
Yining Hong
6723ce514a49116d8ec02f73
Beide Liu
6723ce514a49116d8ec02f74
Maxine Wu
6723ce514a49116d8ec02f75
Yuanhao Zhai
6723ce514a49116d8ec02f76
Kai-Wei Chang
6723ce514a49116d8ec02f77
Lingjie Li
6723ce514a49116d8ec02f78
Kevin Lin
6723ce514a49116d8ec02f79
Chung-Ching Lin
claimed_verified
2025-07-23T08:38:48.611Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
6723ce514a49116d8ec02f7a
Jianfeng Wang
6723ce514a49116d8ec02f7b
Zhengyuan Yang
claimed_verified
2024-11-05T07:58:46.978Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
6723ce514a49116d8ec02f7c
Yingnian Wu
6723ce514a49116d8ec02f7d
Lijuan Wang
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video
  Generation
Human beings are endowed with a complementary learning system, which bridges
the slow learning of general world dynamics with fast storage of episodic
memory from a new experience. Previous video generation models, however,
primarily focus on slow learning by pre-training on vast amounts of data,
overlooking the fast learning phase crucial for episodic memory storage. This
oversight leads to inconsistencies across temporally distant frames when
generating longer videos, as these frames fall beyond the model's context
window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning
system for action-driven long video generation. Our approach incorporates a
masked conditional video diffusion model for the slow learning of world
dynamics, alongside an inference-time fast learning strategy based on a
temporal LoRA module. Specifically, the fast learning process updates its
temporal LoRA parameters based on local inputs and outputs, thereby efficiently
storing episodic memory in its parameters. We further propose a slow-fast
learning loop algorithm that seamlessly integrates the inner fast learning loop
into the outer slow learning loop, enabling the recall of prior multi-episode
experiences for context-aware skill learning. To facilitate the slow learning
of an approximate world model, we collect a large-scale dataset of 200k videos
with language action annotations, covering a wide range of scenarios. Extensive
experiments show that SlowFast-VGen outperforms baselines across various
metrics for action-driven video generation, achieving an FVD score of 514
compared to 782, and maintaining consistency in longer videos, with an average
of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm
significantly enhances performances on long-horizon planning tasks as well.
Project Website: https://slowfast-vgen.github.io
SlowFast-VGen integrates a dual-speed learning system with a masked conditional video diffusion model for slow learning and a temporal LoRA module for fast learning, improving long video generation and consistency.
masked conditional video diffusion model temporal LoRA module slow-fast learning loop FVD score long-horizon planning tasks