title: SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video
    Generation
summary: Human beings are endowed with a complementary learning system, which bridges
  the slow learning of general world dynamics with fast storage of episodic
  memory from a new experience. Previous video generation models, however,
  primarily focus on slow learning by pre-training on vast amounts of data,
  overlooking the fast learning phase crucial for episodic memory storage. This
  oversight leads to inconsistencies across temporally distant frames when
  generating longer videos, as these frames fall beyond the model's context
  window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning
  system for action-driven long video generation. Our approach incorporates a
  masked conditional video diffusion model for the slow learning of world
  dynamics, alongside an inference-time fast learning strategy based on a
  temporal LoRA module. Specifically, the fast learning process updates its
  temporal LoRA parameters based on local inputs and outputs, thereby efficiently
  storing episodic memory in its parameters. We further propose a slow-fast
  learning loop algorithm that seamlessly integrates the inner fast learning loop
  into the outer slow learning loop, enabling the recall of prior multi-episode
  experiences for context-aware skill learning. To facilitate the slow learning
  of an approximate world model, we collect a large-scale dataset of 200k videos
  with language action annotations, covering a wide range of scenarios. Extensive
  experiments show that SlowFast-VGen outperforms baselines across various
  metrics for action-driven video generation, achieving an FVD score of 514
  compared to 782, and maintaining consistency in longer videos, with an average
  of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm
  significantly enhances performances on long-horizon planning tasks as well.
  Project Website: https://slowfast-vgen.github.io
paper_authors._id: 6723ce514a49116d8ec02f72
paper_authors.name: Yining Hong
paper_authors._id: 6723ce514a49116d8ec02f73
paper_authors.name: Beide Liu
paper_authors._id: 6723ce514a49116d8ec02f74
paper_authors.name: Maxine Wu
paper_authors._id: 6723ce514a49116d8ec02f75
paper_authors.name: Yuanhao Zhai
paper_authors._id: 6723ce514a49116d8ec02f76
paper_authors.name: Kai-Wei Chang
paper_authors._id: 6723ce514a49116d8ec02f77
paper_authors.name: Lingjie Li
paper_authors._id: 6723ce514a49116d8ec02f78
paper_authors.name: Kevin Lin
paper_authors._id: 6723ce514a49116d8ec02f79
paper_authors.name: Chung-Ching Lin
paper_authors._id: 6723ce514a49116d8ec02f7a
paper_authors.name: Jianfeng Wang
paper_authors._id: 6723ce514a49116d8ec02f7b
paper_authors.name: Zhengyuan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:46.978Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 6723ce514a49116d8ec02f7c
paper_authors.name: Yingnian Wu
paper_authors._id: 6723ce514a49116d8ec02f7d
paper_authors.name: Lijuan Wang
paper_title: SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video
    Generation
paper_summary: Human beings are endowed with a complementary learning system, which bridges
  the slow learning of general world dynamics with fast storage of episodic
  memory from a new experience. Previous video generation models, however,
  primarily focus on slow learning by pre-training on vast amounts of data,
  overlooking the fast learning phase crucial for episodic memory storage. This
  oversight leads to inconsistencies across temporally distant frames when
  generating longer videos, as these frames fall beyond the model's context
  window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning
  system for action-driven long video generation. Our approach incorporates a
  masked conditional video diffusion model for the slow learning of world
  dynamics, alongside an inference-time fast learning strategy based on a
  temporal LoRA module. Specifically, the fast learning process updates its
  temporal LoRA parameters based on local inputs and outputs, thereby efficiently
  storing episodic memory in its parameters. We further propose a slow-fast
  learning loop algorithm that seamlessly integrates the inner fast learning loop
  into the outer slow learning loop, enabling the recall of prior multi-episode
  experiences for context-aware skill learning. To facilitate the slow learning
  of an approximate world model, we collect a large-scale dataset of 200k videos
  with language action annotations, covering a wide range of scenarios. Extensive
  experiments show that SlowFast-VGen outperforms baselines across various
  metrics for action-driven video generation, achieving an FVD score of 514
  compared to 782, and maintaining consistency in longer videos, with an average
  of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm
  significantly enhances performances on long-horizon planning tasks as well.
  Project Website: https://slowfast-vgen.github.io
paper_ai_summary: SlowFast-VGen integrates a dual-speed learning system with a masked conditional video diffusion model for slow learning and a temporal LoRA module for fast learning, improving long video generation and consistency.
paper_ai_keywords: masked conditional video diffusion model temporal LoRA module slow-fast learning loop FVD score long-horizon planning tasks