title: Pyramidal Flow Matching for Efficient Video Generative Modeling
summary: Video generation requires modeling a vast spatiotemporal space, which demands
  significant computational resources and data usage. To reduce the complexity,
  the prevailing approaches employ a cascaded architecture to avoid direct
  training with full resolution. Despite reducing computational demands, the
  separate optimization of each sub-stage hinders knowledge sharing and
  sacrifices flexibility. This work introduces a unified pyramidal flow matching
  algorithm. It reinterprets the original denoising trajectory as a series of
  pyramid stages, where only the final stage operates at the full resolution,
  thereby enabling more efficient video generative modeling. Through our
  sophisticated design, the flows of different pyramid stages can be interlinked
  to maintain continuity. Moreover, we craft autoregressive video generation with
  a temporal pyramid to compress the full-resolution history. The entire
  framework can be optimized in an end-to-end manner and with a single unified
  Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
  supports generating high-quality 5-second (up to 10-second) videos at 768p
  resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
  will be open-sourced at https://pyramid-flow.github.io.
paper_authors._id: 67074f465ea108eecd170d48
paper_authors.name: Yang Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T10:59:07.115Z
paper_authors._id: 64b79ec0a8c39dc078897430
paper_authors.avatarUrl: /avatars/8339f3ab9bb7baaf69bf174eafb7282c.svg
paper_authors.fullname: Yang Jin
paper_authors.type: user
paper_authors.user: rain1011
paper_authors._id: 67074f465ea108eecd170d49
paper_authors.name: Zhicheng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:39.179Z
paper_authors._id: 62fc758172a7ab50b4b89c8c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677251053045-62fc758172a7ab50b4b89c8c.jpeg
paper_authors.fullname: Zhicheng Sun
paper_authors.type: user
paper_authors.user: feifeiobama
paper_authors._id: 67074f465ea108eecd170d4a
paper_authors.name: Ningyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:55:43.710Z
paper_authors._id: 66af935fa20def3de33f54ed
paper_authors.avatarUrl: /avatars/2666e97165377d979c530d96a1089d86.svg
paper_authors.fullname: Li Ningyuan
paper_authors.type: user
paper_authors.user: Ninggggy
paper_authors._id: 67074f465ea108eecd170d4b
paper_authors.name: Kun Xu
paper_authors._id: 67074f465ea108eecd170d4c
paper_authors.name: Kun Xu
paper_authors._id: 67074f465ea108eecd170d4d
paper_authors.name: Hao Jiang
paper_authors._id: 67074f465ea108eecd170d4e
paper_authors.name: Nan Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:55:08.855Z
paper_authors._id: 6461e1ab5e410d3ff8ec84e4
paper_authors.avatarUrl: /avatars/16d5fc2994993ab65e8e43798cd864c4.svg
paper_authors.fullname: Nan Zhuang
paper_authors.type: user
paper_authors.user: Payne53
paper_authors._id: 67074f465ea108eecd170d4f
paper_authors.name: Quzhe Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:55:02.968Z
paper_authors._id: 642fd69e8169f1dce23c0e64
paper_authors.avatarUrl: /avatars/8dc14c7850b6274ab24906ea5b2e5e8a.svg
paper_authors.fullname: Quzhe Huang
paper_authors.type: user
paper_authors.user: quzhe
paper_authors._id: 67074f465ea108eecd170d50
paper_authors.name: Yang Song
paper_authors._id: 67074f465ea108eecd170d51
paper_authors.name: Yadong Mu
paper_authors._id: 67074f465ea108eecd170d52
paper_authors.name: Zhouchen Lin
paper_title: Pyramidal Flow Matching for Efficient Video Generative Modeling
paper_summary: Video generation requires modeling a vast spatiotemporal space, which demands
  significant computational resources and data usage. To reduce the complexity,
  the prevailing approaches employ a cascaded architecture to avoid direct
  training with full resolution. Despite reducing computational demands, the
  separate optimization of each sub-stage hinders knowledge sharing and
  sacrifices flexibility. This work introduces a unified pyramidal flow matching
  algorithm. It reinterprets the original denoising trajectory as a series of
  pyramid stages, where only the final stage operates at the full resolution,
  thereby enabling more efficient video generative modeling. Through our
  sophisticated design, the flows of different pyramid stages can be interlinked
  to maintain continuity. Moreover, we craft autoregressive video generation with
  a temporal pyramid to compress the full-resolution history. The entire
  framework can be optimized in an end-to-end manner and with a single unified
  Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
  supports generating high-quality 5-second (up to 10-second) videos at 768p
  resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
  will be open-sourced at https://pyramid-flow.github.io.
paper_ai_keywords: pyramidal flow matching algorithm denoising trajectory pyramid stages autoregressive video generation temporal pyramid end-to-end manner unified Diffusion Transformer (DiT)