title: Video-Infinity: Distributed Long Video Generation
summary: Diffusion models have recently achieved remarkable results for video
  generation. Despite the encouraging performances, the generated videos are
  typically constrained to a small number of frames, resulting in clips lasting
  merely a few seconds. The primary challenges in producing longer videos include
  the substantial memory requirements and the extended processing time required
  on a single GPU. A straightforward solution would be to split the workload
  across multiple GPUs, which, however, leads to two issues: (1) ensuring all
  GPUs communicate effectively to share timing and context information, and (2)
  modifying existing video diffusion models, which are usually trained on short
  sequences, to create longer videos without additional training. To tackle
  these, in this paper we introduce Video-Infinity, a distributed inference
  pipeline that enables parallel processing across multiple GPUs for long-form
  video generation. Specifically, we propose two coherent mechanisms: Clip
  parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
  and sharing of context information across GPUs which minimizes communication
  overhead, while Dual-scope attention modulates the temporal self-attention to
  balance local and global contexts efficiently across the devices. Together, the
  two mechanisms join forces to distribute the workload and enable the fast
  generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
  method generates videos up to 2,300 frames in approximately 5 minutes, enabling
  long video generation at a speed 100 times faster than the prior methods.
paper_authors._id: 667a29ea42ef3dbfae30e96b
paper_authors.name: Zhenxiong Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:05:14.275Z
paper_authors._id: 664ef197d3f8759dddfbc765
paper_authors.avatarUrl: /avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
paper_authors.fullname: Tan Zhen Xiong
paper_authors.type: user
paper_authors.user: ZhenXiong
paper_authors._id: 667a29ea42ef3dbfae30e96c
paper_authors.name: Xingyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:32.043Z
paper_authors._id: 634cfebc350bcee9bed20a4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
paper_authors.fullname: Xingyi Yang
paper_authors.type: user
paper_authors.user: adamdad
paper_authors._id: 667a29ea42ef3dbfae30e96d
paper_authors.name: Songhua Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:05:22.109Z
paper_authors._id: 642ad435a096201096ecc580
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642ad435a096201096ecc580/uRcnkMrXbX5aTNiGcpC9d.png
paper_authors.fullname: Songhua Liu
paper_authors.type: user
paper_authors.user: flyingman
paper_authors._id: 667a29ea42ef3dbfae30e96e
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:05:44.065Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: Video-Infinity: Distributed Long Video Generation
paper_summary: Diffusion models have recently achieved remarkable results for video
  generation. Despite the encouraging performances, the generated videos are
  typically constrained to a small number of frames, resulting in clips lasting
  merely a few seconds. The primary challenges in producing longer videos include
  the substantial memory requirements and the extended processing time required
  on a single GPU. A straightforward solution would be to split the workload
  across multiple GPUs, which, however, leads to two issues: (1) ensuring all
  GPUs communicate effectively to share timing and context information, and (2)
  modifying existing video diffusion models, which are usually trained on short
  sequences, to create longer videos without additional training. To tackle
  these, in this paper we introduce Video-Infinity, a distributed inference
  pipeline that enables parallel processing across multiple GPUs for long-form
  video generation. Specifically, we propose two coherent mechanisms: Clip
  parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
  and sharing of context information across GPUs which minimizes communication
  overhead, while Dual-scope attention modulates the temporal self-attention to
  balance local and global contexts efficiently across the devices. Together, the
  two mechanisms join forces to distribute the workload and enable the fast
  generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
  method generates videos up to 2,300 frames in approximately 5 minutes, enabling
  long video generation at a speed 100 times faster than the prior methods.
paper_ai_keywords: diffusion models video generation frame constraints memory requirements processing time GPU communication modular GPUs Video-Infinity Clip parallelism Dual-scope attention self-attention long-form video generation communication overhead local contexts temporal contexts device optimization