Video-Infinity: Distributed Long Video Generation
Diffusion models have recently achieved remarkable results for video
generation. Despite the encouraging performances, the generated videos are
typically constrained to a small number of frames, resulting in clips lasting
merely a few seconds. The primary challenges in producing longer videos include
the substantial memory requirements and the extended processing time required
on a single GPU. A straightforward solution would be to split the workload
across multiple GPUs, which, however, leads to two issues: (1) ensuring all
GPUs communicate effectively to share timing and context information, and (2)
modifying existing video diffusion models, which are usually trained on short
sequences, to create longer videos without additional training. To tackle
these, in this paper we introduce Video-Infinity, a distributed inference
pipeline that enables parallel processing across multiple GPUs for long-form
video generation. Specifically, we propose two coherent mechanisms: Clip
parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
and sharing of context information across GPUs which minimizes communication
overhead, while Dual-scope attention modulates the temporal self-attention to
balance local and global contexts efficiently across the devices. Together, the
two mechanisms join forces to distribute the workload and enable the fast
generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
method generates videos up to 2,300 frames in approximately 5 minutes, enabling
long video generation at a speed 100 times faster than the prior methods.
667a29ea42ef3dbfae30e96b
Zhenxiong Tan
admin_assigned
2024-06-25T08:05:14.275Z
664ef197d3f8759dddfbc765
/avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
Tan Zhen Xiong
user
ZhenXiong
667a29ea42ef3dbfae30e96c
Xingyi Yang
claimed_verified
2024-06-25T07:33:32.043Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
667a29ea42ef3dbfae30e96d
Songhua Liu
admin_assigned
2024-06-25T08:05:22.109Z
642ad435a096201096ecc580
https://cdn-avatars.huggingface.co/v1/production/uploads/642ad435a096201096ecc580/uRcnkMrXbX5aTNiGcpC9d.png
Songhua Liu
user
flyingman
667a29ea42ef3dbfae30e96e
Xinchao Wang
admin_assigned
2024-06-25T08:05:44.065Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
Video-Infinity: Distributed Long Video Generation
Diffusion models have recently achieved remarkable results for video
generation. Despite the encouraging performances, the generated videos are
typically constrained to a small number of frames, resulting in clips lasting
merely a few seconds. The primary challenges in producing longer videos include
the substantial memory requirements and the extended processing time required
on a single GPU. A straightforward solution would be to split the workload
across multiple GPUs, which, however, leads to two issues: (1) ensuring all
GPUs communicate effectively to share timing and context information, and (2)
modifying existing video diffusion models, which are usually trained on short
sequences, to create longer videos without additional training. To tackle
these, in this paper we introduce Video-Infinity, a distributed inference
pipeline that enables parallel processing across multiple GPUs for long-form
video generation. Specifically, we propose two coherent mechanisms: Clip
parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
and sharing of context information across GPUs which minimizes communication
overhead, while Dual-scope attention modulates the temporal self-attention to
balance local and global contexts efficiently across the devices. Together, the
two mechanisms join forces to distribute the workload and enable the fast
generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
method generates videos up to 2,300 frames in approximately 5 minutes, enabling
long video generation at a speed 100 times faster than the prior methods.
Video-Infinity uses distributed inference with Clip parallelism and Dual-scope attention to significantly accelerate long-form video generation across multiple GPUs.
diffusion models video generation GPU Clip parallelism Dual-scope attention temporal self-attention