FIFO-Diffusion: Generating Infinite Videos from Text without Training
We propose a novel inference technique based on a pretrained diffusion model
for text-conditional video generation. Our approach, called FIFO-Diffusion, is
conceptually capable of generating infinitely long videos without training.
This is achieved by iteratively performing diagonal denoising, which
concurrently processes a series of consecutive frames with increasing noise
levels in a queue; our method dequeues a fully denoised frame at the head while
enqueuing a new random noise frame at the tail. However, diagonal denoising is
a double-edged sword as the frames near the tail can take advantage of cleaner
ones by forward reference but such a strategy induces the discrepancy between
training and inference. Hence, we introduce latent partitioning to reduce the
training-inference gap and lookahead denoising to leverage the benefit of
forward referencing. We have demonstrated the promising results and
effectiveness of the proposed methods on existing text-to-video generation
baselines.
664c08072f72dbb18fbcd9c9
Jihwan Kim
claimed_verified
2024-05-21T07:23:41.782Z
651f62b9b54adab2d4454490
/avatars/ecfaf6283c4961278cfee5e9df18fccf.svg
Jihwan Kim
user
jjihwannn
664c08072f72dbb18fbcd9ca
Junoh Kang
admin_assigned
2024-05-21T08:10:32.535Z
664c41d2533c5d0c9c826734
/avatars/3c1c89790cc61ca0ef29b7c8e24867d5.svg
Junoh Kang
user
Junoh
664c08072f72dbb18fbcd9cb
Jinyoung Choi
664c08072f72dbb18fbcd9cc
Bohyung Han
FIFO-Diffusion: Generating Infinite Videos from Text without Training
We propose a novel inference technique based on a pretrained diffusion model
for text-conditional video generation. Our approach, called FIFO-Diffusion, is
conceptually capable of generating infinitely long videos without training.
This is achieved by iteratively performing diagonal denoising, which
concurrently processes a series of consecutive frames with increasing noise
levels in a queue; our method dequeues a fully denoised frame at the head while
enqueuing a new random noise frame at the tail. However, diagonal denoising is
a double-edged sword as the frames near the tail can take advantage of cleaner
ones by forward reference but such a strategy induces the discrepancy between
training and inference. Hence, we introduce latent partitioning to reduce the
training-inference gap and lookahead denoising to leverage the benefit of
forward referencing. We have demonstrated the promising results and
effectiveness of the proposed methods on existing text-to-video generation
baselines.
FIFO-Diffusion generates long text-conditional videos through iterative denoising with latent partitioning and lookahead strategies to minimize the training-inference gap.
diffusion model FIFO-Diffusion diagonal denoising latent partitioning lookahead denoising text-to-video generation