  We propose Latent-Shift -- an efficient text-to-video generation method based
on a pretrained text-to-image generation model that consists of an autoencoder
and a U-Net diffusion model. Learning a video diffusion model in the latent
space is much more efficient than in the pixel space. The latter is often
limited to first generating a low-resolution video followed by a sequence of
frame interpolation and super-resolution models, which makes the entire
pipeline very complex and computationally expensive. To extend a U-Net from
image generation to video generation, prior work proposes to add additional
modules like 1D temporal convolution and/or temporal attention layers. In
contrast, we propose a parameter-free temporal shift module that can leverage
the spatial U-Net as is for video generation. We achieve this by shifting two
portions of the feature map channels forward and backward along the temporal
dimension. The shifted features of the current frame thus receive the features
from the previous and the subsequent frames, enabling motion learning without
additional parameters. We show that Latent-Shift achieves comparable or better
results while being significantly more efficient. Moreover, Latent-Shift can
generate images despite being finetuned for T2V generation.

Computer Vision and Pattern Recognition