a narration of the entire video where each prompt corresponds to a scene from the video. This allows
for creating dynamically changing scenes. To the best our knowledge, this paper is the ﬁrst work to
generate such videos. An example of this can be seen in Fig. 1 and on phenaki.github.io. The way it
works is that we generate a video with the ﬁrst prompt and then extend it in time by conditioning a
possibly new prompt and on the last N, typically 5, previously generated frames.

7

Published as a conference paper at ICLR 2023

Table 3. Video reconstruction results on Moments-in-Time. The number of tokens is computed for 10 frames
with the exception of C-ViViT which is for 11, due to the isolated initial frame.

Method
Conv VQ-GAN [14]
Conv VQ-GAN + Video loss
ViT VQ-GAN [64]
ViT VQ-GAN + Video loss
C-ViViT VQ-GAN (Ours)

FID #
7.5
13.7
3.4
3.8
4.5

FVD # Number of Tokens #
306.1
346.5
166.6
173.1
65.78

2560
2560
2560
2560
1536
PHENAKI- VARIABLE LENGTH VIDEO GENERATION FROM OPEN DOMAIN TEXTUAL DESCRIPTIONS