and hz = hx
hp

, wz = wx
wp

3

Published as a conference paper at ICLR 2023

dimensions to have a tensor of shape (tz+1)⇥wz⇤hz⇥dz where the spatial and temporal dimensions
are separated. Then multiple transformer layers are applied along the spatial dimensions with all-
to-all attention. This is followed by multiple transformer layers over the temporal dimension with
causal attention such that each spatial token only observes spatial tokens from previous frames in
an auto-regressive manner. The effect of this is that the ﬁrst frame can be completely independently
encoded. This opens up the possibility of text to image training to be embedded naturally into
our video model. The second advantage is that we can condition the video generation process on
a number of starting frames. The resulting patch embeddings z of shape tz ⇥ wz ⇥ hz ⇥ dz are
then tokenized into learned codewords cz by vector quantization. The codebook learning will be
discussed later together with the losses.
PHENAKI- VARIABLE LENGTH VIDEO GENERATION FROM OPEN DOMAIN TEXTUAL DESCRIPTIONS