T5X

Token

Masked/Empty 

Token

Transformer

Linear

Embedding

Frozen Model

Operation

Figure 2. The architecture of Phenaki. Left: C-ViViT encoder architecture. The embeddings of
images and video patches from raw frames x are processed by a spatial and then a causal transformer
(auto-regressive in time) to generate video tokens z. Center: MaskGiT is trained to reconstruct
masked tokens z predicted by a frozen C-ViViT encoder and conditioned on T5X tokens of a given
prompt p0. Right: How Phenaki can generate arbitrary long videos by freezing the past token and
generating the future tokens. The prompt can change over time to enable time-variable prompt (i.e.
story) conditional generation. The subscripts represent time (i.e. frame number).

2 THE PHENAKI MODEL
PHENAKI- VARIABLE LENGTH VIDEO GENERATION FROM OPEN DOMAIN TEXTUAL DESCRIPTIONS