Language Models for Video and Image Generation In
contrast, video language models are derived from the gen-
eral family of transformer-based language models [45,
63] that easily combine multiple tasks in pretraining and
demonstrate powerful zero-shot capabilities.
Image gen-
eration language models can generate images autoregres-
sively [73] or via masked prediction [12, 13]. Both fam-
ilies have been extended to text-to-video [35, 36, 64, 72]
using paired data. Other text-to-video work with trans-
formers only leverages video-text pairs for training, but we
also leverage unpaired videos (without text) and the same
video for different tasks. Because video language mod-
els can ﬂexibly incorporate many tasks [41, 74], including
video-to-video, we extend this family of work to text- and
multimodal-conditioned tasks in this work with a synergis-
tic pretraining strategy across many tasks.
VideoPoet