training tasks that extend beyond text-to-video generation.
A notable advantage of employing LLMs in video gen-
eration lies in the ease of integrating existing LLM frame-
works. This integration allows for reusing LLM infrastruc-
ture and leverages the optimizations our community has de-
veloped over many years for LLMs, including optimizations
in learning recipes for model scaling [8, 16], training and
inference infrastructure [17, 20], hardware, among other in-
novations. This couples with their ﬂexibility in encoding
many diverse tasks in the same model [8, 16, 34, 41, 45],
which stands in contrast to most diffusion models where
architectural changes and adapter modules are the dom-
inant approach used to adapt the model to more diverse
tasks [15, 22, 38, 80].

In this paper, we investigate the application of language
models in video generation, following the canonical train-

2
VideoPoet