The training of VideoPoet consists of two stages: (1)
pretraining and (2) task-adaptation. During pretraining,
VideoPoet incorporates a mixture of multimodal pretraining
objectives within an autoregressive transformer framework.
After pretraining, the model functions as a versatile multi-
task video generation model such as text-to-video, image-
to-video, video editing and video-to-video stylization, as
shown in Figure 1. Unlike [57], these capabilities are inher-
ently integrated into a single LLM, rather than relying on a
separate diffusion model controlled by text prompts. Dur-
ing subsequent task-adaptation, the pretrained model can be
further ﬁne-tuned either to enhance its generation quality on
the training tasks or to perform new tasks.
VideoPoet