After pretraining, we can ﬁne-tune the pretrained model
to either improve its performance on speciﬁc tasks or to en-
able it to undertake new tasks. For example, we ﬁnetune the
model on both text-to-video and image-to-video tasks using
a high-quality data subset. We observe improved generation
quality which aligns with ﬁndings from [82]. Furthermore,
we note that the ﬁne-tuned model mitigates the issue of de-
coding collapse, characterized by the degradation of pre-
dictions into repetitive tokens. This improvement not only
enhances the model’s output diversity but also enables in-
creasing the Classiﬁer-Free Guidance [31] scale, leading to
an overall enhancement in quality. In addition, we also ﬁne-
tune the pretrained model to perform video-to-audio gener-
ation.
VideoPoet