In this work, we present GPT4Video to address the
limitations, which is a universal framework for augmenting
LLMs with both video understanding and generation capa-
bilities, as exemplified in Figure 1 (Ours). Inspired by previ-
ous success [50], we leverage the robust capabilities of well-
pretrained LLM and visual feature extractor for video un-
derstanding. In addition, we propose a simple and effective
video abstractor that employs attention mechanisms over
both spatial and temporal dimensions to facilitate precise
alignment between video content and textual descriptions.
For video generation, instead of incorporating “generative
vokens” as done in previous work, we develop a multi-
turn, interleaved multimodal instruction-following dataset
that trains the LLM to produce the detailed video descrip-
tions at the opportune moment, which serves as an input
prompt for a pre-trained text-to-video model. Our approach
has the following appealing advantages:
GPT4Video