Video-LaVIT: Unified Video-Language Pre-training with Decoupled
  Visual-Motional Tokenization
In light of recent advances in multimodal Large Language Models (LLMs), there
is increasing attention to scaling them from image-text data to more
informative real-world videos. Compared to static images, video poses unique
challenges for effective large-scale pre-training due to the modeling of its
spatiotemporal dynamics. In this paper, we address such limitations in
video-language pre-training with an efficient video decomposition that
represents each video as keyframes and temporal motions. These are then adapted
to an LLM using well-designed tokenizers that discretize visual and temporal
information as a few tokens, thus enabling unified generative pre-training of
videos, images, and text. At inference, the generated tokens from the LLM are
carefully recovered to the original continuous pixel space to create various
video content. Our proposed framework is both capable of comprehending and
generating image and video content, as demonstrated by its competitive
performance across 13 multimodal benchmarks in image and video understanding
and generation. Our code and models will be available at
https://video-lavit.github.io.
65c1bb8ba4c55b9d62c43b19
Yang Jin
claimed_verified
2024-10-10T11:00:00.802Z
64b79ec0a8c39dc078897430
/avatars/8339f3ab9bb7baaf69bf174eafb7282c.svg
Yang Jin
user
rain1011
65c1bb8ba4c55b9d62c43b1a
Zhicheng Sun
claimed_verified
2024-06-25T07:34:33.836Z
62fc758172a7ab50b4b89c8c
https://cdn-avatars.huggingface.co/v1/production/uploads/1677251053045-62fc758172a7ab50b4b89c8c.jpeg
Zhicheng Sun
user
feifeiobama
65c1bb8ba4c55b9d62c43b1b
Kun Xu
65c1bb8ba4c55b9d62c43b1c
Kun Xu
65c1bb8ba4c55b9d62c43b1d
Liwei Chen
65c1bb8ba4c55b9d62c43b1e
Hao Jiang
65c1bb8ba4c55b9d62c43b1f
Quzhe Huang
65c1bb8ba4c55b9d62c43b20
Chengru Song
65c1bb8ba4c55b9d62c43b21
Yuliang Liu
65c1bb8ba4c55b9d62c43b22
Di Zhang
65c1bb8ba4c55b9d62c43b23
Yang Song
65c1bb8ba4c55b9d62c43b24
Kun Gai
65c1bb8ba4c55b9d62c43b25
Yadong Mu
Video-LaVIT: Unified Video-Language Pre-training with Decoupled
  Visual-Motional Tokenization
In light of recent advances in multimodal Large Language Models (LLMs), there
is increasing attention to scaling them from image-text data to more
informative real-world videos. Compared to static images, video poses unique
challenges for effective large-scale pre-training due to the modeling of its
spatiotemporal dynamics. In this paper, we address such limitations in
video-language pre-training with an efficient video decomposition that
represents each video as keyframes and temporal motions. These are then adapted
to an LLM using well-designed tokenizers that discretize visual and temporal
information as a few tokens, thus enabling unified generative pre-training of
videos, images, and text. At inference, the generated tokens from the LLM are
carefully recovered to the original continuous pixel space to create various
video content. Our proposed framework is both capable of comprehending and
generating image and video content, as demonstrated by its competitive
performance across 13 multimodal benchmarks in image and video understanding
and generation. Our code and models will be available at
https://video-lavit.github.io.
A framework for pre-training multimodal Large Language Models using video decomposition into keyframes and temporal motions achieves competitive performance across benchmarks.
multimodal Large Language Models video decomposition keyframes temporal motions tokenizers generative pre-training video content image and video understanding image and video generation