title: Video-LaVIT: Unified Video-Language Pre-training with Decoupled
    Visual-Motional Tokenization
summary: In light of recent advances in multimodal Large Language Models (LLMs), there
  is increasing attention to scaling them from image-text data to more
  informative real-world videos. Compared to static images, video poses unique
  challenges for effective large-scale pre-training due to the modeling of its
  spatiotemporal dynamics. In this paper, we address such limitations in
  video-language pre-training with an efficient video decomposition that
  represents each video as keyframes and temporal motions. These are then adapted
  to an LLM using well-designed tokenizers that discretize visual and temporal
  information as a few tokens, thus enabling unified generative pre-training of
  videos, images, and text. At inference, the generated tokens from the LLM are
  carefully recovered to the original continuous pixel space to create various
  video content. Our proposed framework is both capable of comprehending and
  generating image and video content, as demonstrated by its competitive
  performance across 13 multimodal benchmarks in image and video understanding
  and generation. Our code and models will be available at
  https://video-lavit.github.io.
paper_authors._id: 65c1bb8ba4c55b9d62c43b19
paper_authors.name: Yang Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T11:00:00.802Z
paper_authors._id: 64b79ec0a8c39dc078897430
paper_authors.avatarUrl: /avatars/8339f3ab9bb7baaf69bf174eafb7282c.svg
paper_authors.fullname: Yang Jin
paper_authors.type: user
paper_authors.user: rain1011
paper_authors._id: 65c1bb8ba4c55b9d62c43b1a
paper_authors.name: Zhicheng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:34:33.836Z
paper_authors._id: 62fc758172a7ab50b4b89c8c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677251053045-62fc758172a7ab50b4b89c8c.jpeg
paper_authors.fullname: Zhicheng Sun
paper_authors.type: user
paper_authors.user: feifeiobama
paper_authors._id: 65c1bb8ba4c55b9d62c43b1b
paper_authors.name: Kun Xu
paper_authors._id: 65c1bb8ba4c55b9d62c43b1c
paper_authors.name: Kun Xu
paper_authors._id: 65c1bb8ba4c55b9d62c43b1d
paper_authors.name: Liwei Chen
paper_authors._id: 65c1bb8ba4c55b9d62c43b1e
paper_authors.name: Hao Jiang
paper_authors._id: 65c1bb8ba4c55b9d62c43b1f
paper_authors.name: Quzhe Huang
paper_authors._id: 65c1bb8ba4c55b9d62c43b20
paper_authors.name: Chengru Song
paper_authors._id: 65c1bb8ba4c55b9d62c43b21
paper_authors.name: Yuliang Liu
paper_authors._id: 65c1bb8ba4c55b9d62c43b22
paper_authors.name: Di Zhang
paper_authors._id: 65c1bb8ba4c55b9d62c43b23
paper_authors.name: Yang Song
paper_authors._id: 65c1bb8ba4c55b9d62c43b24
paper_authors.name: Kun Gai
paper_authors._id: 65c1bb8ba4c55b9d62c43b25
paper_authors.name: Yadong Mu
paper_title: Video-LaVIT: Unified Video-Language Pre-training with Decoupled
    Visual-Motional Tokenization
paper_summary: In light of recent advances in multimodal Large Language Models (LLMs), there
  is increasing attention to scaling them from image-text data to more
  informative real-world videos. Compared to static images, video poses unique
  challenges for effective large-scale pre-training due to the modeling of its
  spatiotemporal dynamics. In this paper, we address such limitations in
  video-language pre-training with an efficient video decomposition that
  represents each video as keyframes and temporal motions. These are then adapted
  to an LLM using well-designed tokenizers that discretize visual and temporal
  information as a few tokens, thus enabling unified generative pre-training of
  videos, images, and text. At inference, the generated tokens from the LLM are
  carefully recovered to the original continuous pixel space to create various
  video content. Our proposed framework is both capable of comprehending and
  generating image and video content, as demonstrated by its competitive
  performance across 13 multimodal benchmarks in image and video understanding
  and generation. Our code and models will be available at
  https://video-lavit.github.io.
paper_ai_keywords: Large Language Models (LLMs) video-language pre-training spatiotemporal dynamics video decomposition keyframes temporal motions tokenizers generative pre-training continuous pixel space multimodal benchmarks image and video understanding image and video generation