title: xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
    Even in VLMs
summary: We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
  videos, particularly designed to efficiently capture temporal information over
  multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
  addition to the conventional visual tokenizer, which maps a sequence of tokens
  over multiple frames into a compact set of visual tokens. This enables
  BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
  vs. 4608 tokens). We explore different types of temporal encoders, including
  learnable spatio-temporal pooling as well as sequential models like Token
  Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
  question-answering accuracies comparable to much larger state-of-the-art models
  (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
  fewer visual tokens. The project website is at
  https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
paper_authors._id: 6718177310ca4f9b8f0be3f8
paper_authors.name: Michael S. Ryoo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-22T21:21:57.508Z
paper_authors._id: 670d93425c92b289329b19db
paper_authors.avatarUrl: /avatars/c7c9b67c331f8974384780e9ce1958a2.svg
paper_authors.fullname: Michael Ryoo
paper_authors.type: user
paper_authors.user: michaelryoo
paper_authors._id: 6718177310ca4f9b8f0be3f9
paper_authors.name: Honglu Zhou
paper_authors._id: 6718177310ca4f9b8f0be3fa
paper_authors.name: Shrikant Kendre
paper_authors._id: 6718177310ca4f9b8f0be3fb
paper_authors.name: Can Qin
paper_authors._id: 6718177310ca4f9b8f0be3fc
paper_authors.name: Le Xue
paper_authors._id: 6718177310ca4f9b8f0be3fd
paper_authors.name: Manli Shu
paper_authors._id: 6718177310ca4f9b8f0be3fe
paper_authors.name: Silvio Savarese
paper_authors._id: 6718177310ca4f9b8f0be3ff
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:41:26.876Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 6718177310ca4f9b8f0be400
paper_authors.name: Caiming Xiong
paper_authors._id: 6718177310ca4f9b8f0be401
paper_authors.name: Juan Carlos Niebles
paper_title: xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
    Even in VLMs
paper_summary: We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
  videos, particularly designed to efficiently capture temporal information over
  multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
  addition to the conventional visual tokenizer, which maps a sequence of tokens
  over multiple frames into a compact set of visual tokens. This enables
  BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
  vs. 4608 tokens). We explore different types of temporal encoders, including
  learnable spatio-temporal pooling as well as sequential models like Token
  Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
  question-answering accuracies comparable to much larger state-of-the-art models
  (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
  fewer visual tokens. The project website is at
  https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
paper_ai_summary: BLIP-3-Video is a video-specific multimodal language model using fewer visual tokens and a temporal encoder for efficient temporal information capture and accurate video question-answering.
paper_ai_keywords: multimodal language model temporal information temporal encoder visual tokenizer spatio-temporal pooling Token Turing Machines video question-answering visual tokens