title: Video-LLaVA: Learning United Visual Representation by Alignment Before
    Projection
summary: The Large Vision-Language Model (LVLM) has enhanced the performance of
  various downstream tasks in visual-language understanding. Most existing
  approaches encode images and videos into separate feature spaces, which are
  then fed as inputs to large language models. However, due to the lack of
  unified tokenization for images and videos, namely misalignment before
  projection, it becomes challenging for a Large Language Model (LLM) to learn
  multi-modal interactions from several poor projection layers. In this work, we
  unify visual representation into the language feature space to advance the
  foundational LLM towards a unified LVLM. As a result, we establish a simple but
  robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images
  and videos, mutually enhancing each other. Video-LLaVA achieves superior
  performances on a broad range of 9 image benchmarks across 5 image
  question-answering datasets and 4 image benchmark toolkits. Additionally, our
  Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on
  MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive
  experiments demonstrate that Video-LLaVA mutually benefits images and videos
  within a unified visual representation, outperforming models designed
  specifically for images or videos.
paper_authors._id: 655abd747d187b097e1176f6
paper_authors.name: Bin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T12:58:28.101Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 655abd747d187b097e1176f7
paper_authors.name: Bin Zhu
paper_authors._id: 655abd747d187b097e1176f8
paper_authors.name: Yang Ye
paper_authors._id: 655abd747d187b097e1176f9
paper_authors.name: Munan Ning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T12:53:35.582Z
paper_authors._id: 62a9748708a7ea93ff6d48ab
paper_authors.avatarUrl: /avatars/c6358f81e31f7d31e14899011ce67e09.svg
paper_authors.fullname: Munan Ning
paper_authors.type: user
paper_authors.user: MasonNing
paper_authors._id: 655abd747d187b097e1176fa
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:32:00.414Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 655abd747d187b097e1176fb
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:13:23.198Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: Video-LLaVA: Learning United Visual Representation by Alignment Before
    Projection
paper_summary: The Large Vision-Language Model (LVLM) has enhanced the performance of
  various downstream tasks in visual-language understanding. Most existing
  approaches encode images and videos into separate feature spaces, which are
  then fed as inputs to large language models. However, due to the lack of
  unified tokenization for images and videos, namely misalignment before
  projection, it becomes challenging for a Large Language Model (LLM) to learn
  multi-modal interactions from several poor projection layers. In this work, we
  unify visual representation into the language feature space to advance the
  foundational LLM towards a unified LVLM. As a result, we establish a simple but
  robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images
  and videos, mutually enhancing each other. Video-LLaVA achieves superior
  performances on a broad range of 9 image benchmarks across 5 image
  question-answering datasets and 4 image benchmark toolkits. Additionally, our
  Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on
  MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive
  experiments demonstrate that Video-LLaVA mutually benefits images and videos
  within a unified visual representation, outperforming models designed
  specifically for images or videos.
paper_ai_summary: Video-LLaVA is a unified large vision-language model that enhances performance across various image and video benchmarks by integrating visual representations into the language feature space.
paper_ai_keywords: Large Vision-Language Model LVLM unified tokenization multi-modal interactions Video-LLaVA mutual enhancement image benchmarks image question-answering datasets image benchmark toolkits Video-ChatGPT MSRVTT MSVD TGIF ActivityNet